Generate Sitemaps for All of ASP.NET Core

While ASP.NET Core is a robust web framework, it lacks some core features that make executing a creative-focused site more straightforward. One of those features is the ability to generate a sitemap. If folks can’t find your content, then for all intents and purposes, it doesn’t exist. Sitemaps tell search engines which parts of your site are essential for user searches.

In this post, I’ll be combining an open-source project called DotnetSitemapGenerator with some of my custom infrastructure code to generate sitemap nodes from ASP.NET Core Minimal APIs, ASP.NET Core MVC actions, and Razor Pages. The post covers all the server-side rendered paradigms within ASP.NET Core, except for Blazor. As a bonus, I also generate sitemap nodes from a database using Entity Framework Core.

What is a Sitemap?

A sitemap is a file provided by your web application that provides search engines with information about pages, videos, and other potential points of interest. Search engines use this file to crawl your site more efficiently, helping keep your search results up to date and more relevant for prospective visitors. A sitemap is typically hosted at the root of a website at /sitemap.xml by convention but can be configured depending on your use case.

While a sitemap is optional for a well-linked site, there are advantages to having one for your application.

For instance, crawlers might have difficulty distinguishing between valuable pages if you have an extensive website. A large site can lead to issues indexing newer pages, and thus you may lose valuable visitors for more recent content.

Newer sites might also struggle to have crawlers discover their existence, as fewer or no inbound links exist. Generating a sitemap and submitting it to popular search engines ensures your site finds its way quickly into the results of users. As your site becomes more popular, this is less likely a concern, but we all start somewhere.

You may want search engines to index your rich media content separately from your written content. Having separate sitemaps can mean new avenues for users to discover your site. Sitemaps can clarify what value you are providing by creating distinctions between different types of content.

While sitemaps can be helpful for many sites, they do not guarantee all items within your XML files will be indexed by search engines. That said, there are values for change frequency and priority that can help search engines prioritize their crawlers. Generally, it’s better to be safe than sorry when there’s so much competition for visitors.

A Basic Sitemap

Sitemaps, in their simplest form, are a collection of URLs. Let’s look at the final sample we’ll generate in this post.

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url>
        <loc>http://localhost:5077/test</loc>
        <changefreq>daily</changefreq>
    </url>
    <url>
        <loc>http://localhost:5077/</loc>
    </url>
    <url>
        <loc>http://localhost:5077/Privacy</loc>
    </url>
    <url>
        <loc>http://localhost:5077/cool/3</loc>
    </url>
    <url>
        <loc>http://localhost:5077/products/1/express-galaxy-tumbler</loc>
    </url>
    <url>
        <loc>http://localhost:5077/products/2/aero-life-air-purifier</loc>
    </url>
    <url>
        <loc>http://localhost:5077/products/3/ocean-wave-projector</loc>
    </url>
    <url>
        <loc>http://localhost:5077/products/4/illuminated-globe-decor</loc>
    </url>
    <url>
        <loc>http://localhost:5077/products/5/moonlight-cushion</loc>
    </url>
    <url>
        <loc>http://localhost:5077/products/6/sunrise-alarm-clock</loc>
    </url>
    <url>
        <loc>http://localhost:5077/products/7/frosty-mini-fridge</loc>
    </url>
    <url>
        <loc>http://localhost:5077/products/8/breeze-tower-fan</loc>
    </url>
    <url>
        <loc>http://localhost:5077/products/9/comet-electric-scooter</loc>
    </url>
    <url>
        <loc>http://localhost:5077/products/10/starlight-projector</loc>
    </url>
</urlset>

The XML file is presentationally unremarkable but essential to building an online presence.

The XML file has endpoints from ASP.NET Core MVC controllers, Razor Pages, and Minimal APIs. Let’s look at how we register our Sitemap infrastructure, and then we’ll go through each mechanism that provides these values to our sitemap.

Setting up the ASP.NET Core Infrastructure

We’ll start inside Program.cs to see how all the pieces come together, then explore how we pull these nodes into our sitemap XML file.

using Microsoft.EntityFrameworkCore;
using Shopping.Models;
using Shopping.Sitemap;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddRazorPages();
builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddHttpContextAccessor();

// my unique extension method for sitemap information
builder.Services.AddSitemap();
builder.Services.AddOutputCache(options => {
    options.AddPolicy("sitemap", b => b.Expire(TimeSpan.FromSeconds(1)));
});

builder.Services.AddDbContext<Database>(
    ob => ob.UseSqlite("Data Source = database.db")
);

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
    // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
    app.UseHsts();
}

app.UseHttpsRedirection();
app.UseStaticFiles();
app.UseRouting();
app.UseOutputCache();

app.MapSitemap().CacheOutput("sitemap");

app.MapGet("cool/{id}", () => "cool beans")
   .WithSitemap("cool", new { id = 3 });

app.UseAuthorization();
app.MapRazorPages();
app.MapControllers();
app.Run();

We first start by adding our services to the ASP.NET Core services collection. The registrations include our sitemap providers, the Endpoints Api Explorer (for Minimal APIs), and the HttpContextAccessor for generating links.

builder.Services.AddEndpointsApiExplorer();
builder.Services.AddHttpContextAccessor();
// my unique extension method for sitemap information
builder.Services.AddSitemap();

Further down the file is our sitemap.xml endpoint, enabling output caching.

app.MapSitemap().CacheOutput("sitemap");

I’ve defined these methods in the project’s SitemapExtensions.cs file.

using Shopping.Sitemap.Providers;

namespace Shopping.Sitemap;

public static class SitemapExtensions
{
    public static RouteHandlerBuilder MapSitemap(this IEndpointRouteBuilder endpoints, string path = "sitemap.xml")
    {
        return endpoints.MapGet(path, async (SitemapBuilder sitemap) =>
        {
            var xml = await sitemap.GenerateAsync();
            return Results.Stream(xml, "text/xml");
        });
    }

    public static void AddSitemap(this IServiceCollection services)
    {
        // add sitemap services
        services.AddScoped<ISitemapUrlProvider, PagesSitemapUrlProvider>();
        services.AddScoped<ISitemapUrlProvider, EndpointsSitemapUrlProvider>();
        services.AddScoped<ISitemapUrlProvider, ProductSitemapSitemapUrlProvider>();
        services.AddScoped<SitemapBuilder>();
    }

    public static RouteHandlerBuilder WithSitemap(this RouteHandlerBuilder endpoint,
        string name, object? defaults = null)
    {
        return endpoint
            // adds RouteName and EndpointName
            .WithName(name)
            .WithMetadata(new SitemapAttribute
            {
                RouteValues = new RouteValueDictionary(defaults)
            });
    }
}

The AddSiteMap method registers our URL providers, which will ultimately be used by our SitemapBuilder to combine all the nodes for our sitemap.xml file.

Now that we have infrastructure covered let’s talk about each ASP.NET Core approach your endpoints may be implemented in.

ASP.NET Core Razor Pages and MVC Actions

Razor Pages and MVC Views are considered Actions in ASP.NET Core, so they behave very similarly within the context of sitemap generation. The only difference between the two approaches is how you generate links from each route.

Let’s look at our PagesSitemapUrlProvider, which will support MVC and Razor Pages.

using DotnetSitemapGenerator;
using Microsoft.AspNetCore.Mvc.Infrastructure;
using IsRazorPage = Microsoft.AspNetCore.Mvc.RazorPages.CompiledPageActionDescriptor;
using IsMvc = Microsoft.AspNetCore.Mvc.Controllers.ControllerActionDescriptor;

namespace Shopping.Sitemap.Providers;

public class PagesSitemapUrlProvider : ISitemapUrlProvider
{
    private readonly LinkGenerator _linkGenerator;
    private readonly IHttpContextAccessor _httpContextAccessor;
    private readonly IActionDescriptorCollectionProvider _actionDescriptorCollectionProvider;
    private readonly ILogger<PagesSitemapUrlProvider> _logger;

    public PagesSitemapUrlProvider(
        LinkGenerator linkGenerator,
        IHttpContextAccessor httpContextAccessor,
        IActionDescriptorCollectionProvider actionDescriptorCollectionProvider,
        ILogger<PagesSitemapUrlProvider> logger)
    {
        _linkGenerator = linkGenerator;
        _httpContextAccessor = httpContextAccessor;
        _actionDescriptorCollectionProvider = actionDescriptorCollectionProvider;
        _logger = logger;
    }

    public Task<IReadOnlyCollection<SitemapNode>> GetNodes()
    {
        var httpContext = _httpContextAccessor.HttpContext!;
        var nodes = new List<SitemapNode>();

        foreach (var descriptor in _actionDescriptorCollectionProvider.ActionDescriptors.Items)
        {
            // LastOrDefault is used to get the closest SitemapAttribute to the endpoint
            var exists = descriptor.EndpointMetadata.LastOrDefault(em => em is SitemapAttribute); 
            if (exists is not SitemapAttribute sitemapAttribute) continue;
            
            var url = descriptor switch
            {
                // Razor Pages
                IsRazorPage razorPage =>
                    _linkGenerator.GetUriByPage(httpContext, page: razorPage.ViewEnginePath),
                // ASP.NET Core MVC
                IsMvc controller =>
                    _linkGenerator.GetUriByAction(httpContext,
                        action: controller.ActionName,
                        controller: controller.ControllerName,
                        // use the values provided by the user (if any)
                        values: sitemapAttribute.RouteValues),
                _ => null
            };

            if (ShouldAddUrl(nodes, url))
            {
                nodes.Add(new SitemapNode(url)
                {
                    ChangeFrequency = sitemapAttribute.ChangeFrequency,
                    Priority = sitemapAttribute.Priority
                });
            }
        }

        return Task.FromResult<IReadOnlyCollection<SitemapNode>>(nodes);
    }
    
    private static bool ShouldAddUrl(List<SitemapNode> nodes, string? url)
    {
        // if the url failed to generate, don't add a record
        if (string.IsNullOrWhiteSpace(url)) return false;
        // if it already exists based on the URL, don't add it
        return !nodes.Exists(n => n.Url.Equals(url, StringComparison.OrdinalIgnoreCase));
    }
    
}

The trick in this method is utilizing the information from ActionDescriptor to determine how we want to generate links using the LinkGenerator instance. We also require an HttpContext to determine values for the base URL.

Now all we need to do is decorate the endpoints we want to include in our Sitemap generation process. Here’s an example using custom attributes on our controller, where the closest attribute will be used when creating the sitemap node.

using Microsoft.AspNetCore.Mvc;
using Shopping.Sitemap;

namespace Shopping.Controllers;

[ControllerSitemap]
public class TestController : Controller
{
    [ActionSitemap]
    [Route("test")]
    public IActionResult Index()
    {
        return View();
    }
}

public class ControllerSitemapAttribute : SitemapAttribute
{
}

public class ActionSitemapAttribute : SitemapAttribute
{
    public ActionSitemapAttribute()
    {
        ChangeFrequency = DotnetSitemapGenerator.ChangeFrequency.Daily;
    }
}

The implementation in Razor Pages is similar, but here we use the base SitemapAttribute.

using Microsoft.AspNetCore.Mvc.RazorPages;
using Microsoft.EntityFrameworkCore;
using Shopping.Models;
using Shopping.Sitemap;

namespace Shopping.Pages;

[Sitemap]
public class IndexModel : PageModel
{
    private readonly ILogger<IndexModel> _logger;
    private readonly Database _db;

    public IndexModel(ILogger<IndexModel> logger, Database db)
    {
        _logger = logger;
        _db = db;
    }

    public List<Product> Products { get; set; } = new();

    public async Task OnGet()
    {
        Products = await _db
            .Products
            .OrderBy(p => p.Id)
            .ToListAsync();
    }
}

Cool! What about Minimal API endpoints?

ASP.NET Core Minimal API endpoints

While most developers use ASP.NET Core Minimal APIs for API endpoints that return JSON or XML, developers may also choose to return any content type, which may include HTML files. So you may find these endpoints straddling the line between API and presentational duties.

For completeness, let’s add an option for Minimal API endpoints.

using DotnetSitemapGenerator;
using Microsoft.AspNetCore.Mvc.ApiExplorer;

namespace Shopping.Sitemap.Providers;

public class EndpointsSitemapUrlProvider : ISitemapUrlProvider
{
    private readonly LinkGenerator _linkGenerator;
    private readonly IHttpContextAccessor _httpContextAccessor;
    private readonly IApiDescriptionGroupCollectionProvider _apiDescriptionGroupCollectionProvider;
    private readonly ILogger<PagesSitemapUrlProvider> _logger;

    public EndpointsSitemapUrlProvider(
        LinkGenerator linkGenerator,
        IHttpContextAccessor httpContextAccessor,
        IApiDescriptionGroupCollectionProvider apiDescriptionGroupCollectionProvider,
        ILogger<PagesSitemapUrlProvider> logger)
    {
        _linkGenerator = linkGenerator;
        _httpContextAccessor = httpContextAccessor;
        _apiDescriptionGroupCollectionProvider = apiDescriptionGroupCollectionProvider;
        _logger = logger;
    }

    public Task<IReadOnlyCollection<SitemapNode>> GetNodes()
    {
        var httpContext = _httpContextAccessor.HttpContext!;
        var nodes = new List<SitemapNode>();

        // Minimal Apis that might return HTML
        foreach (var group in _apiDescriptionGroupCollectionProvider.ApiDescriptionGroups.Items)
        {
            var endpoints =
                group
                    .Items
                    .Where(i => HttpMethods.IsGet(i.HttpMethod ?? ""))
                    .Where(i => i.ActionDescriptor.EndpointMetadata.Any(em => em is SitemapAttribute));

            foreach (var endpoint in endpoints)
            {
                var attribute = endpoint
                    .ActionDescriptor
                    .EndpointMetadata
                    .LastOrDefault(a => a is SitemapAttribute);

                if (attribute is not SitemapAttribute sitemapAttribute)
                    continue;

                var routeName = endpoint
                    .ActionDescriptor
                    .EndpointMetadata
                    .Where(m => m is RouteNameMetadata)
                    .Cast<RouteNameMetadata>()
                    .Select(a => a.RouteName)
                    .FirstOrDefault();

                if (routeName is null)
                    continue;

                var url = _linkGenerator.GetUriByName(
                    httpContext,
                    routeName,
                    values: sitemapAttribute.RouteValues
                );

                if (ShouldAddUrl(nodes, url))
                {
                    nodes.Add(new SitemapNode(url)
                    {
                        ChangeFrequency = sitemapAttribute.ChangeFrequency,
                        Priority = sitemapAttribute.Priority
                    });
                }
            }
        }

        return Task.FromResult<IReadOnlyCollection<SitemapNode>>(nodes);
    }

    private static bool ShouldAddUrl(List<SitemapNode> nodes, string? url)
    {
        // if the url failed to generate, don't add a record
        if (string.IsNullOrWhiteSpace(url)) return false;
        // if it already exists based on the URL, don't add it
        return !nodes.Exists(n => n.Url.Equals(url, StringComparison.OrdinalIgnoreCase));
    }
}

We’ll need to annotate our API endpoints to get Minimal API endpoints into the sitemap.

app.MapGet("cool/{id}", () => "cool beans")
   .WithSitemap("cool", new { id = 3 });

In this example, we have a route value that needs to be satisfied before generating the link. Using the WithSitemap method, we can provide a default value. The extension method also adds a name to our endpoint metadata, which allows us to generate the correct link.

We’re on a roll. What about data-driven pages?

Database-driven pages with Entity Framework Core

Many sites have dynamic pages built on a dataset. Dynamic URLs are standard on shopping sites with an extensive product catalog. Let’s see how we can implement a ProductSitemapUrlProvider.

using DotnetSitemapGenerator;
using Microsoft.EntityFrameworkCore;
using Shopping.Models;

namespace Shopping.Sitemap.Providers;

public class ProductSitemapSitemapUrlProvider : ISitemapUrlProvider
{
    private readonly Database _db;
    private readonly LinkGenerator _linkGenerator;
    private readonly IHttpContextAccessor _httpContextAccessor;

    public ProductSitemapSitemapUrlProvider(
        Database db, 
        LinkGenerator linkGenerator, 
        IHttpContextAccessor httpContextAccessor)
    {
        _db = db;
        _linkGenerator = linkGenerator;
        _httpContextAccessor = httpContextAccessor;
    }
    
    public async Task<IReadOnlyCollection<SitemapNode>> GetNodes()
    {
        var elements = new List<SitemapNode>();
        var products = await _db.Products.OrderBy(x => x.Id).ToListAsync();
        
        foreach (var product in products)
        {
            var url = _linkGenerator.GetUriByPage(
                _httpContextAccessor.HttpContext!,
                page: "/Products",
                values: new { product.Id, product.Slug });
            
            elements.Add(new SitemapNode(url));
        }

        return elements;
    }
}

That’s pretty straightforward. We must point to our product detail page, a Razor page at /Products, and get a unique URL for each product.

Conclusion

Working sample at GitHub repository for ASP.NET Core Sitemap

With some help from an OSS project, I created a robust framework for generating sitemaps in ASP.NET Core using different programming approaches. While I’m sure this approach has more room for improvement, I’m pretty happy with the final result. What do you think?

I added this project to GitHub, and I don’t intend to make it a NuGet package soon since I see developers adapting the code for each project. If you’re inspired to make it a NuGet package, you can do so; just let me know.

Thanks for reading and sharing my posts with friends and colleagues.