XML Sitemaps for Larger Sites: What Actually Belongs in One

A sitemap for a ten-page site is barely a decision — list everything, done. A sitemap for a few hundred or a few thousand pages is a different exercise entirely, and the mistakes that show up at that scale are almost always about inclusion, not format.

Include what you want indexed, not everything that exists

A sitemap is a list of URLs you're telling search engines are worth crawling and indexing. That's a narrower category than "every URL my site can technically produce." Internal search result pages, filtered or sorted product listing variants, tag archive pages with three items in them, and duplicate content reachable through multiple paths are all real URLs — and all bad sitemap entries. Each one is a small vote for "index this," spent on a page that adds little.

Keep it in sync with reality

The most common large-site sitemap problem isn't a bad initial list — it's staleness. A sitemap generated once at launch and never regenerated silently drifts: it lists pages that have since been removed or redirected (wasted crawl budget on dead ends) and misses everything added since (the pages most likely to need discovery help, since they haven't accumulated internal links yet). A sitemap that's regenerated on every deploy, or on a daily schedule, stays honest; a static file checked in once does not.

Split large sitemaps deliberately

Search engines cap how many URLs a single sitemap file can list (50,000 is the common ceiling). Past that, a sitemap index file — a small file that just points to several sub-sitemaps — is the standard pattern. Done well, splitting by content type (products, blog posts, category pages) also gives you a diagnostic tool: if one sub-sitemap consistently shows worse indexing coverage in Search Console than the others, you've isolated exactly which section of the site has a problem, rather than guessing across an undifferentiated pile of URLs.

lastmod is worth setting accurately

The optional <lastmod> tag isn't decorative — it's a real signal search engines use to decide whether a previously-crawled page is worth revisiting. Setting it accurately (to when the page's content actually changed, not to "the day the sitemap was regenerated" for every URL uniformly) helps a crawler prioritize genuinely updated content over pages that haven't changed in months. Setting it inaccurately — the same timestamp on every URL, every single day — teaches search engines to ignore the field entirely, which defeats its purpose for the pages where it would have mattered.