Index Check Ops → URL list assembly

Building URL Lists From Sitemaps and Server Logs

Every index check is exactly as good as its input list. Feed a checker duplicates, parameter variants, and dead URLs, and the output is a percentage of noise. List assembly is unglamorous, fully automatable, and the highest-return step in the whole system.

The three sources and what each is for

• Sitemaps — the "should exist" set. What the site declares as canonical inventory. Google's sitemap documentation is explicit that listing a URL is a hint, not a command — which is precisely why the declared set needs verification against the index. The free sitemap URL extractor flattens index files and nested maps into a plain list.
• Server logs — the "Google sees" set. URLs Googlebot actually requests, including ones no sitemap admits to. The delta between logs and sitemap is its own finding: parameter blowouts, forgotten sections, crawl waste.
• External reports — the "paid for" set. Link-vendor deliveries and placement lists. These never match sitemaps and never should — they get their own list and their own check cadence.

Normalization as a script, not a habit

# assemble.sh — sitemap set, normalized and deduped
cat sitemap-urls.txt \
  | sed -E 's/[?&](utm_[^&=]+|gclid|fbclid)=[^&]*//g' \
  | sed -E 's/\?$//' \
  | awk '!seen[$0]++' \
  > checkable.txt
wc -l checkable.txt   # the number that goes in the log

Two rules the script can't skip: resolve to final canonical targets (checking a URL that declares its canonical elsewhere produces a meaningless "not indexed"), and drop non-200s before the check — dead URLs belong in a fix list, not an index report.

Tag by template while you're there

A one-column list answers "how many". A two-column list — URL plus template tag — answers "what broke". Tagging is usually a regex per URL pattern, done once. Every downstream piece, from delta alerts to client reports, gets sharper when the batch sent to the bulk index checker carries template structure the report can be grouped by afterward.