A tiny Python checker for non-200 URLs in your sitemap

#python #sitemap #seo #qa

Your sitemap is a promise to search engines: these are the pages that matter. If some of those URLs return errors or redirect elsewhere, the promise is broken, and nobody is told. A short script can read the sitemap and report every URL that does not answer with a plain 200.

This one uses only the Python standard library, so it runs anywhere Python 3 is installed.

What it does

Downloads the sitemap you give it.
If it is a sitemap index, follows each child sitemap.
Requests every listed URL.
Prints any URL that returns something other than 200, or that ends up at a different address after redirects.

The script

Save it as sitemap_check.py:

import sys
import time
import urllib.error
import urllib.request
import xml.etree.ElementTree as ET

TIMEOUT = 15
HEADERS = {"User-Agent": "sitemap-checker/1.0"}


def fetch(url):
    request = urllib.request.Request(url, headers=HEADERS)
    return urllib.request.urlopen(request, timeout=TIMEOUT)


def locs(sitemap_url):
    with fetch(sitemap_url) as response:
        root = ET.fromstring(response.read())
    found = [el.text.strip() for el in root.findall(".//{*}loc") if el.text]
    if root.tag.endswith("sitemapindex"):
        return [url for child in found for url in locs(child)]
    return found


def check(url):
    try:
        with fetch(url) as response:
            return response.status, response.geturl()
    except urllib.error.HTTPError as err:
        return err.code, url
    except Exception as err:
        return type(err).__name__, url


def main(sitemap_url):
    problems = 0
    for url in locs(sitemap_url):
        status, final = check(url)
        if status != 200 or final != url:
            problems += 1
            note = f"\t-> {final}" if final != url else ""
            print(f"{status}\t{url}{note}")
        time.sleep(0.2)
    print(f"{problems} problem URL(s)")
    return 1 if problems else 0


if __name__ == "__main__":
    if len(sys.argv) != 2:
        sys.exit("usage: python sitemap_check.py <sitemap-url>")
    sys.exit(main(sys.argv[1]))

Run it with the address of your own sitemap:

python sitemap_check.py <your-sitemap-address>

How the pieces work

findall(".//{*}loc") matches loc elements whatever namespace the sitemap declares, which avoids hard-coding the namespace string.
urllib follows redirects automatically and geturl() returns the final address. Comparing it with the requested URL is how the script spots redirects, which would otherwise look like a normal 200.
HTTPError carries the status code for 404, 410, 500 and similar, so those are reported rather than crashing the run.
Anything else, such as a timeout or a DNS failure, is reported by exception name.
time.sleep(0.2) keeps the run polite. Raise it if your server struggles, and only run the script against sites you manage.
The exit code is 1 when problems are found, so the script can sit in a scheduled job or a CI step.

Reading the output

Each line shows the status, the sitemap URL and, for redirects, the destination. Treat them differently:

404 or 410. Remove the URL from the sitemap, or restore the page if it should exist.
5xx. Check the server logs. Run the script again before assuming it is permanent, since a busy moment can cause one-off errors.
Redirects. The sitemap should list the final URL. Update the sitemap source, whether that is a plugin setting or a build step, rather than editing the file by hand.
Trailing-slash differences. A URL listed without a slash that redirects to one with a slash is a configuration mismatch worth fixing once, in the generator.

Limits

This script checks status codes, not content. A page can answer 200 and still be empty, noindexed or showing a soft error. It also fetches each page body, which is simple but slower than a header-only request; for very large sitemaps, consider a crawler built for the job.

Run it before each release and after any migration. A clean report means every URL you told search engines about still answers properly. If you need a team to look after the build, hosting and templates behind the sitemap, ask about website build and maintenance support, which sits inside a monthly subscription at Gorilla Vibe.

Written with AI assistance for the Gorilla Vibe marketing team.

DEV Community