Generating hreflang sitemaps from a page list (and validating the reciprocal links)

#seo #webdev #python #i18n

Bilingual sites (English and Arabic is our everyday case) usually get hreflang wrong in one of three ways: a page points at a translation that does not point back, an alternate URL redirects or returns 404, or the language codes are invalid. Putting hreflang in the XML sitemap instead of the HTML head makes it easier to generate and to test in one place. Here is a small, dependency-free Python script that does both.

The input

A CSV with one row per page group. Each column is a language-region code and each cell is that version's URL:

en-ae,ar-ae
https://example.com/en/services/seo,https://example.com/ar/services/seo
https://example.com/en/contact,https://example.com/ar/contact
https://example.com/en/about,

An empty cell means the page has no translation. That page still gets a sitemap entry, just without alternates.

The generator

import csv
from xml.sax.saxutils import escape

def build_sitemap(csv_path, out_path, default_lang="en-ae"):
    rows = list(csv.DictReader(open(csv_path, encoding="utf-8")))
    lines = ['<?xml version="1.0" encoding="UTF-8"?>',
             '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" '
             'xmlns:xhtml="http://www.w3.org/1999/xhtml">']
    for row in rows:
        group = {lang: url.strip() for lang, url in row.items() if url and url.strip()}
        for lang, url in group.items():
            lines.append("  <url>")
            lines.append(f"    <loc>{escape(url)}</loc>")
            if len(group) > 1:
                for alt_lang, alt_url in group.items():
                    lines.append(f'    <xhtml:link rel="alternate" hreflang="{alt_lang}" href="{escape(alt_url)}"/>')
                if default_lang in group:
                    lines.append(f'    <xhtml:link rel="alternate" hreflang="x-default" href="{escape(group[default_lang])}"/>')
            lines.append("  </url>")
    lines.append("</urlset>")
    open(out_path, "w", encoding="utf-8").write("\n".join(lines))

build_sitemap("pages.csv", "sitemap-hreflang.xml")

Two details matter. Every URL in a group lists all members of the group, itself included; that is what makes the annotations reciprocal. And x-default points at one chosen version, here English, for visitors whose language matches none of the listed ones.

The validator

Generating correct XML is only half of it. The URLs themselves must answer 200 and must not redirect, otherwise search engines ignore the pair.

import urllib.request, urllib.error

def check(url, timeout=15):
    req = urllib.request.Request(url, method="HEAD", headers={"User-Agent": "hreflang-check/1.0"})
    class NoRedirect(urllib.request.HTTPRedirectHandler):
        def redirect_request(self, *a, **k):
            return None
    opener = urllib.request.build_opener(NoRedirect)
    try:
        return opener.open(req, timeout=timeout).status
    except urllib.error.HTTPError as e:
        return e.code
    except Exception as e:
        return f"error: {e.__class__.__name__}"

VALID = {"en-ae", "ar-ae", "en", "ar", "x-default"}  # extend for your markets

for row in csv.DictReader(open("pages.csv", encoding="utf-8")):
    for lang, url in row.items():
        if lang.lower() not in VALID:
            print(f"BAD CODE  {lang}")
        if url and url.strip():
            status = check(url.strip())
            if status != 200:
                print(f"{status}  {lang}  {url}")

Run it before every deploy that touches URLs. A clean run prints nothing.

Things the script will not catch

Canonical conflicts. If the Arabic page has a canonical pointing at the English page, the hreflang pair is contradicted. Check that each version is self-canonical.
Content that is not actually translated. An "Arabic" URL that still serves English text is a quality problem, not a markup problem.
Region codes you do not mean. ar-ae targets Arabic speakers in the UAE; ar targets Arabic speakers anywhere. Use region codes only when the content really differs by region.

Submitting it

Reference the new file in robots.txt (Sitemap: https://example.com/sitemap-hreflang.xml) or submit it in Search Console, and remove any conflicting hreflang tags from the HTML head so the two sources cannot disagree.

More about how we approach search for bilingual and multilingual sites is on our SEO service page.

Written with AI assistance for the Gorilla Vibe marketing team.

DEV Community