DEV Community

RoamProxy
RoamProxy

Posted on

A full-width email address got past our signup checks. Normalize with NFKC, then insist on ASCII.

#python #unicode #webdev #security

One of our verification emails bounced this week with an error we had never seen:

422 Invalid `to` field. The email address contains non-ASCII characters.
Enter fullscreen mode Exit fullscreen mode

The address looked normal at a glance. It was not. The domain had been typed in full-width characters, the forms you get from a CJK input method: ｅｘａｍｐｌｅ.tech instead of example.tech. Our signup form accepted it, our database stored it, and only the mail provider noticed.

That is a correctness bug, but it is also a consistency bug. Anything that keys on the domain (rate limits, duplicate checks, a blocklist) was looking at a string that did not match the "same" domain in ASCII.

What browsers do and don't stop

In our testing, Chrome's <input type="email"> refused the full-width address, so a person using the form normally cannot submit one. A script posting straight to the endpoint can. Client-side validation is a UX feature; the server still has to decide what an email address is.

The fix: fold first, then validate

Unicode NFKC normalization maps compatibility characters to their plain forms. Full-width letters, the full-width ＠ and the full-width ． all fold to ASCII:

import unicodedata

def norm_email(raw):
    if not raw:
        return ""
    s = unicodedata.normalize("NFKC", raw).strip().lower()
    if not s.isascii():
        return None   # still not ASCII after folding: reject
    return s

norm_email("ＡＢＣ＠ＥＸＡＭＰＬＥ．ＣＯＭ")   # '[email protected]'
norm_email("user@bücher.de")              # None
Enter fullscreen mode Exit fullscreen mode

Two rules made this safe:

  1. Normalize once, at the boundary, before anything else reads the value. Rate limits, duplicate checks and the stored address all see the same string. If one code path normalizes and another doesn't, you have built a bypass.
  2. Apply it to every way an account gets created. We have email signup and Google sign-in. Both go through the same function. An OAuth provider hands you an email string too; it is still input.

The trade-off we accepted

user@bücher.de is a real, valid internationalized address, and we now reject it. Our mail provider cannot deliver to non-ASCII addresses anyway, so accepting them only meant creating accounts that can never be verified. If your provider supports SMTPUTF8, you would keep NFKC and drop the ASCII check, or convert the domain to punycode instead.

Tests that pin the behaviour

  • full-width input is stored as its ASCII form, and the verification mail goes to the ASCII address
  • input that is still non-ASCII after folding creates no account
  • a blocklisted domain is still blocked when typed in full-width
  • plain ASCII addresses behave exactly as before
  • an existing Google-linked account still signs in by its stable account ID, whatever the email string looks like

Ten small tests, one function, and a class of "same address, different bytes" problems gone.

We run RoamProxy, a pay-as-you-go residential proxy service. This came out of our own signup flow.

Top comments (0)