DEV Community

orca_forge
orca_forge

Posted on Originally published at forge.workstyle.tech

When '少々' becomes 'しょも' — Japanese characters removed by whitelist

#tts #unicode

📝 Originally published (in Japanese) at forge.workstyle.tech.

I received a report about the avatar for the inquiry desk:

"しょうしょうおまちください" becomes "しょもおまちください".

Since I had just retrained the voice model multiple times, I first suspected the model. To cut to the chase, the model, parameters, and cache were all fine, but the input text passed to TTS was corrupted.

Debugging from the downstream

I'll list my suspicions in the order I checked them. This order itself is a lesson learned.

Model: I traced the database to see which model the inquiry site was using. It was correctly using the latest trained model.

Cache: There was a TTS cache table, so I checked if it was returning old audio. The entries were from a different provider and over a month old, so they were unrelated.

Synthesis parameters: The runtime was using style_weight=2.0 and sdp_ratio=0.8. My validation used 1.0 / 0.4, so I thought this might be the cause. I tested various combinations:

style_weight 1.0 / 2.0 / 3.0 / 4.0   → all ratio 1.00
sdp_ratio    0.2 / 0.4 / 0.6 / 0.8   → all ratio 1.00
All 12 styles × weight 2.0             → all ratio 1.00
Enter fullscreen mode Exit fullscreen mode

Emotional styles: I synthesized "少々お待ちください。" with all 12 styles, and they were all normal.

Pipeline: The voice pipeline had been moved to a separate service, so I checked if it was synthesizing independently. Synthesis was handled on the backend, and the pipeline was the same.

After several hours, everything checked out.

Printing the preprocessed output once

The only thing left was the input text.

>>> _clean_tts_text('少々お待ちください。')
'少お待ちください。'
Enter fullscreen mode Exit fullscreen mode

The repeating character "々" was missing. And when this corrupted text was synthesized, it sounded like this:

Input '少々お待ちください' → Heard '少々お待ちください' (normal)
Input '少お待ちください'    → Heard 'ショーをお待ちください'   ← this
Enter fullscreen mode Exit fullscreen mode

"ショーを" was heard as "しょも". A 5-minute check was done last.

Cause: "々" is not in the Kanji range

The preprocessor had a whitelist to remove emoticons, emojis, and special characters.

# TTS preprocessing: Remove unnecessary symbols and emoticons
_TTS_ALLOWED_RE = re.compile(
    r"[^぀-ゟ"   # Hiragana
    r"゠-ヿ"     # Katakana
    r"一-鿿"     # Kanji
    r"ｦ-ﾟ"     # Half-width Katakana
    r"a-zA-Zａ-ｚＡ-Ｚ"
    r"0-9０-９"
    r"、。！？,.ー"
    r"\s"
    r"]"
)
Enter fullscreen mode Exit fullscreen mode

The intention is clear, and the implementation is straightforward. The problem is that 一-鿿 (CJK Unified Ideographs) does not include "々". "々" is 々, which is in the CJK Symbols and Punctuation block. It's classified as a symbol, not a Kanji character.

For Japanese speakers, "々" is considered a Kanji character, but Unicode classifies it differently.

Found 7 missing characters

Since one character was missing, there might be others. I checked all characters used in Japanese that are outside the CJK Unified Ideographs.

Character Example Result Impact
々 少々・日々 少・日 "しょも"
〆 〆切 切 "きり"
〇 〇月〇日 月日 Dates disappear
𠮷 (CJK Extension A) 𠮷野家 野家 Proper nouns break
髙 﨑 (Compatibility Ideographs) 﨑山 山 Names break
〜 10〜20分 1020分 Numbers become something else
: 3:30 330 Times become something else

At the inquiry desk, "﨑山さま" becoming "山さま" is quite bad. "10〜20分" becoming "せんにじゅっぷん" is similarly problematic.

On the other hand, % & 「」 are also removed, but this is intentional as they're unnecessary for reading. It's not about keeping everything.

Keeping symbols didn't fix it

I straightforwardly added 〜 and : to the allowlist. It got worse.

'午後330に開始します'   → Heard '午後330に…'          (numbers incorrect)
'午後3:30に開始します'  → Heard '5も30、30に開始します'  ← worse when kept
Enter fullscreen mode Exit fullscreen mode

TTS couldn't interpret : as a time and produced noise like "ごも". 〜 was treated as a comma, not "から".

Removing changes the meaning, keeping makes it unreadable. Neither was correct.

Opening up to Japanese

The correct solution was a third option: convert to Japanese at the preprocessing stage.

_TTS_CLEAN_PATTERNS = [
    # ⚠️ Symbols with numerical meaning are neither removed nor kept but "converted to Japanese".
    # Removing turns "10〜20分" into "1020分", and keeping makes TTS unreadable,
    # producing noise like "3:30" → "5も30、30" (verified).
    (re.compile(r"(\d)\s*[〜～~]\s*(\d)"), r"\1から\2"),   # 10〜20 → 10から20
    (re.compile(r"(\d{1,2})\s*[:：]\s*(\d{2})"), r"\1時\2分"),      # 3:30 → 3時30分
    ...
]
Enter fullscreen mode Exit fullscreen mode

⚠️ Place substitutions before removal. Otherwise, 10〜20 would first become 1020, and the substitution target would be lost.

And add necessary characters to the allowlist.

r"一-鿿"     # Kanji
# ⚠️ Characters necessary for Japanese reading but outside CJK Unified Ideographs.
# In practice, "少々お待ちください" became "少お待ちください",
# and was pronounced as "ショーをお待ちください".
r"々〆〻"  # 々 〆 〻 (repeating characters, abbreviations)
r"〇"           # 〇 (Chinese numeral zero)
r"㐀-䶿"    # CJK Extension A (variant characters, names)
r"豈-﫿"    # CJK Compatibility Ideographs (髙 﨑, etc., name variants)
Enter fullscreen mode Exit fullscreen mode

Verified with actual audio:

'少々お待ちください'          → '少々お待ちください'       ✅
'髙橋・﨑山'                 → '髙橋・﨑山'              ✅
'10から20分ほどかかります'     → '10から20分ほどかかります'  ✅
'午後3時30分に開始します'      → '午後3時30分に開始します'   ✅
'受付は9時00分から17時00分です' → '9時0分から17時0分'        ✅
Enter fullscreen mode Exit fullscreen mode

⚠️ There's another pitfall with wave dashes. 〜 (U+301C) and ～ (U+FF5E) are different characters, and which one is used varies by environment. Including only one would miss the other. Both were added.

Also found: Pronunciation dictionary wasn't applied

During troubleshooting, I discovered that the pronunciation dictionary wasn't applied at all to this inquiry site. The dictionary is scoped per project, and all 44 existing entries were tied to different projects.

I tested with the inquiry voice:

Notation Correct Pronunciation Actual Pronunciation
液冷 エキレイ できげ
主な オモナ オーナー
従量課金 ジューリョーカキン 重量価値
行っています オコナッテイマス 言っています

"行っています" becoming "言っています" is frequent in customer service phrases and changes the meaning.

Of the 44 entries, excluding 10 for company product names and personal names, 34 were general terms needed across all sites (technical terms and Japanese words with split kun'yomi/on'yomi). I deployed these to 3 other projects.

Lessons learned

Check the input first. I spent hours eliminating the model, parameters, cache, and pipeline, only to finish by printing the preprocessed output once. The order was backward.

A whitelist decides what to remove, not what to allow. If you list what to remove, unexpected characters pass through. Listing what to allow turns oversights into immediate omissions. For languages with many character types like Japanese, whitelisting everything is difficult.

When it seems like a remove-or-keep choice, there's a third option. Symbols were caught between "removing changes meaning" and "keeping makes unreadable," but there was the option to convert to Japanese. I was stuck thinking preprocessing was "where unnecessary things are removed," not "where meaning is preserved while form is changed."

Check all characters in the same category. When "々" was found missing, I investigated other characters that might be missing for the same reason. Seven were found. If I'd stopped at fixing one, the next report would've been about name variants.

Series: Mass-producing practical voices from diffusion TTS

This is a record of designing voices from a single caption line, creating training corpora, and mass-producing role-specific practical voices. This article is Part 3: Quality Gates.

← Previous: Where did AI's habit of elongating "こんにちわー" come from?

→ Next: The hallucination countermeasure code only worked when there was no hallucination

All 18 articles in the series

  1. The TTS chosen for sound quality was too slow for conversation
  2. Drawing voices like a gacha
  3. Having a machine select "narrator-like voices" from 24 candidates
  4. The stricter the quality gate, the more monotone voices survive
  5. Speaking speed can't be changed after training
  6. TTS that changes "recording location" every time it generates
  7. One rough clip makes the entire style hoarse
  8. Where did AI's habit of elongating "こんにちわー" come from? 9. "少々" becoming "しょも" — The allowlist was cutting Japanese characters ← You are here
  9. The hallucination countermeasure code only worked when there was no hallucination
  10. The "3 characters" allowed by the quality gate became the model's catchphrase
  11. I was discarding candidates over fixable flaws
  12. There are flaws transcription can't find
  13. 70 minutes of training material disappeared in a network blink
  14. How writing "ja" as "JP" created a jargon model
  15. 4 registration paths, 0 management screens
  16. Each deployment was overwriting the other's work
  17. Pushing unmeasured things with thresholds always fails

The insights are summarized in the notebook on mass-producing practical voices from diffusion TTS.

Top comments (0)