DEV Community

orca_forge
orca_forge

Posted on Originally published at forge.workstyle.tech

Where Did the AI Learn to Stretch Its Greetings Like 'Kon-nichiwaa'?

#tts #whisper

πŸ“ Originally published (in Japanese) at forge.workstyle.tech.

When I had the trained voice model read "こんにけは" (Hello), it elongated the phrase to "こんにけわぁ." The script didn't include any instructions for elongation.

The feedback was as follows:

For "こんにけは," it's pronounced as "こんにけわぁ" with an accent at the end. It feels like something got mixed in.

"Something got mixed in" was accurate, and indeed, something had been mixed in. The training corpus contained a clip with an elongated ending.

The issue was that the mechanism to detect it was fundamentally non-functional by design.

Script Matching Was in Place

In corpus generation, the TTS-read audio is transcribed using Whisper and then compared against the script.

def _kana(s: str) -> str:
    # Katakana to Hiragana conversion
    return "".join(chr(ord(c) - 0x60) if "γ‚‘" <= c <= "γƒΆ" else c for c in s)

_PUNCT_RE  = re.compile(r"[γ€γ€‚οΌοΌŸ!?…・\sγ€Œγ€γƒΌγ€œ,\.]")
_REPEAT_RE = re.compile(r"(.)\1+")

def _collapse(s):
    return _REPEAT_RE.sub(r"\1", _PUNCT_RE.sub("", s or ""))

def judge_transcript(script_text, transcript, ...):
    a = _kana(_collapse(script_text))
    b = _kana(_collapse(transcript))
    sm = difflib.SequenceMatcher(None, a, b)
    ...
Enter fullscreen mode Exit fullscreen mode

The comparison is done after normalization. It's a straightforward implementation.

Take a look at _PUNCT_RE here. Among the characters to be removed is γƒΌ (the long vowel mark). And _REPEAT_RE compresses consecutive identical characters into one.

Script: こんにけは
Transcription: こんにけわー

After normalization:
  Script β†’ こんにけは
  Transcription β†’ こんにけわ      ← The "γƒΌ" is removed
Enter fullscreen mode Exit fullscreen mode

The match rate is high. It becomes a difference of just one character between "は" and "わ." The information that the ending was elongated is discarded during the normalization step.

The same thing happens with consecutive vowels.

Transcription: こんにけわあ  β†’  _REPEAT_RE compresses consecutive "あ" β†’  こんにけわ
Enter fullscreen mode Exit fullscreen mode

This means this verification cannot detect elongated endings no matter what. The normalization written to ignore long vowels worked the same way even when we wanted to detect them.

The normalization itself is correct. If you want to absorb variations in notation and check for content match, long vowels should be removed. The problem was that there was only one normalization for two purposes.

Judging with Raw Transcription

I separated the judgment for content match from the judgment for elongated endings. The latter uses the raw string before normalization.

_TAIL_LONG_RE = re.compile(r"[ーぁ-γ‚“]$")

def trailing_elongation_mismatch(script_text: str, raw_transcript: str) -> bool:
    """Detects elongated endings not present in the script.

    ⚠️ Pass the raw transcription from Whisper to raw_transcript.
    Kana normalization discards long vowels, so it cannot be detected with the normalized string.
    """
    script = (script_text or "").rstrip("γ€‚γ€οΌοΌŸ!? ")
    trans  = (raw_transcript or "").rstrip("γ€‚γ€οΌοΌŸ!? ")
    if not script or not trans:
        return False

    # Check if the last character of the script is "elongated" in the transcription
    tail_script = script[-1]
    # Long vowel mark is present in the transcription but not in the script
    if "γƒΌ" not in script and trans.endswith("γƒΌ"):
        return True
    # Consecutive identical vowels are present only in the transcription (e.g., "です" β†’ "ですぅ," "ですう")
    if len(trans) > len(script) and trans[len(script)-1:].startswith(tail_script):
        extra = trans[len(script):]
        if extra and all(c in "γγƒγ…γ‡γ‰γ‚γ„γ†γˆγŠγƒΌ" for c in extra):
            return True
    return False
Enter fullscreen mode Exit fullscreen mode

Since the judgments are separated, the caller checks them separately.

res  = judge_transcript(text, tr["text"])                  # Content match (with normalization)
tail = trailing_elongation_mismatch(text, tr["text"])      # Elongated ending (raw string)

if tail:
    continue          # If the ending is elongated, immediately redraw (even if the content matches, it's not accepted)
if res.ok:
    save(wav)
Enter fullscreen mode Exit fullscreen mode

Elongated endings are disqualified even if the content matches. No matter how high the content match rate is, if the ending is elongated, it's not included in the training material. Relaxing this would lead to the habit being ingrained.

Designing Tolerance

However, setting it to zero completely would reduce yield. In emotionally charged speech, some elongation occurs naturally.

Ultimately, I used this two-condition OR logic:

def accept(text, transcript):
    v    = judge_transcript(text, transcript)
    tail = trailing_elongation_mismatch(text, transcript)
    return (v.ratio >= 0.82 and tail <= 2) or (v.ratio >= 0.70 and tail == 0)
Enter fullscreen mode Exit fullscreen mode
  • If the content matches well (0.82 or higher), allow up to two elongated endings.
  • If the content match is somewhat low (0.70 or higher), no elongated endings are allowed.

This ensures that clips with "suspicious content and elongated endings" are reliably discarded. It tolerates recognition degradation due to emotional expression while not allowing the elongation habit to pass.

Hitting from the Caption Side

Another effective measure was including it in the caption during generation.

When designing role-specific voices, I added this to the caption:

A female announcer's voice accurately reading a news script. Clear and easy to understand,
with a calm and intellectual tone, pronouncing each word distinctly, including the endings.
Enter fullscreen mode Exit fullscreen mode

The last part, "pronouncing each word distinctly, including the endings," is the key.

The effect was clear. After generating 24 candidates Γ— 5 probe sentences = 120 clips and measuring, there were zero elongated endings. Before the quality gate could reject them, elongated speech simply wasn't generated.

Role-included caption (distinct endings) β†’ Elongated endings 0/120
Enter fullscreen mode Exit fullscreen mode

The gate is a mechanism to "discard bad ones," but discarding reduces yield. If you can prevent it from being generated upstream, that's cheaper. For TTS that can be instructed via captions, it's worth including the items the quality gate checks in the caption.

Relationship with Speech Style

As I investigated this phenomenon, a deeper structure emerged. The desirability of elongated endings varies by use case.

  • Narrator, Announcer, Call Center β†’ Prefer tight endings.
  • VTuber, Streaming β†’ Elongated endings feel more natural.

So, I defined "speech styles" for each use case and varied the corpus script and quality gate strictness by speech style. For business-style speech, the ending gate is strictly applied, while for casual styles, it's relaxed.

What's important is that speech style is baked into the corpus and cannot be changed during synthesis (as discussed in Speaking Style is Baked into the Corpus). If you want both "tight endings" and "elongated endings" for the same voice, you need to train two models with the same design values (caption + seed) but different speech styles. That's exactly what I did.

Summary

Separate normalization by purpose. Normalization for content matching and normalization for detecting specific anomalies are different. Trying to do both with one normalization causes one of them to fail.

Be aware of discarded information. Including γƒΌ in _PUNCT_RE was the right decision, but a judgment that needed that information arose later. Adding a comment in the normalization code about "what is being discarded" helps the next person notice.

It's cheaper to prevent it upstream. Discarding at the gate reduces yield. If you can instruct the generation side, hit it there.

Symptoms appear in the model's behavior. Even if you look at the corpus data, it just shows "a few clips with elongated endings," which doesn't seem abnormal. The habit only appears after training and speaking. Inspecting the dataset alone is not enough; you need a process to check the output after training.

Series: Mass-Producing Practical Voices from Diffusion TTS

This is a record of designing voices from a single caption, manufacturing training corpora, and mass-producing role-specific practical voices. This article is Part 3: Quality Gate.

← Previous: One Rough Clip Ruins the Whole Style

β†’ Next: The Character That Broke the TTS Input

All 18 Articles in the Series

  1. The TTS Chosen for Quality Was Too Slow for Conversation
  2. Voice Gacha
  3. Having a Machine Select "Narrator-like Voices" from 24 Candidates
  4. The Stricter the Quality Gate, the More Monotone Takes Survive
  5. Speaking Style is Baked into the Corpus
  6. TTS That Changes the "Recording Room" Every Time It Generates
  7. One Rough Clip Ruins the Whole Style 8. Where Did the AI's Habit of Elongating "こんにけわー" Come From? ← You are here
  8. The Character That Broke the TTS Input
  9. The Code to Counter Hallucinations Only Worked When There Was No Hallucination
  10. The "Three Characters" Allowed by the Quality Gate Became the Model's Quirk
  11. Discarding Candidates Over Fixable Flaws
  12. There Are Flaws That Can't Be Found in Transcriptions
  13. 70 Minutes of Training Material Vanished in a Network Blink
  14. From "ja" to "JP": Creating a Jargon Model
  15. Four Registration Paths, Zero Management Screens
  16. Deployments Kept Overwriting Each Other's Work
  17. If You Chase What You Can't Measure with Thresholds, You'll Always Fail

The insights are compiled in the notebook Mass-Producing Practical Voices from Diffusion TTS Manufacturing Pipeline.

Top comments (0)