The Tighter the Quality Gate, the More the Flat Reads Survive — Selection Bias from Verification

📝 Originally published (in Japanese) at forge.workstyle.tech.

To create a voice synthesis model capable of expressing emotions, I was automatically generating a training corpus. For each of the 12 emotions, such as joy, sadness, anger, and fear, I prepared several audio clips with emotional intonation. Quality checks were, of course, included. I transcribed the generated audio using Whisper, compared it with the script, and only adopted the ones that were read correctly.

The resulting model, however, was remarkably monotone.

The cause was the quality check itself.

Switching Emotional Styles Still Sounds the Same

This model has a style for each emotion. You can switch between joy style, sadness style, and synthesize accordingly. However, switching styles barely changes the impression of the voice.

Looking at the numbers, it was clear. I measured the cosine similarity between the audio synthesized in each emotional style and the neutral style. If emotions were conveyed, the value should decrease, meaning it deviates from the neutral style.

Bulk generation recipe corpus: cos 0.77–0.94
Good individual from a different system: cos 0.164

A value close to 0.9 means that even when synthesized in joy style, it sounds almost the same as the neutral style. The emotional style feature was essentially dead.

Honestly, when I first listened to it, I thought, "Well, it's not bad," and almost let it slide. It was only when I saw the 0.9 value that I was convinced something was wrong.

There Were Three Causes

Two of them were configuration issues, and the third is the main topic.

Exclamations Sound Like a Different Speaker Due to CFG Settings

In scripts starting with exclamations like "Wow!" or "Huh!", only the beginning sounded like a different speaker. The parameter controlling fidelity to the reference audio couldn't maintain the speaker's identity when emphasizing emotions.

Emojis Are Read Aloud

When I included emojis in the script as emotional markers, they were either read aloud or triggered strange sound effects. Markers should have been placed outside the text.

Whisper Validation Favors Flat Takes

This was the main issue.

Validation Tends to Reject Emotionally Charged Audio

Multiple candidate takes are generated, and those that pass Whisper validation are adopted. What happens here?

Emotionally charged audio tends to have trembling voices (fear), elongated or bouncing endings (joy), distorted volume (anger), and fading endings (sadness). These changes make it harder for Whisper to recognize, causing the transcription to deviate from the script and fail validation.

On the other hand, monotone takes without emotion are clear and easy to recognize, so they pass smoothly.

In other words, selecting based on "whether it reads the script correctly" means emotionally flat takes are more likely to survive. The stricter the gate, the stronger this bias becomes. The better the gate functions, the flatter the corpus becomes.

The goal (training on emotionally rich voices) and the means (selecting based on script fidelity) were in direct conflict. And since each part was functioning normally, no errors were thrown.

Separate Filtering and Ranking

The solution was to make the selection a two-step process.

# Before: Adopt the first one that passes
for seed in seeds:
    wav = gen(text, ref, seed)
    if judge(text, whisper(wav)).ok:
        return wav                      # ← The first to pass = the flattest take

# After: Collect all passes, then select the most emotionally charged one
candidates = []
for seed in seeds:
    wav = gen(text, ref, seed)
    if judge(text, whisper(wav)).ok:            # ① Filtering
        candidates.append((wav, style_distance(wav, neutral_ref)))
if not candidates:
    return None
return max(candidates, key=lambda c: c[1])[0]   # ② Farthest from neutral

① is a quality filter and isn't used for ranking. The rank is determined by ②.

Measuring Style Distance

Whether an emotion is conveyed is measured by the style embedding distance from the same speaker's voice synthesized in a neutral style. The TTS used can extract style vectors from audio, so I utilized that.

def style_distance(wav, neutral_wav):
    a = extract_style_vector(wav)          # Use the TTS's embedding extraction API
    b = extract_style_vector(neutral_wav)
    cos = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
    return 1.0 - cos                       # Larger value means farther

If embedding extraction isn't available, a combination of F0 median, range (semitones), and RMS variance can be used as a substitute. The idea of measuring deviation from the neutral take remains the same.

The key is to look at the relative distance from neutral, not absolute values. Creating absolute criteria like "joy style should be bright" would require adjusting thresholds for each speaker, but relative distance is speaker-independent.

Don't Use Simple Thresholds for Acceptance

I made the acceptance criteria a combination of two conditions.

def accept(text, transcript):
    v    = judge_transcript(text, transcript)
    tail = trailing_elongation_mismatch(text, transcript)   # Unscripted ending elongation
    return (v.ratio >= 0.82 and tail <= 2) or (v.ratio >= 0.70 and tail == 0)

Even if the match ratio is low, it passes if there's no ending elongation. This allows for some recognition degradation due to emotional expression while strictly checking for unscripted elongation.

The reason for being strict about the latter is clear: the model learns to elongate endings even when not written. In fact, an early model started elongating "こんにちは" (hello) to "こんにちわー", caused by elongated clips in the corpus (see Where Did the AI's Habit of Elongating "こんにちわー" Come From?).

⚠️ This ending check must be done on Whisper's raw transcription. Kana normalization drops long vowel marks, so comparing normalized strings won't detect elongation. Different strings are input for match ratio and ending checks.

Emotional Anchor Method

Another layer of ingenuity was needed: creating anchors for each emotion beforehand.

1. For each emotion, generate multiple "emotionally strong sentences" outside the script with different seeds.
2. Select the one with the farthest style distance from the passes → Register as anchor.
3. Generate corpus sentences using the anchor as the reference audio.
4. For sentence selection, choose the one with the farthest style distance from the passes.

Here's the code implementation:

# Stage 1: Confirm emotional anchors
ANCHOR_TEXTS = {
    "joy":     "やった、ついにできましたね！本当に、本当に嬉しいです！",
    "fear":    "怖い、怖いです。どうしよう。",
    "sadness": "もう、どうにもならないんです……。",
    ...
}
anchors = {}
for emo, text in ANCHOR_TEXTS.items():
    cands = []
    for seed in ANCHOR_SEEDS:                       # Multiple seeds
        wav = gen(text, base_ref, seed)             # Speaker ensured by long reference
        if accept(text, whisper(wav)):
            cands.append((wav, style_distance(wav, neutral)))
    anchors[emo] = max(cands, key=lambda c: c[1])[0]   # Most emotionally charged
    register_voice(f"anchor_{emo}", anchors[emo])

# Stage 2: Generate sentences using anchors as reference
for emo, lines in CORPUS.items():
    for line in lines:
        cands = [w for w in (gen(line, anchors[emo], s) for s in SEEDS)
                 if accept(line, whisper(w))]
        clip = max(cands, key=lambda w: style_distance(w, neutral))
        save(clip, line, group=emo)

Essentially, I first confirm a sample of "how this voice speaks in this emotion" before mass production. The bulk generation failed because it tried to produce both emotional expression and speaker identity in one go. With anchors, emotion is inherited from the reference's prosody, and speaker identity is ensured by the anchor itself.

Choosing anchor sentences from outside the script is also key. Using corpus sentences as anchors would create an imbalanced dataset with only those sentences having extreme emotional expression.

Anchor method: cos 0.45–0.61
Bulk generation: cos around 0.8

Pitfalls Encountered

Trembling emotions require different anchor creation methods. For emotions like fear where the voice trembles, higher fidelity to the reference is needed to prevent the voice from sounding like a different person mid-sentence. Settings need to be adjusted for each emotion type. Increasing fidelity weakens emotional expression, so you must decide whether to prioritize speaker identity or expression for each emotion.

Stuttering consonants in the script sound unnatural. Writings like "だ、誰ですか" (W-who is it?) or "や、やだ" (Y-yuck) are rendered unnaturally by TTS. Rewriting them as word repetitions, like "怖い、怖いです" (Scary, scary it is), produced more natural results. This requires thinking opposite to human acting guidance.

Reference audio has a length limit. Up to 55 seconds works, but 110 seconds triggers a GPU assertion error. Also, memory fragmentation occurs with sequential generation, so even 55 seconds fails on small GPU slices. For long references, set memory allocator settings (PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True) and secure ample GPU slices.

Filtering and Ranking with the Same Metric Causes Bias

I think this failure pattern isn't limited to voice synthesis.

• In code generation, selecting based solely on "whether tests pass" favors simple implementations.
• In summarization, selecting based on "match ratio with the original" favors extractive summaries.
• In image generation, selecting based on "prompt fidelity" favors mundane compositions.

All these are correct as quality checks but wrong as selection criteria. Checks should be limited to filtering, and ranking should be based on a separate metric directly tied to the goal.

And the tricky part is that this bias doesn't throw errors. All parts function normally, producing a normally flat dataset. To notice, you must check the final product, think "something's off," and trace back.

The mundane conclusion is that automated pipelines need a final inspection step. But in this case, even listening to the product almost made me overlook it, so intuition alone wasn't enough. The decisive factor was the numerical value of "0.9 similarity to neutral." Both intuition and numbers are needed, is my takeaway from this experience.

Series: Mass-Producing Practical Voices from Diffusion TTS

This is a record of designing voices from a single caption, manufacturing training corpora, and mass-producing role-specific practical voices. This article is Part 2: Manufacturing.

← Previous: Having a Machine Select "Narrator-like Voices" from 24 Candidates

→ Next: Speaking Speed Can't Be Changed After Training

All 18 Parts in the Series

1. The TTS Chosen for Sound Quality Was Too Slow for Conversation
2. Voice Gacha
3. Having a Machine Select "Narrator-like Voices" from 24 Candidates 4. The Stricter the Quality Gate, the More Monotone Takes Survive ← You are here
4. Speaking Speed Can't Be Changed After Training
5. TTS That Changes "Recording Location" Every Generation
6. One Rough Clip Ruins the Entire Style
7. Where Did the AI's Habit of Elongating "こんにちわー" Come From?
8. "少々" Becoming "しょも" — Permitted Character List Was Erasing Japanese
9. Hallucination Countermeasure Code Only Ran When There Was No Hallucination
10. Quality Gate Allowed "3 Characters" That Became the Model's Catchphrase
11. Was Rejecting Candidates Over Fixable Flaws
12. There Are Flaws Transcription Can't Catch
13. 70 Minutes of Training Material Vanished in a Network Blink
14. From "ja" to "JP": Creating a Jargon Model
15. 4 Registration Paths, 0 Management Screens
16. Deployments Kept Overwriting Each Other's Work
17. If You Chase What You Can't Measure with Thresholds, You'll Always Fail

The insights are compiled in the notes on Mass-Producing Practical Voices from Diffusion TTS Manufacturing Pipeline.