TTS that Changes 'Recording Location' Each Time It's Generated — Inconsistent Audio Quality Ruins Style

📝 Originally published (in Japanese) at forge.workstyle.tech.

We're generating training corpora for voice models using a different TTS system. Same speaker settings, same model, same server. Yet, each clip has a different sound texture.

Specifically, I noticed this when switching between emotional styles. When switching from a joyful style to a sad one, not only does the tone of voice change, but the sound quality itself shifts. One sounds like it was recorded close by, while the other seems slightly distant. It's like the same speaker is talking from different rooms, creating a sense of dissonance.

The culprit was the training material. Each clip in the corpus had different frequency characteristics depending on the generation conditions.

Why does the texture change?

This corpus is generated using different reference audio (anchors) for each emotion (as discussed in "The stricter the quality gate, the more monotone takes survive").

joy text → generated using joy anchor
sadness text → generated using sadness anchor
anger text → generated using anger anchor

Since the reference audio differs, the TTS not only mimics the tone of voice but also the acoustic characteristics of the reference. As each anchor is generated separately, they have slightly different spectral shapes. This results in a mismatch in sound texture between emotional groups.

Furthermore, extreme acting materials (screaming, laughing hysterically, etc.) inherently have different sound pressure ranges. When trained together, the model learns not only the style but also "this style sounds distant."

This is a problem that doesn't occur with human recordings, where the same microphone and room are used. Generated audio lacks the concept of a recording environment, so channel characteristics change with conditions.

Matching LTAS

The solution was to align all clips to a common frequency characteristic before training.

LTAS (Long-Term Average Spectrum) represents the average frequency distribution of the entire audio, reflecting speaker and room characteristics. We use this as our reference.

Here's the process:

1. Calculate the LTAS from the entire corpus or a set of reference clips to create a reference spectrum.
2. Calculate the LTAS for each clip and compute the difference from the reference.
3. Apply an EQ to each clip to cancel out this difference.
4. Finally, match the RMS.

def match_ltas(wav, ref_ltas, max_gain_db=10.0):
    """Align clip LTAS to reference (±max_gain_db clipping)"""
    spec = stft(wav)
    ltas = np.mean(np.abs(spec), axis=1)              # Average over time

    gain_db = 20 * np.log10((ref_ltas + eps) / (ltas + eps))
    gain_db = np.clip(gain_db, -max_gain_db, max_gain_db)   # Prevent extreme corrections
    gain    = 10 ** (gain_db / 20.0)

    spec = spec * gain[:, None]
    return istft(spec)

Why clip at ±10dB

We set a limit on the correction amount. Without limits, it would forcefully boost bands that don't exist in the original clip. Since there's no signal in those bands, only noise would be amplified.

Especially in the high frequencies, some generated audio clips have almost nothing above 8kHz. Trying to match the reference in these cases would introduce a hissing noise. By limiting to ±10dB, even if it doesn't perfectly align, it won't break.

Zero-phase processing

When applying EQ, we ensure phase is not altered. Regular filters introduce group delay, causing time shifts across bands. While subtle to the human ear, it's undesirable for training material.

We either manipulate only the amplitude of the STFT while keeping the phase intact or apply the filter in both forward and backward directions to cancel phase rotation (equivalent to filtfilt). We chose the former approach.

Matching RMS last

After aligning frequency characteristics, we match loudness.

def normalize_rms(wav, target_dbfs=-20.0):
    rms = np.sqrt(np.mean(wav ** 2))
    gain = 10 ** (target_dbfs / 20.0) / (rms + 1e-9)
    return np.clip(wav * gain, -1.0, 1.0)

We target -20dBFS. The key is to match RMS, not peak. Peak normalization would be influenced by single loud sounds (like the start of a scream), causing overall volume inconsistencies.

Effects

The sound quality jumps when switching emotional styles disappeared. This was the primary goal.

There was also an unexpected side effect: extreme acting material became trainable.

Before normalization, training with screaming clips tended to destabilize the model. The sound pressure and spectrum were so different that the model struggled to treat them as part of the same style.

After normalization, the temporal structure of screams (rapid pitch sweeps) became trainable. By removing differences in sound pressure and spectrum, the remaining "movement" information reached the model.

In other words, normalization, which seems like "discarding information by aligning," actually discards what should be discarded (channel characteristics) and highlights what should be preserved (prosodic movement).

How much to align?

Overdoing it causes other problems.

Risk of losing speaker characteristics. LTAS includes vocal tract characteristics, so strong alignment can dilute voice individuality. In this case, since we're aligning within a single speaker's corpus, it's not an issue, but caution is needed when mixing multiple speakers.

Emotional acoustic features also appear in frequency. Anger tends to have stronger high frequencies, while sadness has weaker ones. Complete alignment would erase this. The ±10dB limit helps here too, as it prevents strong corrections, preserving some emotion-related differences.

Choosing the reference requires judgment. We used the corpus average, but selecting "the best clips" as the reference is another approach. With an average, if there are many poor clips, the reference itself can be compromised.

Implementation notes

Process all clips before training. Normalizing one by one during generation means processing before the reference spectrum is finalized. Wait until all clips are ready, then process them together.

Keep original clips. Make normalization reversible. You might want to retrain with different parameters (max dB, target RMS).

Listen to normalized audio. Even if numbers align, there might be added noise or unnaturalness. Especially check clips where correction hits the limit.

# Record clips where correction hits the limit
clipped = np.sum(np.abs(gain_db_raw) >= max_gain_db) / len(gain_db_raw)
if clipped > 0.3:
    logger.warning(f"{clip_id}: Over 30% of corrections hit the limit (original material is far from reference)")

If many corrections hit the limit, the material might be too different from the rest, and excluding it might be better than forcing alignment.

Summary

• Generated audio lacks a "recording environment," so channel characteristics change with conditions. This doesn't happen with human recordings.
• Using different references for emotions changes sound quality for each emotion. This appears as jumps when switching styles.
• Align LTAS and RMS. Limit corrections to ±10dB and preserve phase.
• Alignment highlights what should be preserved. Extreme acting temporal structures became trainable.
• Don't over-align. Speaker characteristics and emotion-related frequency differences can be lost. Limits protect quality.

Series: Mass-producing practical voices from diffusion TTS

This series documents designing voices from single-line captions, manufacturing training corpora, and mass-producing role-specific practical voices. This article is Part 2: Manufacturing.

← Previous: Speaking speed can't be changed after training

→ Next: One rough clip ruins the whole style

All 18 parts

1. The TTS chosen for sound quality was too slow for conversation
2. Voice gacha
3. Letting a machine choose "narrator-like voices" from 24 candidates
4. The stricter the quality gate, the more monotone takes survive
5. Speaking speed can't be changed after training 6. TTS changes "recording room" every generation ← You are here
6. One rough clip ruins the whole style
7. Where did the elongated endings come from?
8. "Shomō" instead of "shoshō" — permitted character list was erasing Japanese
9. Hallucination countermeasure code only ran when there was no hallucination
10. The "3 characters" allowed by the quality gate became the model's catchphrase
11. We were discarding candidates over fixable flaws
12. Some flaws can't be found in transcripts
13. 70 minutes of training material vanished in a network blink
14. From "ja" to "JP": Creating a jargon model
15. 4 registration paths, 0 management screens
16. Each deployment erased the other's work
17. If you chase what you can't measure with thresholds, you'll always fail

The insights are compiled in Mass-producing practical voices from diffusion TTS: Manufacturing pipeline.