TTS Chosen for Audio Quality Was Too Slow for Conversations — Real-World Tests Show 2.5x Speed Difference Between 'Design' and 'Production'

📝 Originally published (in Japanese) at forge.workstyle.tech.

I found a diffusion TTS that can generate voices just by writing captions.

"A calm, adult female narrator's voice. With a relaxed tone, it conveys warmth and trust, carefully reading long passages."

When given this description, it speaks exactly as described. You can generate any voice in Japanese without preparing even a second of the speaker's audio data. Moreover, the same caption and random seed will always produce the same voice.

I thought about using it for interactive avatars. This would eliminate the need to prepare a voice for each character.

To get straight to the point, it couldn’t be used for conversations. However, it was too valuable to discard, so I kept it by changing its role.

Real-world measurement: 2.5 times slower on the same GPU

The existing system used a pre-trained TTS (Style-Bert-VITS2-based). I compared synthesizing the same sentence (7.5 seconds long) on the same GPU slice.

RTF (Real-Time Factor) is calculated as "generation time ÷ audio length." The smaller the value, the faster the synthesis.

At 40 steps, it’s 2.5 times slower. Even reducing the steps to 16 still leaves a 1.5 times difference. And below 16 steps, the quality collapses (6 steps were immediately rejected upon listening).

With smaller GPU slices, the gap widens further.

In conversations, each sentence is synthesized and played sequentially, so the time until the first sentence is output determines the user experience. A 1–2 second delay per turn is too much.

Increasing GPU resources didn’t solve the problem

I considered allocating a larger GPU and calculated the impact. Going from 2g.20gb to 7g.80gb would provide four times the resources.

However, measurements revealed a fixed overhead of approximately 1.1 seconds. Even reducing the steps to 12 only lowered the time to 1.1–1.4 seconds, the same as 16 steps. This includes model loading and text processing, which cannot be reduced by GPU parallelism.

Thus, even with four times the resources, the generation time would bottom out at around 1.0–1.2 seconds, still short of the pre-trained model’s 0.8 seconds. Given the cost would be several times higher, I decided not to adopt it.

This approach of measuring fixed overhead first is always useful when deciding on GPU upgrades. Reducing the steps drastically to find the point where further reduction doesn’t speed things up reveals the size of the non-parallelizable portion.

Dividing roles

While it’s slower in terms of speed, diffusion TTS can do something unique: generate voices from captions. Pre-trained models can only produce voices of speakers used in training.

So, I divided the roles like this:

[Design Phase] Diffusion TTS ── Generate voice from caption + seed ── Training corpus (approx. 200 samples)
                                                        ↓
                                                      Train
                                                        ↓
[Runtime] Pre-trained Model ──────────────── Voice users actually hear

Diffusion TTS creates the desired voice, which is then used to generate about 200 samples for training material. A lightweight model trained on this material is used for actual speech.

Since diffusion TTS only runs during voice creation, its slowness isn’t an issue. Generating one voice takes about 70 minutes, but this is a batch process.

Conditions for this division to work

Generation must be deterministic. The same caption and seed must always produce the same voice. Without this, the designed voice cannot be reproduced, and the corpus would need to be recreated each time, resulting in a different voice.

In fact, thanks to this property, only the design ledger is needed to reproduce the model. Even if the pre-trained model file is lost, the same voice can be recreated using the caption and seed. I once lost the driver script but was able to recreate the exact voice by restoring the caption and seed from records (see Voice Gacha).

Audio quality must be suitable for training. Since the generated audio is used as training material, the quality of the diffusion TTS sets the upper limit. This can only be confirmed by actually training and listening. In this case, there were no issues.

Voice consistency must be maintained. The same voice must be maintained throughout the generation of 200 samples. This was challenging. For example, sentences starting with exclamations sounded like a different person at the beginning, and stronger emotional expressions disrupted the speaker’s identity. This was resolved by adjusting the use of reference audio, but that’s another story.

Side effect: Captions also determine prosody

Dividing roles revealed a constraint: captions determine not only the timbre but also the speech rate and intonation.

When creating a male counselor’s voice, I wrote this caption:

A calm and gentle male voice. Low and slow, speaking in a reassuring, settled manner.

The resulting voice was evaluated as "too low and too slow," so I removed "low" and "slow":

A calm and gentle male voice. Speaking in a settled, medium-pitched tone, addressing the listener carefully and naturally to reassure them.

The F0 range increased to 136–159 Hz (originally 105–159 Hz), as intended. However, the speech rate jumped from 5.5–6.7 to 7.3–7.7.

Even adding "slightly slower" only reduced it to 6.4–7.1. F0 and speech rate are linked within the caption, making it impossible to adjust one without affecting the other.

After trying five patterns, it became clear that separating these two is difficult with captions alone. Increasing the pitch always increases the tempo.

And speech rate cannot be changed with synthesis parameters (see Speaking Style is Baked into the Corpus for measurements). This means the combination of pitch and tempo is decided when writing the caption and cannot be adjusted later.

I adopted a workflow of finalizing the design upfront and creating multiple candidates for comparison. Generating one candidate takes a few minutes, so creating a comparison table with 5 patterns × 12 seeds was feasible.

Summary

• Measure RTF on the same GPU and text. Engine specifications vary, so they’re not comparable.
• Measure fixed overhead first. Drastically reducing steps to find the lower limit reveals the non-parallelizable portion.
• If it’s slower but does something unique, divide roles. "Once during design" and "every time at runtime" have vastly different acceptable delays.
• Deterministic generation is valuable in itself. It allows model reproduction from a design ledger.
• Confirm what can and cannot be adjusted later through measurements. In this case, speech rate was "adjustable but unchangeable."

Series: Mass-producing Practical Voices from Diffusion TTS

This is a record of designing voices from a single caption, creating training corpora, and mass-producing practical voices for specific roles. This article is Part 1: Design.

(This article is the series entry point)
→ Next: Voice Gacha

All 18 articles in the series

1. The TTS chosen for quality was too slow for conversation ← You are here

1. Voice Gacha
2. Letting a machine select "narrator-like voices" from 24 candidates
3. The stricter the quality gate, the more monotone voices survive
4. Speaking rate cannot be changed after training
5. TTS that changes the "recording room" every time it generates
6. One rough clip makes the entire style hoarse
7. Where did the AI’s habit of elongating endings come from?
8. "Shomō" instead of "shōsho" — The allowed character list was cutting Japanese
9. Hallucination countermeasure code only worked when there was no hallucination
10. A "three-character" phrase allowed by the quality gate became the model’s catchphrase
11. Candidates were rejected for fixable flaws
12. There are flaws transcription cannot detect
13. 70 minutes of training material disappeared in a network blink
14. From "ja" to "JP": Creating a jargon model
15. Four registration paths, zero management screens
16. Deployments kept overwriting each other’s work
17. Chasing unmeasured metrics with thresholds always fails

The insights are summarized in the notebook Mass-producing Practical Voices from Diffusion TTS.