DEV Community

orca_forge
orca_forge

Posted on Edited on Originally published at forge.workstyle.tech

Gacha for Voices — One Line of Caption and a Random Seed Bring Back the Same Voice Every Time

#tts #mlops

📝 Originally published (in Japanese) at forge.workstyle.tech.

I'm using a TTS that can generate voices from captions. When given a voice description and a random seed, it speaks in the described voice.

{
  "input": "Hello. Thank you for gathering today.",
  "irodori": {
    "caption": "Calm, intellectual adult woman's voice. Smooth and clear, with an elegant and trustworthy tone.",
    "seed": 1042
  }
}
Enter fullscreen mode Exit fullscreen mode

By keeping the caption fixed and changing the seed, you get a series of similar but slightly different voices. It's like a gacha system.

And the same caption and seed will always produce the same voice. This determinism was the most valuable property in operation.

The concept of "design values" for voices

If it's deterministic, the pair (caption, seed) becomes the identifier for the voice. Instead of storing audio files, you only need to save these two values.

When creating seven characters, I kept a ledger like this:

| key    | Name   | seed | caption |
|--------|--------|------|---------|
| luna   | Luna   | 1042 | A bright, sparkly idol-like young woman's voice. High-pitched and glamorous, speaking cheerfully as if addressing fans. |
| haruto | Haruto | 1042 | A fresh and bright young man's voice. Friendly and clear, with a straightforward tone. |
| mio    | Mio    | 777  | A lively, idol-like girl's voice. Slightly high-pitched and energetic, conveying a smiling tone. |
| shiori | Shiori | 1042 | A calm, intellectual adult woman's voice. Smooth and clear, with an elegant and trustworthy tone. |
| sora   | Sora   | 1042 | A boyish, energetic voice. Neutral and boyish, with a bright and clear tone. |
| gen    | Gen    | 1042 | A deep, mature adult man's voice. Low-pitched and calm, speaking strongly like a narrator. |
Enter fullscreen mode Exit fullscreen mode

The seed 1042 is overused because I reused the first successful seed for other captions. Different captions produce different voices even with the same seed, so it's not a problem.

The day the ledger saved me

The system has a two-stage process: generating audio and then training a lightweight model on it. One day, the driver script for generation was lost.

The trained models remained, but without the design values, I couldn't recreate them. Even a single character difference in the caption would result in a different voice.

Without the ledger, I would have had to redesign the voices for all seven characters from scratch. Instead, I restored the design values from the work logs, transcribed them into the ledger, and recreated the exact same voices.

Approved on 2026-08-06, restored from session history on 2026-08-27 (due to lost generation script).
Fully deterministic with the same caption and seed.
Enter fullscreen mode Exit fullscreen mode

Trained model files are hundreds of MBs and less likely to be backed up. Design values are just a few hundred bytes and can be stored in a text file in git. The restoration cost is vastly different.

What to include in the ledger

I eventually settled on this format:

| key  | Name             | seed   | Style         | caption |
|------|------------------|--------|--------------|---------|
| narF | Female Narrator  | 55555  | narration     | A calm adult female narrator's voice. Speaking slowly with warmth and trust, carefully reading long passages. |
| narM | Male Narrator    | 3407   | narration     | A deep, mature male narrator's voice. Low-pitched and calm, speaking slowly and heavily, carefully reading long passages. |
| cnsF | Female Counselor | 1042   | counseling    | A very gentle and calm female voice. Speaking slowly and softly, reassuring the listener. With soft breathing. |
| salM | Male Sales       | 7      | sales         | A bright and trustworthy male voice. Positive and clear, with a refreshing and non-pushy tone for proposals. |
Enter fullscreen mode Exit fullscreen mode

I added the speaking style (conv_style) later (as speaking speed cannot be changed after training). Even with the same caption and seed, changing the script in the training corpus alters the intonation and speaking speed. So, (caption, seed, style) is necessary to uniquely determine a voice.

For models created before adding this column, I can no longer tell which style they were trained with. Once a required identifier field is missing, it cannot be retroactively filled.

Why the ledger is not machine-readable

The ledger is a Markdown table, and I didn't create a mechanism to parse it. There are three reasons:

  1. Humans read and judge it more often. When searching for "a narrator-like voice," it's a human task, and a readable table is sufficient.
  2. Captions are long. With 50-60 Japanese characters, CSV or JSON becomes hard to read. Escaping issues also arise.
  3. The source of truth is in the DB. Actual jobs are stored in the voice_design_jobs table with name / caption / seed / progress.params.conv_style. When needed programmatically, I refer to this. 
SELECT name, seed, progress->'params'->>'conv_style' AS conv_style, caption
FROM voice_design_jobs WHERE status = 'ready' ORDER BY created_at;
Enter fullscreen mode Exit fullscreen mode

So, the ledger is a human index, and the DB is the machine source of truth. This results in duplicate management, but the ledger only includes "approved design values," so failed experiments don't mix in. The purposes are different.

Verifying determinism in operation

If you trust determinism and create a ledger, you need to confirm it's true. I generated the same input twice and checked if the byte sequences matched.

a = gen("Test sentence.", caption, seed=1042)
b = gen("Test sentence.", caption, seed=1042)
assert a == b      # They matched exactly
Enter fullscreen mode Exit fullscreen mode

Even in cases with hallucinations, they matched.

First time: "Why do you think so? That's right! I thought it resembled a circle."
Second time: "Why do you think so? That's right! I thought it resembled a circle."
Enter fullscreen mode Exit fullscreen mode

Even hallucinations are reproducible. This was useful for debugging, as I could keep "always failing inputs" as fixed assets.

⚠️ However, results change with different model versions. If the TTS server image is updated, the same design values might produce different voices. I add dates to the ledger and re-generate major voices for comparison when making significant updates.

Side effect: Voices can be "expanded"

With design values saved, you can add variations of the same voice for different use cases.

The character "Shiori" is trained for narration, but if I need a call center version, I can keep the caption and seed the same, change the style to support, and train another version. Each takes 1-2 hours.

In fact, one character has two variants: "Streaming (Casual)" and "Business (Polite)." The voice is the same, but the intonation and emotional styles differ.

This works because voice identity (caption+seed) and speaking style are separated. Without separation, I would have to recreate voices for each use case.

Summary

  • If generation is deterministic, design values become voice identifiers. They're much smaller than model files and can be stored in git.
  • Identifier fields cannot be retroactively filled. Models created before adding speaking style are now unidentifiable.
  • Separate human and machine sources of truth. Duplicate management is fine if purposes differ.
  • Verify determinism before operation. Just generate the same input twice and check byte equality.
  • Saved design values allow expanding voices. Adding variations for different use cases becomes feasible.

Series: Mass-producing practical voices from diffusion TTS

This is a record of designing voices from single-line captions, generating training corpora, and mass-producing role-specific practical voices. This article is Part 1: Design.

← Previous: The TTS chosen for quality was too slow for conversation

→ Next: Letting a machine choose "narrator-like voices" from 24 candidates

All 18 parts in the series

  1. The TTS chosen for quality was too slow for conversation 2. Voice gacha and the design ledger ← You are here
  2. Letting a machine choose "narrator-like voices" from 24 candidates
  3. The stricter the quality gate, the more flat takes survive
  4. Speaking speed cannot be changed after training
  5. The TTS that changes "recording rooms" every generation
  6. One rough clip ruins the whole style
  7. Where did the AI's habit of elongating endings come from?
  8. "Shomō" instead of "shoyō" — The allowed character list was trimming Japanese
  9. The hallucination prevention code only worked when there were no hallucinations
  10. The "3 characters" allowed by the quality gate became the model's catchphrase
  11. I was discarding candidates over fixable flaws
  12. Some flaws aren't found in transcripts
  13. 70 minutes of training material vanished in a network blink
  14. From "ja" to "JP": Creating a jargon model
  15. 4 registration paths, 0 management screens
  16. Each deploy erased the other's work
  17. Thresholding what you can't measure always fails

The insights are summarized in the notebook: Mass-producing practical voices from diffusion TTS manufacturing pipeline.

Top comments (0)