Originally published on hexisteme notes.
While evaluating Qwen3-TTS for narration, I ran a small A/B to see whether an instruction could raise the speaking rate. It couldn't, I wrote that down, and speed control moved to post-processing. The next day a different experiment turned up a line of code that made half of that A/B meaningless: on the smaller model, the instruction never reached the model. The call returned normal audio, the transcript was perfect, and nothing in the measurements looked wrong.
Before comparing settings of a control knob, show that the knob moves the output at all. Otherwise the comparison measures nothing.
The A/B
The goal was fast narration, around 230 words per minute. The setup was Qwen3-TTS's CustomVoice models through the qwen_tts 0.1.1 package on torch with the MPS backend, at two sizes: 0.6B and 1.7B. generate_custom_voice has no speed or rate parameter. Its only lever is instruct, a free-text instruction. I tried two instructions on both model sizes, one asking for roughly 230 wpm and one asking for auctioneer-fast delivery.
Rate was measured over the voiced part of an 18-word sentence. The 0.6B model came in at 139.1 and 131.8 wpm for the two instructions, and the 1.7B at 140.6 and 126.3. The matched no-instruction runs on the same sentence came in at 129.9 wpm for the 0.6B and 128.3 for the 1.7B. The decision record summarised this as a gain of 0–8% over a 126–130 baseline; against the matched baselines, the largest change was 140.6 against 128.3, or +9.6%. Word error rate stayed at 0%. The decision record concluded that instruction could not get to 230 wpm and pointed to a post-processing time-stretch or the other engine's speed setting instead.
The same record chose the 0.6B as the default model. Its peak memory footprint of 3.22 GiB fit a rule requiring 4 GiB of available memory, and the 1.7B's 5.6 GiB did not.
The line that discarded it
A voice-cloning smoke test the next day recorded this as a side finding. In qwen_tts 0.1.1, generate_custom_voice contains:
if self.model.tts_model_size in "0b6": # for 0b6 model, instruct is not supported
instruct = None
The 0.6B CustomVoice checkpoint's config carries the size tag 0b6. For that model the instruction becomes None before tokenisation and generation proceeds as if none had been passed. The 1.7B checkpoint carries a different tag, so its instructions went through.
Nothing in the installed 0.1.1 package announces the drop. The branch raises no exception and emits no warning or log line. The wrapper file that holds the branch has no logging calls at all, and the two warning_once calls in the model file concern gradient checkpointing. The docstring describes instruct as optional, "If None, treated as empty (no instruction)", and says nothing about model size. Passing an instruction to the 0.6B model is accepted, has no effect, and returns perfectly good audio.
So the two 0.6B "instructed" runs were really two more runs without an instruction. One of them landed at 139.1 wpm, above its matched no-instruction run at 129.9, with no instruction behind it.
The pattern matches Why My Correct Config Value Was Being Ignored: a correct value set in one place gets overruled before it reaches the code that uses it, and nothing throws.
What the A/B still supports
The 0.6B half did not test the instruction's effect. The other half did receive the instruction, and the 1.7B runs reached 140.6 and 126.3 wpm. That pair still supports the narrow claim that these two prompts did not get the 1.7B anywhere near 230 wpm. It rests on one run per prompt, though, and the 0.6B pair shows that a run without any instruction on the same sentence can land at 139.1. The models differ, so this does not show the 1.7B instruction did nothing. It does show that no run in the A/B demonstrated the knob changing the output.
Nothing in the A/B flagged the problem. There was no error, the WER was 0%, and the small rise in wpm looked like a weak effect when one input was in fact missing. The record looked like a finished experiment.
Speed control went to a per-segment ffmpeg atempo stage that rejects factors outside 0.5–2.0. A calibration run set the factor: Qwen3's raw speech ran at about 131 wpm against an effective 183 for the existing engine, so matching it took factors of 1.41 to 1.76 depending on the speaker.
Checking the knob on the clone model
The smoke test that found the line also asked the question the A/B had skipped, this time about tone on the clone model. Qwen3's Base model, which does the cloning, has no instruct argument on its public clone method, and issue #25 says Base does not support instruction. The smoke test passed instruction ids through a non-public path anyway and measured the result. With an "excited" instruction the clone came out at an F0 mean/standard deviation of 237/59 Hz and 6.80 syllables per second. With "calm" it came out at 234/56 Hz and 6.46. There was no error, and these single runs did not show meaningful tone separation.
The reference clip, by contrast, moved the output a lot. Cloning Korean from a calm reference gave 221/52 Hz at 5.99 syllables/s, and an excited reference gave 271/86 Hz at 6.42. Into English, the same two references gave 221/42 and 217/41, so tone transfer across languages was weak.
So tone control goes through a bank of per-tone reference clips, not instructions. On my own recordings the Qwen tone bank spread F0 by 23.3 Hz across five tones and rate by 1.94 syllables/s. Its rank correlations against my own five-tone reading were 0.7 for F0 and 0.6 for rate. With n=5, that shows direction only.
The same check caught a knob that worked on one voice and not another. VoxCPM2's bracketed style prefix separated tones on the stand-in speaker, but on my voice the five tones spread F0 by only 7.3 Hz, with a rank correlation of -0.41. A knob checked on a stand-in has to be checked again on the configuration that ships.
The instrument needs the same check
The Base model's built-in speaker encoder scored the clones at 0.987–0.994 cosine similarity and a different, male preset speaker at 0.946. The clones were above that control, but on a narrow scale. Re-measured with WavLM-SV, which belongs to neither TTS system, the different speaker scored 0.275 and the same speaker across other tone references scored 0.977–0.992. The negative control that tested the wrong field makes a related point about checks: a check that flips on cue still has to be shown to react to the property it guards.
The rule I take from it
Before running an A/B on a control knob:
- Run the knob at its two extremes and measure a number that should move, such as wpm for speed or F0 for tone. If the extremes don't separate by more than repeat runs vary, stop.
- For an optional argument, trace its path to the model input. A library can discard it without raising an error.
- Calibrate the measuring instrument against a control known to differ.
- Repeat the check on the model size, speaker and mode you will ship, not only the one you prototyped on.
What this does not settle
The clone numbers come from one run per combination with seed 7 fixed, and the speed A/B from one synthesis per setting, so they show direction only. The stand-in reference clips were generated by Qwen's own CustomVoice model, which gives Qwen Base a favourable distribution to reproduce. That bias is why the timbre-similarity ranking had to be measured again on my own recordings. The blind listening test on those recordings had one listener and 15 Korean clips. The Qwen tone bank got 2 of 5 tones right, against a chance level of 1 in 5. Serene, calm and warm were mixed up with each other, and my own reading has nearly the same F0 for those three, so part of that limit probably comes from the reference performance. That is an estimate, not a measurement.
The dropped instruction is observed behaviour in qwen_tts 0.1.1. The package comment says instruction is not supported for the 0.6B model, and I did not test whether that checkpoint would respond if the wrapper passed an instruction through. Later releases may handle this differently. The 1.7B speed result still rests on two runs.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (0)