In our (currently) in-house business automation agent, we route prompts to the right agent. This is done using an agent router.
Take-away: before a prompt reaches the active agent, two small local models ask one question: is this the right agent? Either says yes, the prompt stays. Both say no, Kev-4B picks the three best agents to switch to. No data leaves the premises, and a call costs $0.
Why two models
- bge-small (6 ms) is very accurate at catching prompts that don't belong. But it is not good at picking the right agent (in our experiments).
- Kev-4B (local, ~0.12 s) is slightly lower at calling out prompts that don't belong, but good at picking the right target agent: it is in its top 3 for 97% of prompts, against 94% for bge-small.
- Jev is the most accurate on both counts, but it runs over the network and the prompt leaves the premises. It stays as a configurable alternative.
Results *4
150 agent-goals *3, 100 prompts each of three kinds: on-goal (matches the agent's goal), other-goal (belongs to another agent) and out-of-scope (belongs to no agent). A prompt that doesn't match the agent's goal is off-goal, so other-goal and out-of-scope are both off-goal.
- Detection: an off-goal prompt is flagged.
- False alarm: an on-goal prompt is wrongly flagged.
- Right agent for the prompt: an other-goal prompt is sent to the correct agent.
| Option | Detection | False alarm | Right agent for the prompt | Latency *1 | Cost / call |
|---|---|---|---|---|---|
| Jev (top-10 shortlist) | ~100% | 1% | 91% | ~0.46 s (network) | ~$0.00003 |
| Kev-4B *2, flag if P(current) < 0.3 | 99.5% | 8% | 85% (97% in top 3) | ~0.12 s | $0 |
| bge-small, similarity < 0.65 | 100% | 7% | 83% (94% in top 3) | ~0.006 s | $0 |
| bge-small, current goal not in top 5 | 97.5% | 1% | 82% (93% in top 3) | ~0.006 s | $0 |
| Laya *2 (shortlist of 10) | 100% | 19% | 86% | ~0.01 s | $0 |
| Cross-encoder (MiniLM-L6) | 99% | 29% | 75% | ~0.01 s | $0 |
| Cross-encoder (bge-reranker-base) | 99% | 36% | 65% | ~0.02 s | $0 |
Out-of-scope prompts: Kev flags 99 of 100 and correctly shows no suggestion for 80.
*1 Measured on one machine (RTX 5070). Jev's latency includes the network.
*2 Used as they come, not fine-tuned.
*3 Synthetic data (hand-written business prompts) and CLINC150 (a public intent dataset). Not tested in deployment with real customer prompts or usage.
*4 Limited dataset, so the numbers may be optimistic.
Top comments (1)
The false-alarm column is the one I would look at before the others. Kev's 8% and bge's 7% are one prompt apart out of 100 on-goal prompts (Fisher p = 1.0), and the Wilson 95% intervals are roughly 4-15% and 3-14%. The 1% row (bge top-5) against Kev's 8% is a real gap (p about 0.03). The "right agent" column does not separate: 85, 83 and 82 each carry about ±7 points at n=100, and top-3 97% vs 94% gives p of about 0.5.
What a false alarm costs depends on traffic mix, which a 1:1:1 test set hides. With flag precision = (off-goal share x detection) / (that + on-goal share x false alarm), if 90% of real prompts are on-goal, Kev's flag is right only about 58% of the time, bge top-5 about 92%. At 95% on-goal, 40% vs 84%. So the 1% false-alarm option can beat the higher-detection one even though detection looks nearly identical (99.5 vs 97.5, 199/200 vs 195/200).
Two things that would sharpen it: report the actual on-goal share from your logs and compute precision at that share, and keep a slice of real customer prompts that sit near two agents' goals, since hand-written and CLINC150 prompts tend to be cleaner than production ones. How are the two models combined when one says yes and one says no, and does the false-alarm rate hold for that combination?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.