Why Voice Agents Mishear People (And What Actually Fixes It) cover illustration

## Phone audio is a genuinely harder problem Speech recognition demos run on clean audio from a good microphone in a quiet room. Phone calls are none of those things. Traditional telephony carries a narrow slice of the frequency range and compresses it aggressively, which strips out exactly the high-frequency detail that distinguishes similar consonants — the difference between "f" and "s", or "b" and "p", often lives in the part of the signal the network threw away. Add packet loss, mobile handoffs, speakerphone echo, and a caller in a car, and the recogniser is working from a degraded copy. This is not a flaw in a particular vendor; it is a property of the medium. It means you should expect a meaningfully higher error rate on calls than in your testing, and design for it rather than treating each mistake as a bug to be fixed individually. Overview of Why Voice Agents Mishear People (And What Actually Fixes It)

Overview of Why Voice Agents Mishear People (And What Actually Fixes It)

## The errors cluster exactly where the value is Recognition errors are not evenly distributed across a conversation. Ordinary connected speech is handled well, because the model can use context to resolve ambiguity — surrounding words make most guesses recoverable. The errors concentrate on precisely the items that carry no context: proper names, street addresses, email addresses, order numbers, policy numbers, postcodes, and any string of letters and digits read aloud. These are also, inconveniently, the pieces of information most business calls exist to capture. A caller saying "my reference is B as in bravo, four, seven, D" is the hardest input a phone agent will face, and it is a completely routine one. Any evaluation of a voice agent that does not specifically test alphanumeric capture is measuring the easy part. ## What actually helps, in order of return The instinct when recognition is poor is to swap models. That is rarely where the biggest gain is. The highest-return fix is telling the recogniser what to expect: supplying the vocabulary specific to your business — product names, clinic names, local place names, the format your reference numbers take — so that the system weights plausible interpretations correctly. A recogniser that knows your company sells a product called "Aeris" will stop transcribing it as "Eris" or "arrows". After that comes constraining the format at the point of capture, so a postcode is validated as a postcode rather than accepted as free text. Then comes handling the audio path properly — echo cancellation and sensible endpointing. Model choice matters, but it is usually the fourth lever, not the first. ## Design for being wrong, because you will be No configuration makes recognition perfect on a phone line, so the durable fix is conversational rather than technical: confirm anything expensive to get wrong. Read back the address before dispatching an engineer. Confirm the last four digits rather than the whole number. Ask the caller to spell an unusual surname rather than guessing. The craft is in confirming selectively — an agent that repeats every single value back sounds like an interrogation and adds a turn to every exchange, which has its own cost in call length and patience. Confirm what is expensive to get wrong, accept what is cheap to correct later, and let the caller move on. Getting that judgement right is a large part of what separates a voice agent people tolerate from one they do not notice. ## Frequently asked questions Speech recognition demos run on clean audio from a good microphone in a quiet room. Phone calls are none of those things. Recognition errors are not evenly distributed across a conversation. Ordinary connected speech is handled well, because the model can use context to resolve ambiguity — surrounding words make most guesses recoverable. The instinct when recognition is poor is to swap models. That is rarely where the biggest gain is. No configuration makes recognition perfect on a phone line, so the durable fix is conversational rather than technical: confirm anything expensive to get wrong. Read back the address before dispatching an engineer. ## Try Persistence Design, test, and deploy production-ready voice agents. ## Continue reading