Voice Agent Failures: What Breaks After the Demo

Plivo's CEO walks through the five problems that hit AI phone agents in their first week live — and the surprisingly simple fixes for most of them.

Every voice agent demo looks flawless. The bot answers on the first ring, understands the caller, and wraps up the call in under a minute. Then it meets real customers, real accents, and real background noise — and voice agent failures start showing up faster than most teams expect.

Nobody has watched more of these failures up close than Venky B, founder and CEO of Plivo. His company has carried telephony traffic since 2011 and now routes more than a billion voice calls a month, which makes his list of what breaks in production less of an opinion and more of a pattern recognized across thousands of deployments. In a recent talk for the AI Engineer conference, he laid out five specific ways voice agents fail in their first week live — and, more usefully, what actually fixes each one.

Failure One: Latency Nobody Notices Until the Caller Hangs Up

Text-based AI has spent the last year getting smarter by thinking longer — chain-of-thought, extended reasoning, multiple passes before answering. Voice agents don’t get that luxury. The gap between a caller finishing a sentence and the agent starting to respond has to be measured in hundreds of milliseconds, not seconds.

Venky’s numbers are specific: teams aim for under 550 milliseconds of response time, but most production systems land somewhere between 750 and 1,200 milliseconds. Past that threshold, callers don’t complain. They just hang up.

His team’s answer wasn’t a faster version of a large frontier model — it was smaller, open-source models hosted on their own infrastructure, tuned to respond in under 300 milliseconds. Model size wasn’t chosen for capability alone; it was chosen partly by how many tokens a given language needs per spoken word, since that directly affects how fast a model can start generating a response a caller will actually hear.

Failure Two: Assuming the Transcript Is Correct

The second failure sits upstream of almost everything else: trusting transcription output at face value. Speech-to-text is brittle by default, and it gets worse fast with proper nouns, unusual names, and code-switched speech — callers who drift between two languages mid-sentence, which breaks nearly everything built downstream of the transcript.

Venky’s own benchmark for evaluating a voice vendor is disarmingly simple: can it correctly pronounce his own surname and the name of his company? If a vendor’s text-to-speech engine stumbles on that, in his framing, it will stumble on your customers’ names too.

The Reframe That Fixed Data Collection

Of everything covered in the talk, one failure stood out as the one that wrecks half of all voice agent deployments: treating data collection as if it were just transcription. Venky’s fix is a full reframe — decide the shape of the data you need before you ever ask the question.

A phone number, in this model, isn’t a string of words to transcribe. It’s a typed field with a fixed length and a validator. If a caller’s response comes back with a stray digit or letter in the middle, the system either corrects it with enough confidence to move on, or asks the caller to repeat it — the same way a well-built form handles a bad input, rather than the way a stenographer handles a mumble.

The failure that wrecks half of all deployments isn’t the model. It’s treating a phone number like a sentence instead of a form field.

That distinction changes how you test the system, too. Instead of judging the whole conversation end-to-end, each field gets evaluated on its own, like a unit test: did the phone number field pass validation, did the date field resolve a relative phrase like “next Tuesday” correctly, did the email field come back spellable. Venky reports that this single reframe took data collection accuracy from roughly 30 percent to the mid-90s — without any model fine-tuning at all.

It’s a useful lesson well beyond voice: the same instinct — deciding the exact shape of the information you need before you go looking for it — shows up in how well-run companies structure a company knowledge base for AI systems to draw on. Loose, unstructured inputs produce loose, unstructured outputs, whether the input is a phone call or a document library.

Two More Failures Hiding in Plain Sight

Never Feed Raw Model Output Straight Into Speech

The fourth failure is subtler: sending a language model’s raw text output directly to a speech synthesizer. Numbers, abbreviations, dates, and currency symbols that read fine on a screen often sound wrong or robotic out loud unless they’re normalized first — spelled out, reformatted, or adjusted for how a human would actually say them.

Turn-Taking and Interruptions

The fifth failure covers turn detection and barge-in: knowing when a caller has actually finished speaking versus paused, and handling it gracefully when they interrupt the agent mid-sentence. Get this wrong and the agent talks over people, or sits in awkward silence waiting for a caller who is already done. It’s also where the hardest edge case shows up — knowing when the agent has hit the limit of what it should handle alone and needs to route the caller to a human, a boundary worth thinking through before launch, not during an angry call. We’ve written about where that line should sit in how AI agents should hand off to people.

What This Means If You’re About to Ship a Voice Agent

Most voice agent failures aren’t exotic. They’re the predictable result of treating a phone call like a text chat with audio bolted on. Latency budgets, transcript accuracy, structured data capture, speech normalization, and turn-taking are all solvable — but only if you design for them before launch, not after the first angry customer call.

Shipping infrastructure like this to a paying customer carries its own kind of exposure, the same exposure any founder takes on when they hand something unproven to a first customer for hard technology. The evaluation discipline Venky describes — testing fields instead of whole conversations, watching real usage instead of guessing — is also the same feedback loop that improves any customer-facing AI system over time, from voice agents to how a company handles its own sales replies as it learns from customers.

The biggest lesson for founders building or buying this technology isn’t a checklist — it’s a mindset shift. Stop asking whether your voice agent understood the caller, and start asking whether it captured exactly the data you needed, in exactly the shape you needed it, every single time. That’s the difference between a demo that impresses a room and an agent that survives its first thousand real phone calls.

▶

Sources

• 5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo (AI Engineer) — Talk published 2026-09-15, approx. 27 minutes
• Venky B on X — Speaker profile
• Venky B on LinkedIn — Speaker profile