Sub-100ms Responses: Inside Persistence's Voice Architecture cover illustration

## Why latency is the defining metric for voice AI Voice AI is uniquely sensitive to latency in a way that text chatbots are not. When you ask a chatbot a question, a 2-second response feels fine. When you ask a voice agent and it takes 2 seconds to respond, the silence feels like a dropped call. Human conversational tolerance for response latency is approximately 200–500ms before the experience degrades. Above 800ms, most callers mentally categorize the agent as broken, slow, or stupid — regardless of the quality of the response. This is why raw intelligence (better LLM) cannot compensate for high latency. A brilliant answer delivered in 2 seconds loses to a good answer delivered in 150ms. Overview of Sub-100ms Responses: Inside Persistence's Voice Architecture

Overview of Sub-100ms Responses: Inside Persistence's Voice Architecture

## Our numbers — and how we measure them Persistence's median end-to-end latency is 94ms from end of caller speech to first spoken audio byte from the agent. 95th percentile latency is 180ms. These numbers are measured in production across our full global call volume — not in a controlled benchmark environment. We measure continuously and publish internal dashboards that track latency by region, model, and call type. Comparable published numbers from Retell AI cite 800–1200ms. Bland AI does not publish end-to-end latency numbers. Vapi's documentation describes their standard pipeline as 1–2 seconds. The gap is not incremental — it's architectural. ## How we achieve sub-100ms latency Three engineering decisions drive our latency advantage. First: co-location. Our STT and LLM inference run in the same data center, eliminating network round-trips between components. When you chain calls across three separate API providers (a common pattern for competitors), you add 200–400ms of network overhead before any inference runs. Second: streaming everywhere. We stream audio into STT, stream tokens from STT into LLM context, and begin TTS generation before the LLM completes its response. Third: speculative decoding on the TTS side — we begin generating the first words of likely responses before the LLM is certain of the full output, then correct course if needed. This aggressive pipelining compounds into the latency advantage you see in production. ## Regional infrastructure for global calls Latency is also a function of physical distance. A call placed from Tokyo that routes to inference infrastructure in Virginia adds 100–150ms of unavoidable network latency. Persistence operates inference nodes in North America, Europe, and Asia Pacific, with call routing that automatically selects the nearest available node. This geographic distribution is expensive to operate but eliminates a latency ceiling that purely US-based competitors cannot escape for international customers. ## 99.99% uptime: latency under load Fast average latency is meaningless if the system degrades under load. Persistence's infrastructure is designed for 99.99% uptime with automatic failover across redundant inference nodes. We load-test at 3x peak capacity on a weekly basis. When any node degrades, traffic reroutes within 200ms. Our largest customer runs 80,000 concurrent calls at peak. Latency at that volume is indistinguishable from our single-call benchmark numbers. That kind of headroom is only possible with purpose-built infrastructure — not a wrapper around commodity cloud APIs. ## Frequently asked questions Voice AI is uniquely sensitive to latency in a way that text chatbots are not. When you ask a chatbot a question, a 2-second response feels fine. Persistence's median end-to-end latency is 94ms from end of caller speech to first spoken audio byte from the agent. 95th percentile latency is 180ms. Three engineering decisions drive our latency advantage. First: co-location. Latency is also a function of physical distance. A call placed from Tokyo that routes to inference infrastructure in Virginia adds 100–150ms of unavoidable network latency. ## Try Persistence Design, test, and deploy production-ready voice agents. ## Continue reading