How we think about latency in SpeechifyAI voice agents

Baseten published a case study on our voice agent pipeline: How SpeechifyAI built an ultra-low-latency voice agent without compromising on voice quality. Read it because it talks about the part of voice agents that is easy to hand-wave and hard to fix later: where the models run.

A live voice agent is not one model. It is a chain. ASR hears the caller, an LLM decides what to do next, TTS speaks back. Every turn walks that chain, and every hop adds latency. Put those pieces in different clouds, different regions, or behind vendors that won’t give you region control, and you pay a network tax on every single exchange.

That is the whole point of the Baseten partnership for us. Keep the critical path close together. Keep it predictable. Don’t make the caller wait while three separate services bounce packets across the public internet.

The useful bit is co-location

The Baseten case study says the pipeline runs streaming Whisper for ASR, GLM-5.2 for the LLM, and Simba for TTS in one region. Same region, same facility, same turn loop. Their writeup puts the latency win at 75 to 150 ms lower per turn compared with a stitched multi-vendor pipeline.

That number matters. Human turn-taking is brutal. A delay that looks small in a backend trace can feel like interruption, hesitation, or confusion on a phone call. Baseten’s article calls out that 300 to 400 ms of added latency can break the live-call experience. I believe it. You can feel it in your body before you can explain it in a dashboard.

The funny thing is that none of this sounds glamorous. Co-location, same-region hops, traffic-based autoscaling, streaming time-to-first-byte. Good. The boring stuff is where real-time products either feel alive or don’t.

Simba is the part we cannot casually swap

ASR can fail over. LLMs can fail over, within reason. If a backup model is a little slower or phrases the next sentence differently, the caller probably doesn’t notice.

The voice is different.

Kai puts it plainly in the case study: “The voice model is the most crucial single point of failure.” If the voice changes halfway through a call, it is not a subtle infrastructure event. It is audible. The person on the other end hears the product break character.

That is why Simba sits differently in the architecture. It is not just one more provider in a route table. It is the identity layer of the call. Baseten hosting that layer reliably is not an implementation detail for us, it is part of the product quality.

What the case study gets right

The article is not really about chasing benchmark scores. It is about the shape of a real voice agent system once it has to answer calls all day.

It talks about the model chain. It talks about why Speechify evaluated open-source LLMs on voice-agent-specific criteria: low latency, reliable tool calling, and clean instruction-following on imperfect ASR input. It talks about redundancy where redundancy makes sense, and about not pretending TTS can fail over invisibly when voice drift is obvious.

And, yes, it mentions pricing. Infrastructure efficiency flows into pricing when the whole stack is designed as one system instead of stitched together after the fact. The case study says Simba text-to-speech starts at $6 per 1M characters, and voice agents are priced from $0.068/min all-in. That is not magic. It is the consequence of owning the expensive layer and running the chain where latency doesn’t leak through the seams.

Read the Baseten writeup

This post is the short version from our side. Baseten’s case study has the architecture details, the metrics, and Kai’s quotes in one place.

Go read it: How SpeechifyAI built an ultra-low-latency voice agent without compromising on voice quality.

We will write more about the product layer separately. This one is just about the partnership and the infrastructure choice underneath it: keep ASR, reasoning, and voice close together, because real-time voice does not forgive distance.