Voice Agent Tool Calls: What to Run Inline and What to Hand Off

In a chat agent, a slow tool call is a spinner. In a voice agent, it is silence on a phone line while a human being waits, decides you did not hear them, and starts talking again.

That single difference is what makes tool design on voice agents a latency problem before it is an architecture problem. The question is never just “can the agent do this”. It is “can it do this inside the turn, and if not, what does it say while it cannot”.

The turn budget is the whole constraint

A turn walks a chain: speech-to-text hears the caller, a language model decides what to do, text-to-speech answers. Every hop spends part of a budget that human conversation sets, not you. As we covered in how we think about latency, 300 to 400ms of added latency is enough to break the feel of a live call.

A tool call spends from that same budget, and it spends it in the worst place: after the caller has finished speaking and before anything comes back. Sub-300ms time to first byte on the speech model buys you nothing if a database round trip sat in front of it.

So the first thing to know about a tool is not what it does. It is how long it takes, at p95, from your agent’s region.

The four kinds, and what each one costs you

SpeechifyAI agents support four tool kinds, and they have genuinely different latency shapes:

Kind	Where it runs	Latency shape	Safe inline?
System	The agent process. end_call, transfer_to_number, transfer_to_agent, play_keypad_touch_tone, skip_turn	No HTTP at all, milliseconds	Always
Webhook	Your backend, signed HMAC-SHA256	One network round trip plus your handler	Only if your p95 is fast
Client	The caller’s browser or SDK, over the session’s tools data channel	Round trip to a device you do not control	For UI actions, not for data
MCP	Your MCP server, proxied by the worker	The webhook hop plus a proxy hop	Rarely, and never for writes

Two of these get misjudged consistently.

Client tools feel local and are not. They dispatch to the caller’s device, which may be a phone on a train. Use them to navigate a page, fill a form, or update a cart, where the action is the point and the return value is an acknowledgement. Do not use them to fetch data the agent needs in order to speak the next sentence.

MCP tools add a hop you did not have. The worker opens the configured transport at session start and discovers your remote tools, so discovery is not on the critical path, which is good. But every invocation is then proxied through, so an MCP tool is a webhook tool with a proxy in front of it. MCP is the right choice when you already have a server exposing capabilities to several agents and want one place to maintain them. It is the wrong choice for the single order lookup on your critical path.

The rule

Run it inline if it is a read that reliably returns inside your turn budget. Hand it off otherwise.

That splits cleanly along a line worth stating explicitly, because it is not the line people expect. It is not fast versus slow. It is read versus write.

Inline, comfortably:

• Order status, delivery window, account balance, appointment slots. One indexed lookup, no side effects, and the answer is the thing the agent needs to say next.
• Anything in the system built-ins. Transferring, ending the call, sending a DTMF tone, and skipping a turn all run in-process.
• Knowledge base retrieval, which is what it is designed for.

Handed off, always:

• Anything that charges a card, books an irreversible slot, sends a message, or writes to a system of record.
• Anything whose p95 you cannot state from memory.
• Anything that fans out to a third party you do not operate.

Why writes are the harder case, even when they are fast

A slow read is a latency bug. A write inline is a correctness bug waiting for a bad connection.

The caller can hang up at any moment, including the moment between your write committing and the agent speaking the confirmation. If your agent retries a failed-looking call that actually succeeded, you have charged a card twice. Voice makes this worse than chat does, because a dropped call is normal rather than exceptional, and because the caller has no transcript to check.

Handing writes off is what fixes this, and the mechanism is more ordinary than it sounds: the write gets an idempotency key and goes on a queue, the tool returns immediately, and the queue owns the retry. The agent is then free to say what happened without being the thing that has to make it happen.

The handoff pattern

The key detail is what a webhook tool actually returns. Your endpoint receives an envelope and replies with JSON that becomes the tool’s return value, which is the material the model speaks from:

// what your endpoint receives
{ "tool_call_id": "call_abc123", "tool_name": "issue_refund", "arguments": { "order_id": "ORD-42" }, "timestamp": 1713360000000 }

So a handed-off tool does not return a result. It returns a receipt:

// return this immediately, do not await the refund
{ "accepted": true, "reference": "RFD-8814", "eta": "within one business day" }

The agent now has something true and specific to say, in the turn, without the refund having happened yet. “That is submitted, your reference is RFD-8814, and you will have an email within a business day.” The work completes on your queue afterwards.

Three things make this hold up:

1. The receipt has to be honest. accepted means you durably enqueued it. If the queue write itself failed, return an error and let the agent escalate, because an agent that confirms work you dropped is worse than one that transfers.
2. The reference has to be real. Generate it before enqueueing so the caller can quote it to a human later.
3. The follow-up has to actually happen. Handoff moves the obligation, it does not remove it. Something has to send the email, and something has to alert a person when the queue fails.

That last point is where most of these designs quietly rot, and it is a solved problem you should not solve again. Whatever picks the work up on the other side is usually something you already run in Slack, Gmail, or a workflow tool. If you do not have that layer yet, Techsy’s AI playbook library for builders collects working skills and automations for exactly those surfaces, which is a faster start than writing the notification path from scratch.

Say something while you wait

Even a well-chosen inline tool sometimes runs long. Silence is the failure mode, not the wait.

Instruct the agent to narrate before the call, not after: “let me pull that up” costs about a second of speech and buys the entire lookup. That is not a trick, it is what a human agent does, and callers read it as competence rather than delay.

For the opposite case, skip_turn lets the agent deliberately not speak, which is the right handling when the caller is mid-thought and a filler phrase would talk over them.

What does not work is a generic “one moment please” attached to every tool. Callers learn it means nothing within two uses.

Instrument it, or you are guessing

Every tool invocation is persisted on the transcript with role=tool, tool_name, tool_args, and tool_result, readable from GET /v1/agents/conversations/{id}/messages.

That is your latency dataset and almost nobody mines it. Pull it weekly and look for three things:

• Tools whose p95 has drifted past your budget. The one you benchmarked at 120ms in March is not necessarily still that.
• Tools the model calls more than you expected. A tool with a vague description gets called speculatively, and each speculative call is a turn the caller waits through.
• Tools whose result the model then ignores. Usually a schema problem. Parameters take string, number, integer, or boolean, and a string can declare an enum of allowed values. Using the enum is the cheapest accuracy win available, because it removes a class of guess before the call is made.

Worked example: an order line

For a support agent handling order questions, the split ends up roughly:

Capability	Kind	Inline or handoff
Look up order status	Webhook	Inline, indexed read
Read the delivery window	Webhook	Inline, same read
Change a delivery address	Webhook	Handoff, it is a write
Issue a refund	Webhook	Handoff, plus idempotency key
Email the tracking link	Webhook	Handoff, side effect
Transfer to a human	System	Inline, in-process
End the call	System	Inline, in-process

One read on the critical path. Everything with a side effect gets a receipt. That shape holds across most agents worth building, and it is worth designing to before the first tool ships rather than after the first double refund.

Try it

You can start free with 60 voice agent minutes a month, no card, which is enough to wire one webhook tool and measure its real p95 on a live call rather than in a unit test. Tool calling and webhooks are the primary integration surface, so CRM and order-system integrations go through this path rather than through native connectors. See the tools guide for the create-and-attach calls and the signature verification snippet.

Store the webhook signing secret when you create the tool. It is returned once, every later read masks it, and there is no retrieval endpoint.

FAQ

What is a tool call in a voice agent? A function the language model can invoke mid-conversation to fetch data or take an action, such as looking up an order or transferring the call. On SpeechifyAI agents there are four kinds: system built-ins that run in the agent process, webhook tools that call your backend, client tools that run on the caller’s device, and MCP tools proxied through a server you host.

Should a voice agent tool call run synchronously? Only if it is a read that reliably returns inside the turn budget, because the caller is waiting in silence while it runs. Writes, slow lookups, and anything hitting a third party you do not operate should return a receipt immediately and complete asynchronously on a queue.

How fast does a voice agent tool call need to be? Fast enough that the total turn stays inside conversational tolerance, where 300 to 400ms of added latency is already enough to break the feel of a live call. Judge a tool on its p95 from the agent’s region rather than its median on your laptop, and instrument it from the transcript rather than trusting a benchmark taken at integration time.

How do I stop a voice agent double-charging on a retry? Do not perform the charge inside the tool call. Generate an idempotency key, enqueue the write durably, return a reference the agent can read aloud, and let the queue own retries. A caller hanging up between commit and confirmation is normal on a phone line, so the write path has to be safe against it by construction.

Are MCP tools slower than webhook tools? They add a proxy hop, so yes, at invocation time. Discovery is not on the critical path because the worker opens the transport and discovers remote tools at session start. MCP is a good fit when several agents share capabilities you want to maintain in one place, and a poor fit for the single lookup sitting on your latency budget.

What can I use to handle work a voice agent hands off? Anything that owns a durable queue and can notify a human when it fails. In practice most teams route it into a tool they already run, commonly Slack or Gmail, rather than building a notification path specifically for the agent.