Word-level timestamps in a LiveKit voice agent with Speechify

Update, 10 September 2026: simba-3.2 serves every English voice in the catalogue, cloned voices included — the _32 voices are just its built-in examples, not a limited roster. It stays English-only; use simba-3.0 for other languages.

Speechify’s official LiveKit plugin streams word-level timestamps alongside the audio, and you read them by overriding one method on your agent: transcription_node. Each spoken word arrives as a TimedString carrying start_time and end_time in seconds, so you can highlight words as the agent says them without a second API call or a separate alignment pass.

This is Python. The package is livekit-plugins-speechify, published by LiveKit, and it reads your key from SPEECHIFY_API_KEY. If you don’t have one yet, grab a Speechify key first.

Where the timestamps come from

A LiveKit voice agent is a loop: the user speaks, STT turns it into text, the LLM writes a reply, and TTS speaks it back. Each stage is a provider object on an AgentSession.

LiveKit AgentSession

LiveKit handles the realtime room and turn loop. The Python agent passes Deepgram, OpenAI, and Speechify providers into one session.

1. STTDeepgram listensUser audio becomes text inside the LiveKit session.deepgram.STT(model="nova-3")
2. transcript
3. LLMOpenAI repliesThe agent turns the transcript into a short conversational response.openai.LLM(...)
4. text
5. TTSSpeechify speaksThe official LiveKit plugin calls Speechify with the voice and model you choose.speechify.TTS(...)

The Speechify key stays in the Python environment as SPEECHIFY_API_KEY. The browser or room participant never receives it.

The Speechify TTS stage does one extra thing. When it synthesizes a sentence it calls Speechify’s /v1/audio/stream/with-timestamps endpoint, which streams audio chunks and word-level speech marks together. The plugin advertises this to LiveKit as aligned_transcript=True and forwards each word as a TimedString — a normal Python string that also carries start_time and end_time.

Two things to know before you build:

• Speech marks only come from the streaming-native models, simba-3.0 and simba-3.2. The legacy simba-english and simba-multilingual models fall back to a plain non-streamed request with no marks.
• The times are in seconds, measured from the start of that turn’s audio.

Install

Create a fresh environment and install LiveKit Agents, the Speechify plugin, and the STT/LLM plugins this agent uses:

python3 -m venv .venv
source .venv/bin/activate
pip install \
  "livekit-agents[codecs]>=1.8.0" \
  livekit-plugins-speechify \
  livekit-plugins-deepgram \
  livekit-plugins-openai \
  python-dotenv

livekit-plugins-speechify pulls in speechify-api>=4.0.0 on its own. The import you’re wiring toward is:

from livekit.plugins import deepgram, openai, speechify

Configure the environment

Put your keys in .env:

SPEECHIFY_API_KEY=your_speechify_api_key

DEEPGRAM_API_KEY=your_deepgram_api_key
OPENAI_API_KEY=your_openai_api_key

# Only needed when you run against a real LiveKit room (dev/start):
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

The plugin reads SPEECHIFY_API_KEY unless you pass api_key= to speechify.TTS(...) yourself. Keep it server-side, in the Python agent — never hand it to a browser participant.

The agent

Here’s the whole thing — also available as the runnable livekit-captions-speechify-python demo. It’s a standard LiveKit AgentSession with Deepgram STT, an OpenAI LLM, and Speechify TTS. The one addition is transcription_node, where we read the timings.

import logging

from dotenv import load_dotenv
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, ModelSettings, cli
from livekit.agents.voice.io import TimedString
from livekit.plugins import deepgram, openai, speechify

load_dotenv()
logger = logging.getLogger("speechify-captions")


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions=(
                "You are a helpful voice assistant speaking with a Speechify voice. "
                "Keep replies short and conversational."
            )
        )

    async def on_enter(self) -> None:
        self.session.generate_reply(
            instructions="Greet the user and mention your voice is powered by Speechify."
        )

    async def transcription_node(self, text, model_settings: ModelSettings):
        # `text` streams the words the agent is about to speak. With Speechify
        # TTS, each word arrives as a TimedString carrying start/end times (in
        # seconds). Read them here, then yield the chunk on unchanged so the
        # normal transcript still forwards to the room.
        async for chunk in text:
            if isinstance(chunk, TimedString) and isinstance(chunk.start_time, (int, float)) and isinstance(chunk.end_time, (int, float)):
                logger.info("[%6.2f–%6.2f] %s", chunk.start_time, chunk.end_time, chunk)
            yield chunk


server = AgentServer()


@server.rtc_session()
async def entrypoint(ctx: JobContext) -> None:
    session = AgentSession(
        stt=deepgram.STT(model="nova-3"),
        llm=openai.LLM(model="gpt-4.1-mini"),
        # simba-3.2 is streaming-native, so it serves word marks.
        tts=speechify.TTS(voice_id="dominic_32", model="simba-3.2"),
    )
    await session.start(agent=Assistant(), room=ctx.room)


if __name__ == "__main__":
    cli.run_app(server)

Save it as agent.py — the run commands below assume that name. The tts= line is the integration. The transcription_node override is the whole point of this post: it hands you the exact word boundaries the audio is being cut on.

What you get out

Run the agent (below) and each spoken word logs its window:

[  0.00–  0.32] Hi
[  0.32–  0.51] there
[  0.63–  0.98] I'm
[  0.98–  1.44] powered
[  1.44–  1.62] by
[  1.62–  2.10] Speechify

Because transcription_node sits in the pipeline, this is live — the times land as the words are synthesized, not after the turn. That server-side stream of per-word windows is the building block for karaoke-style captions: forward each word with its start_time/end_time to your frontend and schedule the highlight against audio playback. The snippet here logs the timings; wiring them to a caption UI is the piece you add on top. TimedString subclasses str, so any code that already treats the transcript as text keeps working; the timings are just extra attributes hanging off it.

One caveat worth stating plainly: Speechify marks intra-word punctuation as its own token, so a word like “well-known” can arrive as well, -, known. If you’re rendering whole-word highlights, stitch adjacent tokens by their times.

Run it

For local iteration, console mode runs against your microphone and speakers with no room:

python agent.py console

To connect a real LiveKit room, fill in the LIVEKIT_* variables and use development mode:

python agent.py dev

For production, run python agent.py start. At that point you’re operating a realtime service — rotate keys, watch model errors, and measure the whole turn, not just TTS.

Notes on voices and models

dominic_32 with simba-3.2 is a clean pairing for a Python example. Swap the voice freely, but keep the pairing valid — a voice has to support the model you pass. The _32 voices are simba-3.2’s built-in voices, but it serves every English voice in the catalogue; check a voice’s models array to confirm the pairing. Browse them at platform.speechify.ai, or hit the /v1/voices endpoint. If you switch to a legacy model you lose the word marks, so keep captions on simba-3.0 or simba-3.2.

Where to go next

• Want the same captions without LiveKit, straight from the API? Build real-time captions with Speechify TTS speech marks.
• The lower-level streaming shape is in Streaming TTS in Python with Speechify.
• For the number users actually feel, read how we think about latency in Speechify voice agents.
• Model and voice context: Simba 3.2 and the models endpoint.

The Speechify docs and LiveKit’s Speechify plugin guide cover the rest of the surface.

FAQ

How do I get word-level timestamps from Speechify in LiveKit?

Use speechify.TTS(model="simba-3.2") in your AgentSession and override transcription_node on your Agent. The words stream in as TimedString objects with start_time and end_time in seconds. No separate alignment call is needed — the timestamps come back on the same streaming request as the audio.

Which models return speech marks?

The streaming-native models simba-3.0 and simba-3.2. The legacy simba-english and simba-multilingual models don’t serve the streaming route and return no marks.

Are the times in seconds or milliseconds?

The plugin gives you TimedString.start_time / end_time in seconds. (Speechify’s raw API reports marks in milliseconds; the plugin converts them for you.)

Is there a JavaScript version?

This walkthrough is Python. The published LiveKit Speechify plugin is livekit-plugins-speechify on PyPI.

Do I need a LiveKit room to test it?

No. python agent.py console runs locally against your mic and speakers. You still need SPEECHIFY_API_KEY, DEEPGRAM_API_KEY, and OPENAI_API_KEY because the agent still calls those providers.