AI is changing the world. Don't stay behind. Clear summaries, community insight, delivered without the noise. Subscribe to never miss a beat.

How we made a text-to-speech model respond in sub-50 ms

Pushing the Speed-Cost Frontier for Qwen3-TTS

nari-labs.com

August 21, 2026

8 min read

🔥🔥🔥🔥🔥

50/100

Summary

Nari Labs open-sourced a Qwen3-TTS CustomVoice 1.7B serving implementation that it says delivers 10 requests per second on one NVIDIA H100 SXM while keeping p95 time to first audible audio below 50 milliseconds and sustaining real-time playback. The system produces about 630 characters per second at 10 RPS; using a $4.29-per-hour H100 price, Nari estimates a compute cost of roughly $2 per million characters, excluding networking, idle capacity, and operational overhead. The benchmark used five-minute Poisson open-loop workloads, complete-text HTTP requests, streamed PCM audio, audible-onset detection, playback reconstruction, and Deepgram speech-to-text evaluation. Nari reports that its implementation maintains sub-50 ms p95 audible latency through 10 RPS and remains below 100 ms at 20 RPS. After tuning for silence trimming and audio-frame accumulation, VoxServe reached 49.3 ms p95 time to first audio at 1 RPS but rose to 363.2 ms at 6 RPS; vLLM-Omni measured 56.815 ms and 93.451 ms, respectively. The implementation schedules Qwen3-TTS’s Talker, Code Predictor, and Codec as separate jobs under one scheduler. It prioritizes first audio before playback begins, then schedules established streams around audio-buffer deadlines. Additional optimizations include CUDA graphs for the Code Predictor, specialized Triton attention, state-cached incremental Codec decoding, variable chunk sizes, and optional input streaming from an upstream language model.

Key Takeaways

Nari Labs says its Qwen3-TTS CustomVoice 1.7B server achieves 10 RPS with sub-50 ms p95 audible time to first audio on a single NVIDIA H100 SXM.
Nari estimates approximately $2 per million generated characters at full H100 utilization, excluding networking, idle capacity, and operational overhead.
The serving stack independently schedules the Talker, Code Predictor, and Codec so it can prioritize initial audio delivery and later prevent playback-buffer underruns.
Nari reports that, after tuning, VoxServe reached 49.3 ms p95 time to first audio at 1 RPS but 363.2 ms at 6 RPS, while vLLM-Omni measured 56.815 ms and 93.451 ms.
The open-source implementation uses CUDA graphs, specialized attention kernels, cached Codec state, and progressively larger audio chunks to reduce latency while preserving throughput.

What the discussion said

Commenters largely agree that first-audio latency is a make-or-break metric for voice agents, and the reported 34 ms p95 on one H100 drew real interest because it could make streamed responses feel immediate. The open implementation and benchmark were welcomed as useful alternatives to production stacks that have struggled with realtime playback. Several readers also wanted practical details the headline leaves open: cold-versus-warm measurements, consumer-hardware performance, a demonstration, cloud deployment, and whether the project is moving from model research toward serving infrastructure. The thread repeatedly pulled the focus from TTS alone to the end-to-end conversational pipeline. Streaming language-model output into Qwen3-TTS avoids waiting for a completed sentence, but readers stressed that language-model time-to-first-token and speech recognition still consume most of the interaction budget. Local-agent builders were particularly interested in bringing this performance to phones rather than relying on an H100, while warning that aggressive acceleration often eventually damages voice quality. Another split concerned human interaction design: ultra-fast audio is technically impressive, but replies that arrive before a person can plausibly process speech can feel like interruption, while filler sounds used to hide model delay may be funny or uncanny rather than genuinely responsive.

Where opinion split

The sharp disagreement is whether shaving TTS latency below 50 ms directly makes voice agents better. Advocates see it as essential infrastructure for natural, streaming dialogue and say their optimizations preserve the original Qwen output. Skeptics argue that end-to-end LLM latency, voice quality, mobile feasibility, and human conversational timing matter more; an unnaturally instant or filler-led reply can make an agent feel overeager rather than present.

Read original article

Community Sentiment

Positive

Positives

A 34 ms p95 first-audio result at 10 requests per second on a single H100 is seen as a serious step toward voice agents that stop feeling like delayed APIs.
Streaming language-model output straight into the speech model can begin speaking before a full sentence exists, cutting a major source of perceived waiting.
Open-sourcing both the optimized implementation and benchmark gives realtime-voice developers something concrete to test against slower serving stacks.
The authors' claim of matching the original model's output while improving latency is encouraging, because speed gains are worthless if the voice becomes noticeably worse.

Concerns

An H100 benchmark does not answer the deployment question for personal assistants; commenters want the same responsiveness on inexpensive phones rather than datacenter GPUs.
A local-agent developer warns that faster generation eventually collides with a quality ceiling, where reduced synthesis work produces voices too rough for convincing conversation.
TTS startup is only one slice of the delay: language-model first-token latency and speech recognition can still keep an agent far from genuinely realtime.
Sub-50 ms speech can be socially wrong as well as technically right, since people may read an immediate reply as interruption; filler utterances used to mask delay can feel awkward and overeager.