Introducing Real World VoiceEQ: Measuring the human quality of voice AI
Introducing Real World VoiceEQ: Measuring the human quality of voice AI
Voice is rapidly becoming AI's primary interface. From customer support and healthcare to education, entertainment, and personal assistants, speech is increasingly replacing text as the way people interact with AI.
Over the last few years, voice models have improved dramatically. Word error rates continue to fall, latency has reached conversational speeds, and many established benchmarks are approaching saturation. Yet anyone who regularly uses voice AI knows something still feels off.
Voice models can sound like different people over the course of a conversation, miss hesitation or uncertainty, and struggle with accents, noise, or emotional speech. Those shortcomings are easy to miss in benchmarks focused on latency and word error rate. People care whether a voice system can truly listen, respond appropriately, and remain natural and reliable in real conversations.
A broader benchmark for voice AI
To measure those qualities, we built Real World VoiceEQ—a benchmark designed to evaluate the human quality of voice interaction. It assesses whether voice systems can recognize, produce, and respond to the acoustic information transcripts leave out, from tone and emotion to speaker identity and background context.
Real World VoiceEQ evaluates more than 40 leading proprietary and open-source voice models across 15+ key evaluation dimensions and more than 60 metrics spanning Automatic Speech Recognition (ASR), Text-to-Speech (TTS), Speech-to-Speech (S2S), and Speech Understanding.
Real World VoiceEQ was developed from more than 1 million individual human ratings collected across different demographics, speaking styles, and acoustic environments. The current benchmark includes 785,000 TTS ratings and 48,000 STS ratings, making it one of the largest human evaluations of voice AI conducted to date.
Every evaluation was conducted using Kairos, our flexible, voice-native evaluation platform. The same infrastructure enables frontier AI labs and enterprises to run custom evaluations tailored to specific use cases, identify granular failure modes in production voice systems, generate human preference data, and continuously improve models through reinforcement learning and human feedback.
Key findings from Real World VoiceEQ
Progress in voice AI is becoming increasingly specialized.
The race for a single "best" voice model is giving way to a collection of specialized capabilities.
Today's leading systems optimize for different strengths—including technical accuracy, emotional understanding, conversational intelligence, expressiveness, and robustness. One model that excels at repeating booking reference numbers, bank account details, or complex pharmaceutical names may struggle to produce emotionally expressive speech. Another may sound remarkably natural but be less reliable on precision-oriented tasks.
As voice AI matures, measuring progress increasingly requires evaluating these capabilities independently rather than collapsing them into a single overall score. In our TTS evaluations, no system configuration ranked among the top five across all eight capability groups—underscoring why there is no single "best" voice model.
Voice models have become better at speaking than actually listening.
Speech-to-Speech models showed the widest variation of any category we evaluated. Some systems recognized emotion exceptionally well but struggled to respond naturally. We found that access to audio did not guarantee that agents used the paralinguistic information it contained. Some systems remained largely transcript-driven, relying on the words being spoken while overlooking cues such as tone, pacing, hesitation, emphasis, and volume.
Humans naturally use these cues to infer confidence, uncertainty, frustration, sarcasm, and empathy. Today's models often miss them.
Imagine a banking agent asking whether you recognize a potentially fraudulent transaction. A confident "Yes" and a hesitant "…yes…" may have completely different meanings, even though the transcript is identical. Humans recognize that difference immediately. Many of today's voice models do not.