🔍 Read the full analysis: Can The Open TTS Leaderboard Scale Evaluation Across Languages And Voices? on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Hugging Face has introduced the Open TTS Leaderboard, which compares text-to-speech models using automated measures of recognition accuracy, generation speed and speaker similarity. It offers results across languages and a separate listening-and-voting feature, but the project says its scores do not replace human judgments of naturalness or preference.
Hugging Face has launched the Open TTS Leaderboard, a system for comparing text-to-speech models on speech accuracy, generation speed and speaker similarity. The project aims to make evaluations faster and easier to compare across languages, while acknowledging that automated scores do not determine which voices sound most natural or which outputs listeners prefer.
The leaderboard measures speech accuracy by comparing transcripts of generated audio with the original text prompts. It uses Qwen3 automatic speech recognition and reports word error rate and character error rate. The default ranking uses macro-average word error rate across English splits from Seed TTS Eval and CV3 Eval; users can select other languages and switch to a voice-cloning view.
Speed measurements cover two use cases. In offline generation, the leaderboard reports inverse real-time factor using an H200 GPU. For streaming, it measures time to first audio on both an H200 GPU and a CPU. Voice cloning is evaluated with a speaker-similarity score based on WavLM embeddings of generated speech and reference audio.
Hugging Face says these objective evaluations can take a couple of hours, while collecting arena votes can take weeks. Its “Listen” tab lets users compare generated audio and submit preferences. Participants are asked to sign in with a Hugging Face account to help limit spam and bot submissions. The project says those votes may be incorporated into rankings later, but has not given a schedule.
Faster Comparisons for TTS Teams
A shared set of measures could help developers and researchers narrow a crowded field before committing time to listening tests or integrating a model. Hugging Face says its Hub held more than 8,000 TTS models as of September 30, 2026. A repeatable ranking may help users compare candidate models against practical constraints such as transcription accuracy, latency and voice identity preservation.
The distinction between offline speed and time to first audio is relevant to teams building streaming voice assistants and agents, where a delay before speech begins can affect how responsive a system feels. The metrics also make tradeoffs more visible: a model can perform well on accuracy but be slower, or retain a reference speaker’s qualities without ranking highly on another measure.
These results are best treated as screening evidence rather than a final recommendation. Recognition error rates depend on the speech recognizer used, and speaker-similarity scores estimate voice identity rather than expressiveness or listener satisfaction. Human listening remains necessary when those qualities matter to a deployment.
As an affiliate, we earn on qualifying purchases.
How TTS Rankings Differ
Text-to-speech systems turn written text into spoken audio, and frequent model releases make consistent comparison difficult. Existing services cited by Hugging Face, including TTS Arena v2, Artificial Analysis and Voice Arena, use listener comparisons and votes to rank outputs. Such arenas capture direct preferences, but collecting votes takes time and the systems must host the models being tested.
Hugging Face says open-weight models are underrepresented in some existing rankings. It counted 16 open-weight models among 92 on Artificial Analysis as of September 30, 2026, and reported a similar skew on Voice Arena. The company attributes the gap partly to the work involved in hosting open models and to commercial providers’ incentives to seek placement. Those figures describe the rankings it reviewed; they do not establish why every model is absent.
The new leaderboard combines automated measurements with a listening feature instead of relying on votes alone. Hugging Face names k2-fsa/OmniVoice, fishaudio/s2-pro and FunAudioLLM/Fun-CosyVoice3-0.5B-2512 as strong multilingual models. For English error rates, it lists hexgrad/Kokoro-82M, Supertone/supertonic-3 and fishaudio/s2-pro among leading models. These are results within the leaderboard’s stated tests, not a general ranking of voice quality.
“The Open TTS Leaderboard does not replace human preference ranking.”
— Hugging Face
As an affiliate, we earn on qualifying purchases.
Limits of the Published Scores
The source announcement does not provide the full model list, score tables, evaluation sample sizes or uncertainty ranges for individual results. It also does not establish how closely the automated measures track listener judgments across languages, accents and speaking styles.
Language coverage is not uniform. The leaderboard uses character error rate for Chinese, Japanese and Korean. For languages beyond English and Chinese, the announcement says Seed TTS Eval has no audio and results come from CV3 Eval alone. The effects of that difference on comparisons are not specified.
Hugging Face has not said how often rankings will be refreshed, how it will handle model or dataset changes, or when community votes might affect scores. The announcement also does not state a vote threshold or explain how subjective preferences will be combined with automated metrics. Those details matter for judging how stable and representative future rankings will be.
As an affiliate, we earn on qualifying purchases.
Listening Tests and Future Updates
Users can visit the “Listen” tab to choose a language and dataset, compare generated clips, select whether to include voice cloning and cast a preference vote. Hugging Face asks users to sign in, citing efforts to reduce spam and automated submissions. The project says it may add accumulated community votes to the leaderboard, but has not announced when or by what method.
For now, the immediate next step for readers is to compare the published automated results with audio samples for their own language and use case. Further details on ranking refreshes, evaluation coverage and vote integration would help users understand how the leaderboard will develop. Until then, its scores provide one structured comparison, with listening tests needed for questions of naturalness and preference.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does the Open TTS Leaderboard measure?
It reports word and character error rates, offline generation speed, streaming time to first audio and a speaker-similarity score for voice cloning.
Does the leaderboard identify the best-sounding voice?
No. Its automated measures cover accuracy, speed and speaker similarity, but Hugging Face says they do not replace human judgments of naturalness or preference. The separate “Listen” tab lets users compare audio and vote.
Which models does Hugging Face highlight?
For multilingual results, it names k2-fsa/OmniVoice, fishaudio/s2-pro and FunAudioLLM/Fun-CosyVoice3-0.5B-2512. For English error rates, it lists hexgrad/Kokoro-82M, Supertone/supertonic-3 and fishaudio/s2-pro among leaders. These are results for the leaderboard’s tests, not a universal judgment of quality.
How long does an evaluation take?
Hugging Face says its objective evaluation can take a couple of hours, compared with weeks for arena voting. The source does not provide a detailed timing methodology.
Will listener votes affect the rankings?
Hugging Face says community votes may be incorporated later. It has not announced a schedule, a vote threshold or how votes would be combined with automated scores.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
