Methodology
Everything needed to check or repeat these results: the models and settings, every test item, every scoring rule, and the places where the method is weak.
Principles
- Measure, do not rate. A score is a count of something in the audio: words missing from a transcript, beats per minute, semitones of pitch change, seconds of length. There are no opinion scores for "naturalness" or "quality" here.
- Same input for every model. One fixed list of test items. Changing any item creates a new version of the list and a full re-run.
- Raw model, default settings. Each vendor's own API, its default or flagship voice, one request per item, nothing added in between. Two house rules apply to every model: a documented text-normalisation switch is turned on, and a requested emotion is passed through the model's own documented control.
- Publish the evidence. Clips are on the model pages, including the failures and what the recogniser heard.
- Validate the measuring tools. Before a measurement was used, it was checked against a case with a known answer. Three were replaced after failing that check (see "What we tried and dropped").
Speech models tested
| Model | API model | Voice | Reached through | Emotion control used |
|---|---|---|---|---|
| Qwen3 TTS Alibaba | qwen3-tts | Serena | Replicate | Text instruction |
| AnyToSpeech Standard voices AnyToSpeech | site pipeline on gemini-3.8-flash-tts | Kore | AnyToSpeech pipeline | Text instruction |
| AnyToSpeech Premium and Characters AnyToSpeech | site pipeline on inworld-tts-2 | Ashley | AnyToSpeech pipeline | Text instruction |
| Eleven v3 ElevenLabs | eleven_v3 | Sarah | Replicate | Inline tag |
| Eleven Flash v2.5 ElevenLabs | eleven_flash_v2_5 | Sarah | Replicate | None offered |
| Eleven Multilingual v2 ElevenLabs | eleven_multilingual_v2 | Sarah | Replicate | None offered |
| Gemini 3.8 Flash-Lite TTS | gemini-3.8-flash-lite-tts | Kore | Gemini API | Text instruction |
| Gemini 3.8 Flash TTS | gemini-3.8-flash-tts | Kore | Gemini API | Text instruction |
| Gemini 2.5 Pro TTS | gemini-2.5-pro-preview-tts | Kore | Gemini API | Text instruction |
| Kokoro 82M Hexgrad (open source) | kokoro-82m | af_bella | Replicate | None offered |
| Inworld TTS-2 Inworld | inworld-tts-2 | Ashley | Inworld API | Text instruction |
| Inworld TTS-1.5 Max Inworld | inworld-tts-1.5-max | Ashley | Inworld API | None offered |
| MiniMax Speech 2.8 HD MiniMax | speech-2.8-hd | English_Wiselady | Replicate | Emotion setting |
| MiniMax Speech 2.8 Turbo MiniMax | speech-2.8-turbo | English_Wiselady | Replicate | Emotion setting |
| GPT-4o mini TTS OpenAI | gpt-4o-mini-tts | alloy | OpenAI API | Text instruction |
| TTS-1 OpenAI | tts-1 | alloy | OpenAI API | None offered |
| TTS-1 HD OpenAI | tts-1-hd | alloy | OpenAI API | None offered |
| Chatterbox Multilingual Resemble AI (open source) | chatterbox-multilingual | default | Replicate | None offered |
| Grok TTS xAI | grok-text-to-speech | eve | Replicate | Inline tag |
Transcripts come from two recognisers (whisper-1 and gpt-4o-transcribe); the transcript with fewer errors is used for each clip. Models reached through Replicate run on Replicate's hosted version of the vendor's model.
Speech tests and scoring
Word accuracy 25% of overall
Test: 60 everyday English sentences with no numbers or unusual names.
Scoring: Each clip is transcribed by two speech recognisers and compared word for word with the text. We count missing words, added words and wrong words, and keep the transcript with fewer errors so a recogniser mistake is not blamed on the voice. Score = 100 minus the word error rate.
Numbers, dates and symbols 20% of overall
Test: 60 sentences, each with one trap: $1,250.50, 03/14/2025, 7:45 PM, 12.5%, -5°C, Dr. and St., an email address, a URL, a flight number, a version number and so on.
Scoring: A sentence passes when the transcript shows the item was spoken correctly, using a published list of accepted readings for each item. Score = share of the 60 that pass. Where a model has a documented text-normalisation switch, it is turned on.
Long text 15% of overall
Test: One 15,000-character document, sent at five lengths: about 1,000, 2,000, 4,000, 8,000 and 15,000 characters. Each length is a single request with no splitting on our side, so every model is pushed up to and past its own limit.
Scoring: Each recording is transcribed in one-minute pieces and compared word for word with the text sent. A length scores its word accuracy; a length the model rejects, cuts short or garbles scores what comes back, which is 0 when nothing usable does. Score = average across the five lengths. "Longest clean read" is the largest length read with at least 97% of words correct.
Repeated words 10% of overall
Test: 20 lines built on repetition: "No, no, no, no, no. I said no.", a line that repeats a whole sentence, a spoken code word given twice.
Scoring: Word-by-word comparison. Speech recognisers themselves drop repeated sentences, so each clip is also cut at its sentence pauses and every piece is transcribed separately; the best transcript counts.
Other languages 15% of overall
Test: 8 sentences each in Spanish, French, German, Italian, Portuguese, Hindi, Japanese and Mandarin, read with the model's default voice (or its voice for that language when the model requires one).
Scoring: Word error rate per language (character error rate for Japanese and Mandarin). Score = average accuracy across the languages the model supports; the number of supported languages is shown next to it.
Emotion control 15% of overall
Test: 10 neutral sentences, each requested angry, excited, scared and whispered, through the model's own documented control (instruction field, inline tag or emotion setting). Models with no such control are rendered plain and scored the same way.
Scoring: Measured against the same model's neutral reading of the same sentence. Angry, excited and scared pass when median pitch rises by at least 2 semitones; a whisper passes when voiced sound drops by at least half. This checks that the delivery changed in the right direction. It does not prove a listener would name that exact emotion; blind listener voting is planned for that.
Reliability not in overall
Test: Every request in the benchmark except the long-text ladder, about 255 per model.
Scoring: A request is clean when the first attempt returns audio that is not silent. Rate-limit responses caused by the test itself are retried and not counted. Failures are retried once; an item that fails twice counts as fully wrong in its own category.
Speed not in overall
Test: The 60 word-accuracy sentences.
Scoring: Median generation time, and audio length divided by generation time ("times real time"). For models called directly this is the full request time. For models reached through Replicate it is the compute time Replicate reports, which leaves out queueing. The two are not directly comparable, so speed is shown but never counted in the overall score.
A request that fails twice counts as fully wrong in its category. Differences under about 2 points between models are within the noise of the recognisers and should be read as a tie.
Speech test items
Word accuracy (60 sentences)
- The ferry leaves the harbor every morning just after sunrise, and most of the passengers are still half asleep.
- She folded the letter twice, slipped it into her coat pocket, and walked out without saying a word.
- If the weather holds, we should reach the summit before noon and be back at camp by dark.
- Nobody in the village could remember a winter as long or as cold as that one.
- The engineer checked every bolt on the bridge before she allowed the first truck to cross.
- He opened the old book carefully, because the pages were thin and the binding had started to crack.
- Our neighbors planted tomatoes, peppers, and basil along the fence behind their garage.
- When the lights went out, the children laughed and reached for the candles on the shelf.
- The committee agreed to postpone the vote until every member had read the full report.
- A small brown dog followed us all the way from the bakery to the edge of the park.
- You can leave your shoes by the door and hang your jacket on the hook beside the mirror.
- The museum stays open late on Fridays, which is the only time I can visit after work.
- Although the recipe looked simple, the bread came out of the oven flat and far too salty.
- They spent the whole afternoon repairing the roof before the next storm rolled in from the coast.
- Please remind your brother that the library books are due back before the weekend.
- The pilot announced a short delay, thanked everyone for their patience, and switched off the microphone.
- I would rather take the slow train through the mountains than fly and miss the view.
- Her grandmother taught her to knit when she was seven, and she has never stopped since.
- The river was too wide to swim across, so we walked upstream until we found a bridge.
- After the concert, the musicians packed their instruments and drove through the night to the next city.
- The surgeon explained the procedure twice and then asked whether the family had any questions.
- A thin layer of frost covered the windshield, so he scraped it clear with an old plastic card.
- Every spring the orchard fills with bees, and the farmer walks the rows to count the blossoms.
- She had rehearsed the speech for weeks, yet her hands still shook when she reached the stage.
- The detective noticed that the kitchen clock had stopped at exactly the moment the power failed.
- We rented a small cabin by the lake and spent the evenings reading beside the fire.
- The architect proposed a taller building, but the council rejected the plan after a long debate.
- His youngest daughter wants to study marine biology and spend her life near the ocean.
- The bakery on the corner sells out of fresh bread long before most people wake up.
- Before leaving the laboratory, the students washed the glassware and labeled each sample.
- The orchestra tuned for a moment, the conductor raised his arms, and the hall fell silent.
- I forgot my umbrella at the restaurant, and by the time I returned it had vanished.
- The mechanic wiped his hands, shook his head, and told us the engine could not be saved.
- Thousands of birds gather in the marsh each autumn before they begin the journey south.
- She translated the entire novel herself because no publisher was willing to pay for it.
- The lawyer argued that the contract was invalid because nobody had signed the final page.
- On quiet nights you can hear the freight trains crossing the valley from our porch.
- The teacher divided the class into groups and gave each one a different chapter to present.
- He paid for the groceries, carried the bags to the car, and realized he had lost his keys.
- Scientists believe the volcano has been dormant for centuries, though small tremors continue.
- The gardener trimmed the hedge so evenly that it looked like a green stone wall.
- My cousin moved abroad last year and now sends long letters instead of calling.
- The judge listened to both sides, reviewed the evidence, and delivered a brief verdict.
- A sudden gust of wind lifted the tent from the ground and carried it across the field.
- The nurse dimmed the lights, checked the monitor once more, and quietly closed the door.
- They built the bridge from local timber, and it has carried travelers for a hundred years.
- The photographer waited an hour for the clouds to part and still missed the perfect shot.
- Because the elevator was broken, we climbed eleven flights of stairs with all our luggage.
- The author rarely gives interviews, preferring to let the books speak for themselves.
- An old fisherman showed us how to mend the net using a wooden needle and waxed thread.
- The company promised faster delivery, but most customers noticed no difference at all.
- She planted an oak tree the day her son was born, and now it shades the entire yard.
- The children built a fortress from cardboard boxes and defended it until dinner.
- Our guide warned us that the trail becomes slippery whenever the fog settles in.
- The pharmacist double checked the prescription and explained how often to take each tablet.
- He whispered the answer to his friend, but the whole room heard him anyway.
- The festival begins with a parade through the old town and ends with fireworks over the river.
- After months of drought, the first heavy rain turned the dusty road into a stream.
- The carpenter measured the doorway three times before he made a single cut.
- I have always admired people who can stay calm when everything around them is falling apart.
Numbers, dates and symbols (60 sentences)
- The total comes to $1,250.50 after tax.
- The city has 3,482,000 residents.
- She was born in 1987 and moved here in 2004.
- The meeting is on 03/14/2025.
- The train leaves at 7:45 PM.
- Sales grew by 12.5% last quarter.
- Call us at 555-0142.
- Dr. Patel's office is at 40 Elm St.
- The package weighs 2.5 kg and is 30 cm wide.
- Send it to help@example.com.
- Visit www.example.org/start for details.
- He finished 3rd in the 21st race.
- It was -5°C this morning.
- NASA and the FBI issued a joint statement.
- Pi is roughly 3.14159.
- Tickets cost €45 each.
- Flight BA2490 departs from gate C14.
- The ratio is 3:1 and the pH is 7.4.
- The company raised $4.2M in 2023.
- He ran 26.2 miles in 3 hrs 15 min.
- Rent is $2,300 a month.
- A stamp costs $0.73.
- The fee is £18.99.
- The budget is $3.5 billion.
- She earns $85K a year.
- The deadline is Jan. 5, 2026.
- Music from the 1990s is back.
- It was built in the 18th century.
- The store opens at 09:30.
- We land at 12:05 a.m.
- The video is 1 hr 20 min long.
- Dial 1-800-555-0199 for support.
- The limit is 65 mph.
- Bake at 350°F for 25 min.
- The trail is 8.4 km long.
- Prices rose 3-5% this year.
- 5 + 7 = 12.
- Prof. Lee and Sen. Ortiz spoke first.
- Martin Luther King Jr. Blvd. is closed.
- Read chapter No. 9 tonight.
- They climbed Mt. Everest in May.
- Acme Inc. bought Zenith Ltd.
- J. K. Rowling wrote it.
- The CEO met the UN and the EU.
- Save the file as a PDF or a JPEG.
- Connect to Wi-Fi using WPA2.
- Update to version 2.0.1 today.
- It runs iOS 18 and Windows 11.
- Her birthday is the 22nd of June.
- The video has 1.2M views.
- About 15,000 people came.
- This is the 100th episode.
- The rate fell to 0.25%.
- Use promo code SAVE20 at checkout.
- The room number is 12B.
- He ruled from 1837 to 1901.
- Open 9 a.m. to 5 p.m. daily.
- Add 8 oz of butter.
- Winds reached 120 km/h.
- The phone has 256GB of storage.
Repeated words (20 lines)
- No, no, no, no, no. I said no.
- Row, row, row your boat, gently down the stream. Row, row, row your boat, gently down the stream.
- The code word is tango tango seven seven tango. I repeat: tango tango seven seven tango.
- Very, very, very good. Really, really, really, really good.
- Again and again and again and again and again, the waves hit the shore.
- She sells sea shells. She sells sea shells. She sells sea shells by the sea shore.
- Left, right, left, right, left, right, left. Keep marching, keep marching, keep marching.
- It was a long, long, long, long road, and a long, long, long way home.
- Go, go, go, go! Run, run, run!
- Please hold. Please hold. Please hold. Your call is important to us.
- Knock, knock, knock. Nobody answered. Knock, knock, knock. Still nobody answered.
- Yes, yes, yes, a thousand times yes.
- Never, never, never, never give up.
- Check one, check two. Check one, check two. Check one, check two.
- He said stop. She said stop. They all said stop. Nobody stopped.
- Down, down, down the rabbit hole she fell, down, down, down.
- Wait. Wait. Wait. Did you hear that? Wait.
- More and more and more people arrived, and more and more chairs were needed.
- Thank you, thank you, thank you all so much.
- Step by step by step by step, we climbed the tower.
Emotion control (10 sentences × neutral, angry, excited, scared, whispered)
- The package arrived at the front door this morning.
- I am going to the store, and I will be back at six.
- They told me the meeting has been moved to Thursday.
- There are two cars parked outside the building.
- She put the keys on the table next to the window.
- The report is due at the end of the month.
- We are having dinner at my sister's house tonight.
- He said the train leaves from platform four.
- The lights in the hallway are still on.
- Someone left a message for you at the front desk.
Other languages (8 sentences each)
- es El tren sale de la estación a primera hora de la mañana y llega a la costa antes del mediodía.
- es Mi abuela siempre decía que la paciencia es la madre de todas las ciencias.
- es Si mañana no llueve, iremos caminando hasta el mercado del pueblo.
- es Los niños jugaban en la plaza mientras sus padres tomaban café en la terraza.
- es Nunca había visto una ciudad tan tranquila a esas horas de la noche.
- es El médico me recomendó descansar y beber mucha agua durante toda la semana.
- es Compramos pan, queso y fruta para comer junto al río.
- es Cuando terminó la película, nadie se levantó de su asiento.
- fr Le boulanger ouvre sa boutique avant le lever du soleil, et tout le quartier sent le pain chaud.
- fr Nous avons marché longtemps le long de la rivière sans rencontrer personne.
- fr Elle a rangé ses livres, fermé la fenêtre et quitté la maison sans bruit.
- fr Les enfants attendaient le bus sous la pluie en chantant une vieille chanson.
- fr Mon frère travaille dans un petit restaurant près de la gare.
- fr Il faut tourner à gauche après le pont, puis continuer tout droit.
- fr Je me souviens encore du jardin de ma grand-mère et de ses grands arbres.
- fr Le train avait du retard, alors nous avons pris un café en attendant.
- de Der Zug fährt jeden Morgen pünktlich ab, aber heute hatte er eine halbe Stunde Verspätung.
- de Meine Schwester wohnt seit vielen Jahren in einer kleinen Stadt an der Küste.
- de Wenn das Wetter schön bleibt, gehen wir am Wochenende in den Bergen wandern.
- de Die Kinder spielten im Garten, während ihre Eltern das Abendessen vorbereiteten.
- de Ich habe meinen Schlüssel verloren und musste den Nachbarn um Hilfe bitten.
- de Der Arzt sagte, ich solle viel Wasser trinken und früh schlafen gehen.
- de Nach dem Regen roch die ganze Straße nach frischer Erde.
- de Wir haben den ganzen Abend über alte Zeiten gesprochen und viel gelacht.
- it Ogni domenica la famiglia si riunisce intorno al tavolo per un pranzo che dura tutto il pomeriggio.
- it Il vecchio pescatore conosceva il mare meglio di chiunque altro nel villaggio.
- it Abbiamo perso il treno, quindi siamo tornati a casa a piedi sotto la pioggia.
- it I bambini correvano nella piazza mentre le campane suonavano mezzogiorno.
- it Mia sorella lavora in una piccola libreria vicino alla stazione.
- it Dopo cena siamo usciti a fare una passeggiata lungo il fiume.
- it Non avevo mai visto un tramonto così bello sulla collina.
- it Il medico mi ha detto di riposare e di bere molta acqua.
- pt O mercado abre cedo, e os vendedores arrumam as frutas antes de o sol nascer.
- pt Ela guardou a carta na gaveta e nunca mais falou sobre o assunto.
- pt Se tivermos tempo, vamos visitar a aldeia onde os meus avós nasceram.
- pt As crianças brincavam na praia enquanto os pais conversavam à sombra.
- pt O meu irmão trabalha num pequeno café perto da estação.
- pt Depois do jantar, fomos caminhar ao longo do rio.
- pt Nunca tinha visto uma cidade tão calma àquela hora da noite.
- pt O médico disse que eu devia descansar e beber muita água.
- hi सुबह की ट्रेन समय पर आई और हम दोपहर से पहले शहर पहुँच गए।
- hi मेरी दादी हर शाम बच्चों को पुरानी कहानियाँ सुनाती थीं।
- hi अगर कल बारिश नहीं हुई तो हम बाज़ार तक पैदल जाएँगे।
- hi बच्चे पार्क में खेल रहे थे और उनके माता पिता चाय पी रहे थे।
- hi मेरा भाई स्टेशन के पास एक छोटी दुकान में काम करता है।
- hi डॉक्टर ने कहा कि मुझे आराम करना चाहिए और बहुत पानी पीना चाहिए।
- hi खाने के बाद हम नदी के किनारे टहलने गए।
- hi मैंने इतना सुंदर शहर पहले कभी नहीं देखा था।
- ja 朝の電車はいつも混んでいますが、今日は珍しく座ることができました。
- ja 祖母は毎年春になると、庭にたくさんの花を植えます。
- ja 雨が止んだら、川沿いの道を駅まで歩きましょう。
- ja 子供たちは公園で遊んでいて、両親はベンチで話していました。
- ja 兄は駅の近くの小さな店で働いています。
- ja 医者は、よく休んで水をたくさん飲むように言いました。
- ja 夕食の後、私たちは川のそばを散歩しました。
- ja こんなに美しい町は今まで見たことがありません。
- zh 早上的火车准时出发,我们在中午之前就到了海边。
- zh 奶奶每天晚上都会给孩子们讲一个老故事。
- zh 如果明天不下雨,我们就走路去镇上的市场。
- zh 孩子们在公园里玩,父母坐在长椅上聊天。
- zh 我哥哥在火车站附近的一家小店工作。
- zh 医生说我应该好好休息,多喝水。
- zh 晚饭以后,我们沿着河边散步。
- zh 我从来没有见过这么漂亮的城市。
Long text (one document, five lengths)
Lengths sent, in characters: 895, 1,875, 3,979, 7,938, 14,958. Each is the start of the same document, cut at the end of a sentence, and sent as a single request.
The full document (14,958 characters)
Music models tested
| Model | API model | Reached through | Sings | Controls used |
|---|---|---|---|---|
| Stable Audio 2.5 Stability AI | stable-audio-2.5 | Replicate | no | Length field |
| Eleven Music ElevenLabs | music_v1 | Replicate | yes | Length field, instrumental switch |
| ACE-Step 1.5 ACE Studio / StepFun (open source) | ace-step-1.5 | Replicate | yes | Lyrics, tempo and length fields |
| Lyria 3 Clip | lyria-3-clip-preview | Gemini API | yes | Prompt only (fixed ~30 s clips) |
| Lyria 3.5 | lyria-3.5 | Gemini API | yes | Prompt only |
| Lyria 3 Pro | lyria-3-pro-preview | Gemini API | yes | Prompt only |
| MusicGen Large (stereo) Meta (open source) | musicgen-stereo-large | Replicate | no | Length field |
| MiniMax Music 2.6 MiniMax | music-2.6 | Replicate | yes | Lyrics field, instrumental switch |
Not tested yet: Suno (no public api), Udio (no public api). Every model gets the same prompt text; where a model has dedicated fields for lyrics, tempo, length or "instrumental", those are filled in as well.
Music tests and scoring
Lyric accuracy 30% of overall
Test: 12 short songs across styles from pop and rock to hip hop, reggae and a lullaby, each with a verse and chorus we wrote, about 55 words.
Scoring: The vocal stem is separated from the mix, and both the stem and the full mix are transcribed by two speech recognisers. Score = share of the supplied lyric words that can be heard in order, taken from the best transcript. Repeats and extra words do not lower the score; they are reported separately.
Genre 20% of overall
Test: 24 instrumental briefs across 12 genres: lo-fi hip hop, country, funk, techno, punk rock, drum and bass, reggae, disco, heavy metal, jazz, bossa nova and trap.
Scoring: Two AI listeners each pick one label from a fixed list of 16 genres without seeing the prompt. A clip passes when either names the requested genre. Picking the right one of 16 by accident is unlikely, so a pass is strong evidence; a fail can also be a listener error.
Tempo 20% of overall
Test: The same 24 briefs. Each genre is asked for at two different tempos, from 75 to 180 BPM, so a model cannot pass by playing the genre at its usual speed.
Scoring: We measure the beat grid directly from the audio. A clip passes when the grid at the requested tempo, or its half or double, is at least 90% as strong as the strongest grid found at any tempo. Half and double time count because the two are musically the same pulse.
Length 10% of overall
Test: The same 24 briefs, each asking for a 30 second track. Models with a length setting have it set to 30.
Scoring: Pass when the track is within 4.5 seconds of 30.
No vocals when asked 10% of overall
Test: The same 24 briefs, all marked instrumental.
Scoring: The vocal stem is separated. A clip fails only when that stem is loud and contains at least 8 recognisable sung words.
Clean ending 10% of overall
Test: Every track in the benchmark.
Scoring: Pass when the last 0.4 seconds are at least 12 dB quieter than the track as a whole (a fade or a final note). A track that is cut off mid-phrase fails.
Speed not in overall
Test: The 24 instrumental briefs.
Scoring: Median seconds to generate. Direct calls are full request time; Replicate-hosted models use the compute time Replicate reports. Not part of the overall score.
Instrumental briefs
- Lo-fi hip hop beat with mellow electric piano, soft drums and vinyl crackle. 80 BPM. Instrumental only, no vocals. 30 seconds long.
- Country tune with strummed acoustic guitar, fiddle and brushed drums. 95 BPM. Instrumental only, no vocals. 30 seconds long.
- Funk groove with slap bass, tight rhythm guitar and a horn section. 110 BPM. Instrumental only, no vocals. 30 seconds long.
- Techno track with a four-on-the-floor kick drum and an acid bassline. 130 BPM. Instrumental only, no vocals. 30 seconds long.
- Punk rock with distorted power chords and fast, energetic drums. 160 BPM. Instrumental only, no vocals. 30 seconds long.
- Drum and bass with fast breakbeats and a deep sub bass. 174 BPM. Instrumental only, no vocals. 30 seconds long.
- Lo-fi hip hop beat with a dusty jazz guitar loop, soft kick and tape hiss. 92 BPM. Instrumental only, no vocals. 30 seconds long.
- Upbeat country tune with banjo, pedal steel guitar and a train-beat snare. 120 BPM. Instrumental only, no vocals. 30 seconds long.
- Slow funk groove with clavinet, wah guitar and a syncopated bass line. 98 BPM. Instrumental only, no vocals. 30 seconds long.
- Hard techno track with a pounding kick drum, hi-hats and dark synth stabs. 140 BPM. Instrumental only, no vocals. 30 seconds long.
- Fast punk rock with buzzing guitars, driving bass and crashing cymbals. 180 BPM. Instrumental only, no vocals. 30 seconds long.
- Liquid drum and bass with rolling breakbeats, warm pads and a smooth sub bass. 165 BPM. Instrumental only, no vocals. 30 seconds long.
- Reggae groove with offbeat guitar chops, a heavy bass line and a one-drop drum beat. 75 BPM. Instrumental only, no vocals. 30 seconds long.
- Bright reggae track with organ bubble, offbeat guitar and a steady rimshot. 90 BPM. Instrumental only, no vocals. 30 seconds long.
- Disco track with a four-on-the-floor beat, octave bass, string stabs and open hi-hats. 120 BPM. Instrumental only, no vocals. 30 seconds long.
- Laid-back disco groove with rhythm guitar, electric piano and hand claps. 110 BPM. Instrumental only, no vocals. 30 seconds long.
- Heavy metal with palm-muted distorted guitar riffs and double bass drums. 150 BPM. Instrumental only, no vocals. 30 seconds long.
- Mid-tempo heavy metal with a crushing guitar riff, bass and pounding drums. 120 BPM. Instrumental only, no vocals. 30 seconds long.
- Swing jazz with walking upright bass, ride cymbal, piano and a saxophone lead. 140 BPM. Instrumental only, no vocals. 30 seconds long.
- Relaxed jazz trio with brushed drums, upright bass and piano. 100 BPM. Instrumental only, no vocals. 30 seconds long.
- Bossa nova with nylon-string guitar, soft shaker and a gentle rim click. 100 BPM. Instrumental only, no vocals. 30 seconds long.
- Lively bossa nova with nylon-string guitar, flute and light percussion. 125 BPM. Instrumental only, no vocals. 30 seconds long.
- Trap beat with booming 808 bass, rapid hi-hat rolls and a dark bell melody. 150 BPM. Instrumental only, no vocals. 30 seconds long.
- Trap beat with deep 808 bass, snappy snare and a moody piano loop. 130 BPM. Instrumental only, no vocals. 30 seconds long.
Genre labels offered to the AI listeners: lo-fi hip hop, country, funk, techno, punk rock, drum and bass, reggae, disco, heavy metal, jazz, bossa nova, trap, classical, ambient, folk, blues.
Songs with lyrics
Upbeat pop song with a clear female lead vocal, bright synths and a steady beat
Driving rock song with a clear male lead vocal, electric guitars and live drums
Gentle acoustic folk song with a clear lead vocal and fingerpicked guitar
Slow soul ballad with a clear, warm lead vocal, electric piano and soft drums
Country song with a clear male lead vocal, acoustic guitar and pedal steel
Hip hop track with a clear rapped male vocal over a boom bap drum beat and a piano loop
Reggae song with a clear relaxed lead vocal, offbeat guitar and a deep bass line
Disco song with a clear female lead vocal, four-on-the-floor beat, strings and funky bass
Slow piano ballad with a clear, emotional female lead vocal and soft strings
Indie rock song with a clear male lead vocal, jangly guitars and a steady drum beat
Electronic dance pop song with a clear female lead vocal, pulsing synths and a strong kick drum
Soft lullaby with a clear, gentle female lead vocal, music box and quiet acoustic guitar
Known limits
- Ranges, not just scores. Every score carries a 95% range. For pass-or-fail tests it is a Wilson interval; for word-accuracy tests it comes from resampling the items. When two models' ranges overlap, the test cannot separate them.
- One voice, one take. Each speech model is tested with a single default voice and each item is rendered once. Another voice or another take can differ. Three voices and three takes per item are planned.
- The music test is still modest. 24 instrumental briefs and 12 songs per model. One track moves a score by 4 to 8 points; the range shown on each score says how much to trust it.
- It does not measure how good a voice sounds. Naturalness, warmth and audio fidelity need human listeners. Nothing here should be read as "sounds best".
- Emotion control is a narrow measurement. It shows that pitch or voicing moved in the expected direction against the model's own neutral reading. A model that expresses anger through intensity alone, without raising pitch, will score low. It cannot tell angry from excited.
- Recognisers make mistakes. Using two and keeping the better transcript removes most of them, and repeated sentences are transcribed piece by piece, but a small error floor remains, larger outside English.
- Speed depends on where the model is hosted. Direct calls and Replicate-hosted calls are timed differently and are not strictly comparable.
- Models change. Vendors revise models behind the same name. Each result carries its test date.
What we tried and dropped
- AI listeners for emotion. We asked two audio-capable AI models, in five different arrangements, to name the emotion in each clip. They disagreed with each other, one answered "neutral" for almost everything, and they reported whispering from models that cannot whisper. Those results were discarded and replaced by direct acoustic measurement.
- Whole-clip transcripts for repeated sentences. Both recognisers silently dropped a sentence when it repeated the one before, which made accurate voices look like they had skipped text. Clips are now cut at sentence pauses and transcribed piece by piece.
- Guessing the tempo. Estimating a track's tempo with no hint confuses related tempos (a 174 BPM track read as 116). We now test whether a beat grid exists at the requested tempo.
Conflict of interest, and our own rows
AnyToSpeech is a product that sells access to some of the models tested here, including voices from Google, Inworld and OpenAI, and music from Google. No vendor paid for, saw or influenced these results.
The speech leaderboard also includes AnyToSpeech itself, marked "ours", as two rows: Standard voices and Premium and Characters. AnyToSpeech does not make a voice model. Standard voices run on Gemini 3.8 Flash TTS and Premium and Character voices run on Inworld TTS-2; both go through the site's own pipeline, which splits long text at sentence ends and joins the pieces, adds number-reading hints, detects the language, and retries or falls back when a piece fails. The pipeline was called exactly as the site calls it, with no prompt, voice or setting chosen for this test, and it is scored by the same code as every other row. Because it splits and retries internally, its long-text and reliability results describe the product, not a single model request.
The defence against bias is the method: fixed inputs, measured outputs, published rules, and audio you can check.
What is coming
- Blind listener voting: two clips, no names, pick the more natural one. This is the only sound way to rank how good a voice or song sounds.
- A harder word test (rare names, tongue twisters, mixed-language sentences), since ordinary sentences no longer separate models.
- Voice cloning: how close the clone is to the reference speaker.
- Price per hour of audio, at published list prices.
- More languages, three voices per model, three takes per item.
- A larger music test, plus key, instrument and structure checks.
- Re-runs on a fixed schedule, with score history per model.
To have a model added, or to dispute a result, contact us.
Think a score is wrong? Check it.
Every test item, scoring rule and known limit is published, and the audio behind each number is on the page.