Lyric accuracy
Given lyrics, does the model sing those words?
Ranking
Score out of 100, higher is better. 12 items per model; the thin line on each bar is the 95% range.
When two models' ranges overlap, this test cannot tell them apart.
Song by song
Share of the supplied lyric words that can be heard, per song; darker is better.
| Model | Score | pop | rock | folk | soul | country | hiphop | reggae | disco | ballad | indie | dance | lullaby | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | MiniMax Music 2.617 extra words (median) | 99.6 | 100 | 100 | 100 | 100 | 100 | 98 | 100 | 100 | 98 | 100 | 98 | 100 |
| 2 | Lyria 3.50 extra words (median) | 99.2 | 100 | 98 | 98 | 100 | 100 | 100 | 100 | 98 | 100 | 96 | 100 | 100 |
| 3 | Lyria 3 Pro0 extra words (median) | 98.3 | 100 | 100 | 100 | 100 | 98 | 100 | 100 | 81 | 100 | 100 | 100 | 100 |
| 4 | Lyria 3 Clip0 extra words (median) | 98 | 100 | 100 | 77 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |
| 5 | Eleven Music0 extra words (median) | 97.1 | 100 | 91 | 98 | 100 | 100 | 100 | 100 | 97 | 90 | 98 | 98 | 94 |
| 6 | ACE-Step 1.51 extra words (median) | 82.8 | 100 | 93 | 77 | 95 | 100 | 98 | 7 | 100 | 93 | 66 | 80 | 85 |
Instrumental-only models (Stable Audio 2.5, MusicGen Large (stereo)) do not sing and are left out.
What we test and how it is scored
Test. 12 short songs across styles from pop and rock to hip hop, reggae and a lullaby, each with a verse and chorus we wrote, about 55 words.
Scoring. The vocal stem is separated from the mix, and both the stem and the full mix are transcribed by two speech recognisers. Score = share of the supplied lyric words that can be heard in order, taken from the best transcript. Repeats and extra words do not lower the score; they are reported separately.
Full item list and rules: methodology.
Listen for yourself
Previews are the first 45 seconds of each track, in ranking order. More tracks are on each model's page.
Think a score is wrong? Check it.
Every test item, scoring rule and known limit is published, and the audio behind each number is on the page.