What it takes to build a speech-to-text benchmark for Slovak

For English, that answer comes from the Open ASR Leaderboard. For Slovak, we therefore built the Slovak ASR Benchmark, now available on SLAIH. This post, however, is not about its results. We want to show what it actually takes to build a benchmark you can trust, and why the process is harder and more interesting than it looks.

Why audio is harder than text

Benchmarking language models on text is not trivial, but speech recognition adds another layer of difficulty that begins with the data itself. Labeled Slovak speech is scarce. Every hour of audio needs a verified human transcript, and transcription cost scales linearly with duration. There are no shortcuts around human effort: corpora with pseudo-labels (transcripts generated by an ASR model rather than a human) are useless for evaluation, because the reference would be the output of another model, with all of its errors.

The less obvious problem is that even a human transcript does not have a single correct form. Here are a few examples that come up throughout our data:

• Dates and numbers. A speaker says a date. The transcript can read "5. 3. 2026", "piateho marca 2026", or "piateho marca dvetisíc dvadsaťšesť". All are correct. And because Slovak inflects numerals, you cannot fix this by mechanically converting digits into words: "2" alone can appear as "dva", "dve", "dvoch" or "dvom" depending on context.
• Abbreviations and domain jargon. The speaker says "spoločnosť s ručením obmedzeným", the transcriber writes "s.r.o.". In telco recordings, domain terms appear in both formal and colloquial variants. Which form counts as correct?
• Hesitations and repetitions. A speaker stumbles and repeats a word. One transcription convention keeps the repetition, another removes it. A model that faithfully transcribes what was said gets a worse score against a cleaned reference, and vice versa.

This is why text normalization is not just a preprocessing detail. It is a key methodological decision of the benchmark: the normalizer defines what counts as an error and what does not. Two benchmarks with identical data but different normalizers will produce different numbers and can produce different model rankings. We treat our normalization rules as part of the methodology and apply them identically to every model's output and to every reference.

One data source is not enough

The most important thing to keep in mind when building a benchmark is that a single number captures a model's ability only when it rests on representative data. A model's error rate depends on many conditions: the acoustic environment, the recording setup, the style of speech, and the speaker's vocabulary. Any test set is a sample from this multidimensional space. If we build the whole benchmark on a single data source, we describe only a small and unrepresentative slice of that space. The result can be very precise if all the data is similar, and that is exactly what makes it deceptive: a precise number looks trustworthy even though it speaks only about a narrow slice. That is why we look for a test set that covers the space of "all possible" Slovak speech recordings as well as possible.

Our results show how misleading this can be. On read speech, Omnilingual ASR (omniASR) LLM 7B and Whisper large v3 are practically tied, at around 19 to 20 percent word error rate. If you evaluated only on read-speech corpora like Common Voice or FLEURS, which is exactly what most multilingual evaluations do, you would conclude the two models are equivalent. On spontaneous speech, however, omniASR drops to 29 percent while Whisper holds at 18. The models are not equivalent at all. One of them simply happens to fit the part of the space you sampled better.

The same effect appears within a single model. GPT-4o Transcribe ranks among the top three models on customer service calls, yet sits in the middle of the table overall and reaches 31 percent word error rate on municipal council recordings. Even the strongest model in our benchmark, Google Chirp 3, spans more than a fourfold range across domains, from about 3 percent on scripted customer service dialogues to 14 percent on municipal recordings. A single aggregate number hides all of this.

The way results are aggregated is itself an important decision with real consequences. If one large corpus dominates the test set, a naive average is really just a benchmark of that one corpus. An earlier version of our evaluation set had exactly this problem: one source made up the large majority of samples. The current version is built by stratified sampling across domain, gender and speech type, so that no single source dictates the final number and every reported dimension rests on a meaningful amount of data.

The dimensions that matter

The benchmark currently breaks performance down along several axes: seven domains (from parliamentary and municipal recordings through customer service dialogues to read literature), speech type (spontaneous or read), speaker gender, and acoustic conditions. For the last of these, two thirds of the evaluation audio is augmented with five types of distortion (background noise, background speech, white noise, packet loss, phone channel noise) at two intensity levels. This is a deliberate choice: our overall results include degraded audio, so they are intentionally harder than clean-speech results and not directly comparable to figures reported by vendors. Real users call from noisy streets, not from recording studios.

We chose the metrics specifically for Slovak. Word error rate is the standard, but Slovak is morphologically rich, so many word errors are just a single wrong character in an inflectional ending. That is why we report character error rate alongside it: Chirp 3, for example, has roughly 11 percent WER but under 5 percent CER, and that gap is mostly morphology, not a misunderstanding of the audio. For applications where numbers matter, we add a digit sequence error rate, which treats a whole sequence, such as a phone number, as wrong if even a single digit is off. The best models fail on roughly one in five digit sequences, and most fail even more often, which anyone automating call center workflows should keep in mind.

We are equally open about what the benchmark does not yet cover: speaker age, speech tempo, regional dialects, and real (rather than synthetic) acoustic environments. Synthetic augmentation is a useful proxy, but it has its limits. It cannot reproduce reverberation or overlapping speakers, and above all it cannot capture how people actually speak in noise: in loud environments speakers involuntarily raise their voice, slow down and shift pitch, a phenomenon known as the Lombard effect. Adding noise to a clean studio recording gives you noisy audio of calm speech, which is not the same as speech produced in a noisy place.

The tip of the iceberg

The table on SLAIH today is an overview: aggregate metrics over the full evaluation set. Behind it sit the per-domain, per-speech-type and per-noise breakdowns discussed above, and we plan to publish more of these views over time. Some findings already point beyond model selection. Scale alone does not solve data scarcity: within the omniASR family, going from 300 million to 7 billion parameters improves WER only from 33 to 26 percent, still behind the much smaller Whisper large v3 at 19 percent. Meanwhile, our internal experiments with fine-tuning models on Slovak data roughly halve error rates across architectures. So for Slovak, the bottleneck is data, not model capacity. We are therefore preparing a separate benchmark for fine-tuned models. We do not include them here, because they were fine-tuned on data from sources and domains similar to the test set (even though they never saw the specific test recordings), and this distributional overlap gives them an advantage that would distort a direct comparison.

A benchmark of this scope is not something one team builds alone, and we are reminded of this again and again in our work. SkMTEB, the Slovak text embedding benchmark accepted to ACL 2026, grew from a collaboration between KInIT and academic partners at Technical University of Košice and Comenius University. This speech benchmark began as a collaboration with an industry partner, and its next iteration is taking shape together with academic partners who contribute data and scientific expertise. Different projects, different partners, same lesson: for a language the size of Slovak, cooperation between research institutions, universities and companies is the only way world-class language resources get built.

If you have Slovak speech data, or a model you want to see evaluated, get in touch: contact@slaih.sk.