HN user
Voxtral outperforms Whisper
Can I stop you right here? Whisper is a few years old and it wasn't the best model for a long time. There are like 10 models that are smaller and faster and outperform both of them.
And these models existed before Voxtral.
Whisper has the encoder-decoder architecture, so it's hard to run streaming efficiently, though whisper-streaming is a thing.
https://kyutai.org/next/stt is natively streaming STT.
Do you know if OpenAI used the same grading criteria as official judges?
isn't he an IOI medalist? and even if he was an IMO medalist, isn't there a bit of a conflict of interests?
Do I understand it correctly that OpenAI self-proclaimed that they got their gold, without the official IMO judges grading their solutions?
There are larger models in there, a 8B and a 6B. By this logic they should be above 2B model, yet we don't see this. That's why we have open standard benchmarks, to measure this directly - not hypothesize by the models' sizes or do some cross-dataset arithmetics.
Also note that, Voxtral's capacity is not necessarily all devoted to speech, since it "Retains the text understanding capabilities of its language model backbone"
brought back competitive open source audio transcription
Bear in mind that there are a lot of very strong _open_ STT models that Mistral's press-release didn't bother to compare to, making impression they are the best new open thing since Whisper. Here is an open benchmark: https://huggingface.co/spaces/hf-audio/open_asr_leaderboard . The strongest model Mistral compared to is Scribe, ranked 10 here.
This benchmark is for English, but many of those models are multilingual (eg https://huggingface.co/nvidia/canary-1b-flash )
I assure you, before the LLM race, those research shops (DM, FAIR) had many directors that didn't contribute to any product whatsoever.
Yet your Google Research colleagues still earned way more than in academia, even without the promo.
Plus, there were quite a few places where a good publication stream did earn a promotion, without any company/business impact. FAIR, Google Brain, DM. Just not Google Research.
DeepMind didn't have any product impact for God knows how many years, but I bet they did have promos happening:)
Those are likely for Cloud, used by clients.
FAIR is considerably downscaled from what it was before, in eg 2022.
I think all FAIR researchers and engineers are aware of the NC-CA license limitations.
They still use it because releasing the code for an already approved paper under NC-CA is super easy (~ self-approving by clicking a few buttons) vs following a slow open-sourcing process needed for the MIT license (includes approvals from a sponsoring director or two, committing to support the code for at least a year, etc). Releasing under MIT can easily take a few weeks with each stage requiring finding someone responsible and chasing them across time-zones.
The best practice seems to release under NC-CA and re-license it later under MIT. Maybe this will happen here, too.