OpenAI quietly launched Whisper V2 in a GitHub commit

https://github.com/openai/whisper/commit/4179ed2475cc84cba66868b516232ef1b74dacdf

by fudged71 • 4 years ago

335 124 4 years ago

amelius2• 4 years ago

How can the Word Error Rate (WER) be larger than 100% for some languages?

lunixbochs3• 4 years ago

If the target is the words "A A A" and you produce "B B B B", you have more errors than there were words in the target. 3 replacements and 1 insertion.

iKlsR6• 4 years ago

I've been using whisper to get transcripts from my local radio stations. I know it's out of scope for the original project but I hope someone can build a streaming input around it in the future. I currently pipe in and save 10 minute chunks that get sent off for processing.

rexreed• 4 years ago

What processing / server / backend are you using to run the whisper model?

iKlsR6• 4 years ago

Stupid simple setup until I streamline the process some more. Just running on my pc (i7 8700k, gtx 1080) and using ffmpeg for copying the stream to disk and a simple python script that creates the chunks and transcripts.

ortusdux• 4 years ago

I would love to do this for my local police scanner.

posguy• 4 years ago

The codecs and narrow frequencies used on most public safety trunked radio networks is truly terrible. IMBE and AMBE+2 should have never made it past Y2K, yet Motorola and Project 25 have ensured these remain in widespread use on these networks: https://en.wikipedia.org/wiki/Project_25

If Whisper achieves 85% or higher accuracy on this audio, it would be a miracle. Garbage in, garbage out tbh. Project 25 needs to move to a modern codec, ideally not one seeing little development done by one small company.

lights0123• 4 years ago

I've done this with Google's video transcription model, which works surprisingly well.

CiceroCiceronis2• 4 years ago

I wonder how much training on degraded/radio encoded samples would be needed to improve performance in this area. You’re probably not the only person who wants to monitor police radios in their city.

NaturalPhallacy• 4 years ago

Have you tried https://webcaptioner.com/ ?

iKlsR6• 4 years ago

I came up with this specifically because of whisper. This looks nice but doesn't suit my use case as I'm "listening" to over a dozen stations at once and I also stitch the chunks together end of day in a fat wav file.

dgtlntv• 4 years ago

whisper.cpp (https://github.com/ggerganov/whisper.cpp) supports streaming!

synesthesiam• 4 years ago

Have you tried whisper-cpp?

https://github.com/ggerganov/whisper.cpp/tree/master/example...

nico• 4 years ago

Is this English only, or does it support other languages? Thank you.

iKlsR6• 4 years ago

The streaming I was speaking to is from a url or network resource. I keep a directory of m3u files which is just a url inside that you can open in vlc etc. `So ffmpeg -i [url] -c copy [file-name].mp3` does the trick for now. `mpv` can do this while saving to a file but it's a nice point to start from as I'm already getting some ideas how I could use this or roll my own, thanks!

bochoh• 4 years ago

This but for automation of calling in for prizes or entering keywords on station websites.

mindvirus2• 4 years ago

Out of curiosity, when you chunk it, what do you do about words (and maybe even sentences) that might get caught in the middle of the split?

cwkoss• 4 years ago

Oh geez, this is one of those technical problems that have so many possible good answers that it'd make a great interview question.

iKlsR6• 4 years ago

Great question, I've not encountered it too much as I'm looking for specific stuff in the transcript based on keywords so some loss is acceptable and I don't always need the data immediately so it has not been an issue so far but what I do to alleviate this somewhat is have "stitched passes" at set intervals.

News, betting games and some shows HAVE to happen exactly at a certain time and are very rarely late so you can use these known times as checkpoints with some bias. So say I run this command `ffmpeg -i http://someexamplesite.fm/8b0hqm93yceuv -c copy -f segment -segment_time 60 -reset_timestamps 1 zip-%03d.mp3` that automatically chunks the stream into 1 minute files and I know news gets read at 1pm, I can merge everything from 10am to 1pm into say "Segment B" and then process that, you get the idea.

I also have a step in the pipeline after on the transcript after that tries to summarize what it can so any small gaps would likely be inferred as I have not noticed anything too wonky and the summaries and text I get so far have been pretty clean and good enough for my needs. Once I bench this some more however I'm sure I will have this and other interesting problems to solve, the one I'm fighting now a bit is ads that have music and fast talking.

lunixbochs3• 4 years ago

Nice catch. I'll run my test suite [1] on this and report back.

[1] https://twitter.com/lunixbochs/status/1574848899897884672

amelius2• 4 years ago

Would be cool if you could run these speech models in tandem, and compute a new error-rate for the consensus of them.

freediver• 4 years ago

What would you use in production?

nshm6• 4 years ago

Looks like they plugged GPT-4 AI into speech recognition research and now they are going to release huge updates every month.

dweekly• 4 years ago

What is the basis for this claim? How does their GPT-4 work intersect with the work on their ASR model?

nshm6• 4 years ago

Easily, you can ask questions about speech recognition system design. For example you can ask ChatGPT about upsampling vs downsampling, it gives correct answer. It also can correctly recommend you architecture for streaming speech recognition.

wcoenen• 4 years ago

It's a joke... They're saying GPT-4 is doing the research now.

galleywest200• 4 years ago

I have been using Whisper to transcribe my audio notes. I just save my voice memo from my phone to my NAS and my little script does the rest on a loop.

chimineycricket• 4 years ago

How do you handle connectivity from outside your home network (when you're at the grocery store for example)? Do you have a VPN running?

tehf0x• 4 years ago

Not OP but I use syncthing for such things.

andrewmackrodt• 4 years ago

Not OP but regularly access my home network remotely. I have an Asus RT-AX86U which has a lot of features in the stock firmware, including OpenVPN configuration. If I need "LAN" access to my home network, I will open the OpenVPN client and then it will just work, this is useful to access things like local samba shares or SSH into my Linux machines (I was dissatisfied with the log spam of having an open SSH port even with fail2ban and router level protections).

If you have an always running server and only need access via a browser, forwarding port 80/443 to a reverse proxy works well. Dynamic DNS can be used if you don't have a static IP address. Alternatively, if you are more security conscious and don't mind the reliance on CloudFlare, you can use CloudFlare tunnel which won't expose ports from you local network at all. It can also be combined with a third party authentication mechanism.

dgacmu• 4 years ago

Also not OP but I use tailscale for this. It's fantastically easy.

wanderingmind3• 4 years ago

Tailscale + syncthing, works like a charm

AB1908• 4 years ago

Hey there, can you share your script? I'm a little lazy haha

daemoens• 4 years ago

Is there a reason why Spanish and Italian have a lower WER than English? OpenAI is based in America right? They probably focused on English more than anything.

nshm6• 4 years ago

Spanish and Italian are very easy to recognize due to very simple phonetic structure.

0cf8612b2e1e3• 4 years ago

English is the worst. “We don't just borrow words; on occasion, English has pursued other languages down alleyways to beat them unconscious and rifle their pockets for new vocabulary.”

I could believe that other languages with more regular grammar might be easier to parse. Or maybe Romance languages lean on more easily distinguished phonemes.

pessimizer• 4 years ago

Wonderful quote.

paranoidrobot• 4 years ago

I like that quote.

English also tends to take delight in stealing words from other languages, and then using them in the wrong way. Seemingly in an effort to drive speakers of the other language up the wall.

nmfisher• 4 years ago

WER is really dataset-dependent, you can’t really compare across different test sets unless they’re almost identical (acoustic environment, microphone quality, speaker profiles, linguistic domain, etc).

WER for the exact same model can vary wildly between datasets, it’s really only useful for comparing model performance on a single test set.

tpmx• 4 years ago

So in general and from the uninformed outside it seems like OpenAI (120 employees?) is outperforming Alphabet (187k employees). Is there any truth to that notion?

erdaniels• 4 years ago

Outperforming in what sense? AI model capabilities?

bogwog• 4 years ago

Google makes more money than OpenAI, so I wouldn't say that.

Also, OpenAI is basically a subsidiary of Microsoft at this point.

alchemist1e9• 4 years ago

It could well be that 187K employees is a disadvantage, not just economically, but because every project has too many cooks in the kitchen so to say, and it has too much politics.

nomel3• 4 years ago

A tiny fraction of that 187k employees is working on AI. Total employee count is a silly metric.

joewhatkins2• 4 years ago

Google has models that are at parity with or outperform OpenAIs models at most tasks, they just don’t bring them to market quickly or build their brand around them as strongly as OpenAI does.

See Imagen for example - https://imagen.research.google/

If Google is to be believed, this outperforms Dall-E - and I’ve heard from people that use it that in general, it does perform better than Dall-E.

echelon• 4 years ago

Doesn't do them any good to do nothing with it.

I don't get Google's brand or leadership anymore. They're staring disruption to their core revenue stream in the face and acting like the "everything is fine" dog.

Meanwhile Microsoft is playing dimensional chess across multiple industries and key developing areas of research.

espadrine• 4 years ago

If Google is to be believed, this outperforms Dall-E - and I’ve heard from people that use it that in general, it does perform better than Dall-E.

Google actually surpassed its own model[0], first with Parti[1], which unlike DALL-E could even correctly insert text in the image, then with Imagen Video[2], which does as the name implies.

[0] https://paperswithcode.com/sota/text-to-image-generation-on-...

[1] https://parti.research.google/

[2] https://imagen.research.google/video/

Matthias247• 4 years ago

keep in mind that Google also works on other products than AI (e.g. Android), and that the best model is not necessarily the best for the business. E.g. it might be too costly to operate certain models as Googles scale, and so they would need to pick a compromise between quality of results, runtime cost and scalability.

janalsncm2• 4 years ago

It’s apples and oranges imo. Google owns Wayze which is hitting the streets in SF this year with autonomous taxis. That’s huge if it goes well. There are of course other non-public models Google has developed like Imagen (diffusion model) and LaMDA (Google engineer convinced it was sentient). So I wouldn’t discount Google at all.

In terms of granting direct access to machine learning models, OpenAI has Google beat. But that’s not Google’s business model. And it remains to be seen if it’s a viable business model at all.

nshm6• 4 years ago

I've just tested it quickly, large-v2 is actually slightly less accurate than v1. So pick the model carefully.

verdenti• 4 years ago

I don’t understand. Why is v2 worse than v1?

nshm6• 4 years ago

Hard to guess. We test on variety of datasets. New version has more hallucinations in the end of audio.

nshm6• 4 years ago

Other people reporting accuracy issues

https://github.com/openai/whisper/discussions/657

darkpicnic2• 4 years ago

Does anyone know if this new model handles silence better? I was trying to use whisper for transcribing bursts of talking amid large spans of silence, but the frequency of hallucinations was too high.

rozab• 4 years ago

You don't need ML to trim out silence

sdenton4• 4 years ago

Silence is often problem dependent... You may want ML to differentiate between noisy audio with speech and noisy audio without speech.

darkpicnic2• 4 years ago

"Silence" is a problematic term. For me, that word encompasses: squeaky chairs, typing on a loud keyboard, moving objects around on my table, etc. In a perfect world, Whisper —like a human— can easily distinguish a human voice from the din of my office, and only try and transcribe my voice.

Does anyone have solutions for clearing out "silence" from an audio file that works off something a bit more accurate than just "<= decibel x"?

Edited for grammar.

nomel3• 4 years ago

I suspect a simple solution is to remove the silence, as a pre processing step in the pipeline.

gibolt• 4 years ago

Still important for future use to not have invalid results. This is a workaround for now

lunixbochs3• 4 years ago

In large scale tests, I observed hallucinations from Whisper in speech regions of audio.

Yajirobe• 4 years ago

Did they just commit straight to the main branch?

counttheforks• 4 years ago

They probably have an internal repo which gets the commits first, then synced to github.

ducktective• 4 years ago

Why not? Should have commited to, like, dev?

152334H• 4 years ago

"Whisper V2" is a bit of an exaggeration. It's

* only an update to the `large` model (and therefore irrelevant for hobbyist, low CPU use cases)

* architecturally identical to the original whisper model

* just trained on more epochs

yjftsjthsd-h• 4 years ago

Yeah, the commit is literally "add large-v2 model", nothing about the software proper.

IanCal• 4 years ago

The large model can run on consumer GPUs and I'd argue is more relevant to the hobbyist use cases because I'm not working out the efficiency of running thousands of users - just my own videos.

nl• 4 years ago

* only an update to the `large` model (and therefore irrelevant for hobbyist, low CPU use cases)

This is maybe the most important thing there is for a lot of cases since additional training is the thing that is just about impossible to do for most people.

m3at• 4 years ago

Agreed. It's maybe like Linux versioning scheme, where 6.0 is just the one that comes after 5.19:

The major version number is incremented when the number after the dot starts looking "too big." There is literally no other reason. [1]

[1] https://www.kernel.org/category/releases.html

prox• 4 years ago

Is there an easy to install speech engine (training and synthesizing) for windows? So the reverse of recognition?

tiffanyh• 4 years ago

Off topic: what’s OpenAI revenue model?

(I’m under the impression they don’t charge)

jokethrowaway• 4 years ago

They charge

jtxt• 4 years ago

https://openai.com/api/pricing/ (ChatGPT is a free preview right now.)

_joel• 4 years ago

Interesting to see there's work going into M{1,2} support rather than just CPU. Looks like there's an issue with the buffer lengths with pytorch M1 support atm but looks fixable in future pytorch releases.

userhacker5• 4 years ago

I recently swapped out the AI model for voice transcription on revoldiv.com and replaced it with Whisper. The results have been truly impressive - even the smaller models outperform and generalize better than any other options on the market. If you want to give it a try, our model is capable of faster transcription by utilizing multiple GPUs and some other enhancements, and it is all free

thundergolfer3• 4 years ago

Your app is very similar to our demo app! https://modal-labs-whisper-pod-transcriber-fastapi-app.modal...

How come you don't support audio files longer than 1hr? Is it because of $$ cost?

The above demo app gets faster transcription by chunking audio and parellelizing over dozens of CPUs, so you can transcribe a about 1hr of audio for $0.10.

userhacker5• 4 years ago

https://modal-labs-whisper-pod-transcriber-fastapi-app.modal...

Interesting, which model are you using? We use the medium model which is the sweet spot between time/performance ratio. We also chunk, We try to detect words and silences to do better chunking at word boundaries but if you do more chunking and you don't get the word boundaries right it seems like whisper loses some context and the accuracy suffers. We will soon support longer hours. We just want to make sure the wait time for transcription doesn't suffer for most users. But great demo, reach out to me if you want to collaborate

71a54xd• 4 years ago

I'd love to try it out, who should I ping to run this locally on my GPU server?

userhacker5• 4 years ago

It's not a model you can run on your own server but a free service on revoldiv.com. You can expect 40 to 50 second wait time to transcribe an hour long video/audio. We combine whisper with our model to get word level timestamps, paragraph separation and sound detections like laughter, music etc... We recently added very basic podcast search and transcription.

boriscal2• 4 years ago

How does this benchmark against AWS [0] offering?

[0] https://aws.amazon.com/transcribe/

gamegoblin• 4 years ago

I have done some tests for a hobby project (transcribing local fire station radio dispatch), and Whisper is incomparably better than AWS transcribe.

Radio dispatch is very hard to understand, low quality audio. AWS transcribe was essentially useless, incomprehensible transcriptions. Whisper was 95+% accurate, with maybe 1 incorrect word per few sentences, and it was often easy to tell what the correct word would be.

boriscal2• 4 years ago

Interesting, that's good to know, my personal experience was that transcribe handled accents very poorly, but if whisper can handle radio static maybe it can also decipher thick scottish

0cf8612b2e1e3• 4 years ago

If I just want to play with this as a toy without any performance considerations, is it possible to run this on a CPU only? Or do I need to faf about with getting CUDA and that nonsense running?

zachlatta2• 4 years ago

Yes, it runs fine on your computer.

jerpint• 4 years ago

It’ll run on CPU in decent time, it’ll also depend on the kind of cpu you have and the model you pick. In my m1, the base model can do 5-10s in about as long.

I uploaded it to a hugging face space if you just want to fiddle with it without installing

https://huggingface.co/spaces/jerpint/babelfish

thundergolfer3• 4 years ago

Yes, it runs well enough on CPU. https://modal-labs-whisper-pod-transcriber-fastapi-app.modal... runs on CPU, and chunks up podcast audio to parallelize over dozens of CPUs.

GistNoesis• 4 years ago

You probably want to check this port : https://github.com/ggerganov/whisper.cpp

0cf8612b2e1e3• 4 years ago

That indeed seems perfect.

lordswork2• 4 years ago

Has anyone combined Whisper with a model that can assign owners to voices yet?

For example, instead of:

    Hello there!
    Hi 
    How are you?
    Good, and you?

You get something like:

    Voice A: Hello there!
    Voice B: Hi 
    Voice A: How are you?
    Voice B: Good, and you?

itake• 4 years ago

I'd be down to code this up if someone if anyone's interested. contact is in profile.

sbmsr• 4 years ago

Not that I know of, but I use Descript for content creation and it offers something like what you're describing. they have automatic speaker labels built in to their transcription service.

would be lovely to see this feature open sourced.

CiceroCiceronis2• 4 years ago

There is a discussion on the Whisper github page called something like “diarization” which details a few attempts to attain this functionality with additional tools.

zachlatta2• 4 years ago

I modified a Jupyter notebook that’s been going around to do this and have nice Markdown output.

This is the best AI diarization and transcription I’ve been able to get so far: https://github.com/zachlatta/openai-whisper-speaker-identifi...

gok• 4 years ago

This is called "diarization"

nxmnxm99• 4 years ago

Some attempts have been made but it still sucks.

e12e• 4 years ago

Not whisper, but:

The Google Recorder app (...) transcribes meetings and interviews to text, instantly giving you a searchable transcription that is synced to the recorded audio (...) With this new update, the recorder can now identify and label each speaker automatically—an impressive feat. Google Recorder is exclusive to the Pixel 6 and newer Pixel devices.

https://arstechnica.com/gadgets/2022/12/pixel-7-update-adds-...

lordswork2• 4 years ago

That's awesome. I wonder if there's a way to upload existing recordings to Google Recorder to get the transcriptions.

bigfudge• 4 years ago

I’ve had limited success combining the results of pyannotate with whisper transcripts (based on time stamps).

It’s ok, but the quality of speaker identification is nowhere near as good as the transcription itself.

I’d love to see models which try and use stereo information in recordings to solve the problem. Or, given a fixed camera and static speakers, I thought it should even be possible to use video to add information about who is speaking. There doesn’t seem to be anything like that right now tho.

sva_• 4 years ago

This is called speaker diarization, basically one of the 3 components of speaker recognition (verification, identification, diarization).

You can do this pretty conveniently using pyannote-audio[0].

Coincidentally I did a small presentation on this at a university seminar yesterday :). I could post a Jupyter notebook if you're interested.

PS: Bai & Zhang (2020) is a great review on the literature [1]

[0] https://github.com/pyannote/pyannote-audio

[1] https://arxiv.org/abs/2012.00931

wanderingmind3• 4 years ago

Yes please posting a jupyter notebook will be of great help

swyx• 4 years ago

yes please!

aadnk• 4 years ago

It seems like Large-V2 is a huge improvement when transcribing Japanese. I tested it on "Macross Frontier - the Movie", and it no longer breaks after 8 minutes as before:

Large-V1 (transcribed at 2022-10-02):

* SRT: https://pastes.io/yrggofqhof

* Transcript: https://pastes.io/rtp9buhsm0

Large-V2 (latest version at 2022-12-07):

* SRT - https://pastes.io/uiqblpw1qk

* Transcript - https://pastes.io/rheqgnftzl

There's still some timing issues after a period of silence, but using a VAD as a workaround (like I do in my WebUI) may no longer be strictly necessary:

* https://github.com/openai/whisper/discussions/397

getcrunk3• 4 years ago

I’ve only ran this on my cpu. Using the tiny en model.

Anyone ran tiny en on t4 gpu (aws g4dn iirc)? What’s the speed up

userhacker5• 4 years ago

For revoldiv.com we have profiled, many gpus, the best one is 4090. We do a lot of intelligent chunking and detect word boundaries and run the model in parallel in multiple gpus and we get about 40 to 50 seconds for an hour long audio but without expect 7 minutes for an hour long audio on tesla t4

  on tesla-t4-30gb-memory-8vcpu google cloud
   on tiny and tiny.en
    for 10 minute = 30 seconds
   on medium
    for 10 minute = 1m 30s
    for 60 minute = 7m
   on large
    for 60 miutes = 13m
  on NVIDIA GeForce RTX 4090
   on tiny
    for 10-minute = 5.5 seconds
    for 60-minute = 35 seconds
   on base
    for 10-minute = 7 seconds
    for 60-minute = 50 seconds
   on small
    for 10-minute = 14 seconds
    for 60-minute = 1 min 35 sec
   on medium
    for 10-minute = 26 seconds
    for 60-minute = 3 mins
   on large
    for 10-minute = 40 seconds
    for 60-minute = 3 min 54 sec

getcrunk3• 4 years ago

thats crazy. 35 seconds for 60 min on base with a 4090. wow! thanks for the info! also btw i mentioned on another thread i was getting an error. but are you planning on offering this as a paid api?

PaulMest• 4 years ago

There's a 2 week Whisper Fine Tuning event put on by Hugging Faces going on right now [1]. The new "large-v2" model was announced as part of the kickoff [2]. It's supposed to offer 5% - 10% improvements over the previous large model.

[1] https://discuss.huggingface.co/t/open-to-the-community-whisp... [2] https://youtu.be/fZMiD8sDzzg?t=1226

jamsch• 4 years ago

Has there been any work on a model that can take in speech and outputs the phonemes (e.g. in IPA or an English format) similar to what Google's speech pronunciation tool, where it shows a "Sounds like you said ...."?

wanderingmind3• 4 years ago

Has there been any efforts in integrating whisper model into a more advanced engine like Kaldi to leverage the most out of this wonderful model

kamilafsar• 4 years ago

I wonder what the correlation between consistency of written/spoken language is for the breakdown per language here: https://github.com/openai/whisper/blob/main/language-breakdo...

For instance, I know Turkish is very consistent: it was refactored in 1928 with the birth of Turkey. Turkish is quite high in the rankings. I don't think because there's loads of data available, but because of its consistency. Contrary, English has loads of data, which should compensate for it inconsistency.

casey2• 4 years ago

The new large model is garbage at translation (ja) compared to most of the other models.

OpenAI quietly launched Whisper V2 in a GitHub commit

Related Stories