DeepSpeech 0.6

https://hacks.mozilla.org/2019/12/deepspeech-0-6-mozillas-speech-to-text-engine/

by pionerkotik • 7 years ago

278 66 7 years ago

hacks.mozilla.org

tobylane• 7 years ago

I'm starting to look into the MycroftAI. It sounds like with this release I could use Mycroft+DeepSpeech on a new RaspberryPi for a completely offline smart speaker, do I understand that right?

zitterbewegung• 7 years ago

Yes, you are correct tensorflow lite can be done on the raspberry pi. https://www.tensorflow.org/lite/guide/build_rpi

reubenmorais11• 7 years ago

Correct. The latest release with the TensorFlow Lite model runs in real time on a Raspberry Pi 4. I'm not sure if the Mycroft integration is updated to the latest version though, as it was just released.

est312• 7 years ago

Congrats to the team on the new release! Following their progress since a while. It's an important project, and I'm very happy about the size reductions while they still delivered a WER improvement over the last release. Amazing!

reubenmorais11• 7 years ago

Thank you for your work on the Rust bindings :)

detaro2• 7 years ago

It achieves a 7.5% word error rate on the LibriSpeech test clean benchmark

Anyone have a comparison for how good/bad that is compared to other solutions, and what it means for practical usage, if that can be guessed at from a single number?

Traktor• 7 years ago

You can find a leaderboard here: https://paperswithcode.com/sota/speech-recognition-on-libris...

It seems like sota is 2.20% word error rate

lunixbochs3• 7 years ago

I like reading through https://github.com/syhw/wer_are_we sometimes

wstuartcl• 7 years ago

Note that the leaders there are not "near real time" streaming type implementations like this is. They have the benefit of having full context (they act on the full input not a stream of current state). 7.5% is really good for this type of low latency stream based system.

est312• 7 years ago

LibriSpeech test clean is without background noise, with a good microphone, and most likely biased to native speakers (it is 50/50 male/female at least). In summary, it's not what an ASR engine will encounter. However, the dataset is used as a common method to benchmark ASR engines and thus that number is valuable. The SpecAugment paper by Google for example claims a 2.5% word error rate [1] on that corpus (using only LibriSpeech for training data).

[1] https://arxiv.org/pdf/1904.08779.pdf

dabinat2• 7 years ago

It’s worth mentioning that while DeepSpeech 0.6 supports SpecAugment, the released model was not trained with SpecAugment switched on.

reubenmorais11• 7 years ago

Author here. I would add to what the sibling comments have mentioned by saying that SotA results should be taken with a grain of salt. Our engine is capable of streaming (processing the audio as it's being recorded), which is not doable with architectures that have bidirectional decoders or attention mechanisms that require the whole encoder input ahead of time.

For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds. This is why, if you run a standard test set like LibriSpeech on, say, a commercial offering from Google, it will perform considerably worse than state of the art according to Google papers.

This repository [0] has a benchmark of some commercial offerings. Our model beats all of those on Librispeech clean and other (except for Speechmatics on Librispeech clean), as well as on Common Voice. But note that the Common Voice corpus used in that benchmark is very old.

In sum, I would compare this against solutions that go for the same space: fast, client-side ASR, rather than state of the art.

[0] https://github.com/Franck-Dernoncourt/ASR_benchmark#benchmar...

albertzeyer2• 7 years ago

Hi! I'm from the RWTH team (https://paperswithcode.com/sota/speech-recognition-on-libris...). Our best system (2.3% WER) (trained only on the Librispeech data, i.e. much less data than your model) currently is a hybrid HMM/NN model, and you are right, the acoustic model uses a BLSTM. However, we have shown in other work (https://www-i6.informatik.rwth-aachen.de/publications/downlo...) that you can expect similar performance in an online system, or not much worse (there is also many related work, which often is maybe 5-10% relative worse than the offline system).

I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right? As far as I remember, you use CTC, right? I always wondered why you have chosen CTC, and not some better model, like RNN-T, RNA, or some of the streaming attention variants. They should give you all the same properties (online capable, low latency, simple, end-to-end), but much better WER performance. Or is this simply because there currently is no simple ready-to-use implementation for those? Note that we published some TF code recently for some streaming attention variants, and plan to publish some RNN-T/RNA code soon.

option• 7 years ago

NVIDIA has QuartzNet which contains only 19M weights and achieves 3.9% on test clean without language model and less then 3 with LM. Code (Pytorch): https://github.com/NVIDIA/NeMo Paper: https://arxiv.org/pdf/1910.10261.pdf

IshKebab4• 7 years ago

Our engine is capable of streaming

A bit misleading given that decoding can't be streamed yet.

IshKebab4• 7 years ago

For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds.

You should follow Google's approach - give fast live results that don't depend on data from the future, but also go back and correct old words when you do have that data. It's kind of how humans work really.

lunixbochs3• 7 years ago

For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds.

My wav2letter research does not use streaming, despite supporting streaming, because I get lower average latency (roughly 0.02x RTF including decoding, or 80ms for a 4000ms input audio like you describe in the article) by running the entire encoder CNN at once instead of running it in chunks.

This is also nice because the CPU usage is practically nothing (0.1%) until you stop talking.

I have a web demo here [1], with a relatively terrible language model (mostly just been working on acoustic modeling so far, as it’s just me). Most of the latency is waiting for the JavaScript VAD (which is much slower than webrtcvad and I couldn’t figure out how to tune it) and waiting for the network. If you look at the network inspector, the server should report its encode and decode times.

[1] https://web2letter-west-1.talonvoice.com/

gok2• 7 years ago

7.5% WER is much much worse than the state of the art, but it's also important to recognize that LibriSpeech is kind of a ridiculous data set. It's 1000 hours of volunteers reading public domain novels out loud. It's very unlikely to match your use case. It's widely used in academia purely because it's larger than other freely available data sets.

bayesian_horse3• 7 years ago

I wonder if it is any good at out-of-vocabulary words? Is it hard to teach it new things like medical terms and such?

reubenmorais11• 7 years ago

In the post I mention a Brazilian company who's using DeepSpeech for vocabulary constrained, medical applications. They've trained their own Brazilian Portuguese models.

The English acoustic model that we released does not have a fixed vocabulary. What determines the vocabulary is the language model, which can be created from text. So if you have in-domain text, and would like to try it out, I would say the first step is to create a language model using your text and then experiment with it.

We have some documentation on how to create the LM here: https://github.com/mozilla/DeepSpeech/tree/v0.6.0/data/lm

It's not super detailed, but I'd be happy to answer questions on our Discourse: https://discourse.mozilla.org/c/deep-speech

faissaloo• 7 years ago

I did write a small program to make custom vocabularies and datasets for DeepSpeech a while back when I was messing with it: https://github.com/faissaloo/deepspeech_dataset_builder It could use some TLC but it does the job.

candiodari• 7 years ago

It's using letters, not words, so in theory anything that can be spelled in English can be output:

https://2r4s9p1yi1fa2jd7j43zph8r-wpengine.netdna-ssl.com/fil...

Or, if you prefer the code: https://github.com/mozilla/DeepSpeech/blob/0427c1572ac8f7253...

(well, I suppose technically it's in https://github.com/mozilla/DeepSpeech/blob/master/data/alpha... through https://github.com/mozilla/DeepSpeech/blob/master/util/text...., but I mean come on)

In practice ... meh. It gets some things predictably wrong. They're readable, just wrong. So the good news is that WER is words it has 100% correct. The remaining errors aren't usually totally unreadable, just irritating, going from "totext" instead of "to text", but of course names go pretty wrong and don't get better. It butchers my name, of course.

If you test it yourself, please do realize it really doesn't deal very well with accents.

kbumsik• 7 years ago

Looks great. Is it related to any of Mozilla's product?

reubenmorais11• 7 years ago

We'll have more to share about that soon :)

bayesian_horse3• 7 years ago

I just found https://voice.mozilla.org/

Besides being a great resource for speech analysis, this could be a real game changer for acquiring listening comprehension in a foreign language.

I feel that even after a few years of learning a new language I still have trouble with listening. Part of that is that it's often all or nothing, even one or two unknown words in a sentence means I can't understand the sentence. But worse is that most language teaching materials use a very small set of native speakers, which deprives the learner's brain of being able to generalize.

pete762• 7 years ago

Some of the English samples by non-natives were completely incomprehensible. I hope the system learns to be accent free.

bayesian_horse3• 7 years ago

They have a validation process, and the data is mainly meant to train speech-to-text.

For that, it's important to include accents and even non-native speakers, especially in English.

I think that those wavenet models for TTS were able to be conditioned by gender and maybe accent, if they had the data.

Barrin92• 7 years ago

given that there are more non-native than native english speakers on the planet I think it makes a fair bit of sense to include non-native data in the training set

franciscop• 7 years ago

Tensorflow lite, interesting. Has anyone tried this with Google Coral USB accelerator? I got one laying around but very little experience with ML. I could get the USB Accelerator working with the pretrained posenet after much messing around, but my chances with DeepSpeech are small to none. This seems like the perfect fit though.

dabinat2• 7 years ago

To the best of my knowledge, DeepSpeech currently isn’t compatible with any inference or training accelerators (other than NVIDIA GPUs).

But the whole point of the TFLite model is that it has low resource requirements so an accelerator for inference shouldn’t be necessary.

reubenmorais11• 7 years ago

TFLite on Android with the NNAPI delegate is theoretically supposed to leverage hardware accelerators. We haven't tested it extensively though.

therealmarv• 7 years ago

So who will be developing an open source assistant like Goggle Home / Alexa? :)

Erlich_Bachman• 7 years ago

If you do a 1 minute google search you will find that there are multple open source voice assistant offerings already, for example mycroft.

detaro2• 7 years ago

Most of which are not fully open, e.g. do not have open speech recognition.

bmn__• 7 years ago

I do not understand how to use Deepspeech even in the most simple use case.

1. I want to teach it ten words. How do I do this?

2. I want to speak into my microphone (available as a Pulseaudio device) and recognise the words and output the words as a text stream on stdout. How do I do this?

This is the documentation:

https://deepspeech.readthedocs.io/en/v0.6.0/Python-Examples.... https://deepspeech.readthedocs.io/en/v0.6.0/Python-API.html

It does not answer the questions I have.

https://deepspeech.readthedocs.io/en/v0.6.0/DeepSpeech.html

The introduction page is full of incomprehensible jargon.

reubenmorais11• 7 years ago

1. I want to teach it ten words. How do I do this?

I'm assuming you mean "my target vocabulary is only ten words". In that case, you have a couple of options:

1. Just use it as is and see how it does.

2. Create a language model containing just those 10 words so that the engine is constrained to that vocabulary, which should improve accuracy.

3. Train a (smaller) model from scratch, which requires (audio, transcript) pairs. This is basically research work, there's no recipe you can just follow. The advantage is that if you can make it work, it can be a much smaller model than a model trained for general English dictation.

2. I want to speak into my microphone (available as a Pulseaudio device) and recognise the words and output the words as a text stream on stdout. How do I do this?

We have a few examples with microphone input, but they're all based on voice activity detection: https://github.com/mozilla/DeepSpeech/tree/master/examples

I should write an example that is just continuous streaming and output without any voice activity detection.

I'd be happy to discuss further on Discourse: https://discourse.mozilla.org/c/deep-speech

na85• 7 years ago

Just use it as is and see how it does.

How? The docs seem to be suffering from the same problem as the git docs: they assume the reader is already a domain expert, which would seem to run counter to the stated goal of being simple (and ubiquitous).

I tried to figure out how to "just use" deepspeech and failed.

IshKebab4• 7 years ago

I think you're being a little harsh.

1. Train it using the training dataset but change all word labels that aren't in your 10 word list to "OTHER_WORD" or whatever. There's probably not much point doing this. Alternatively I guess you can restrict the beam search to only look for your 10 words. Yes this is really complicated - it's a speech recognition engine - they are really complicated!

2. The API docs look pretty clear to me:

createStream()[source]
Create a new streaming inference state. The streaming state returned by this function can then be passed to feedAudioContent() and finishStream().
feedAudioContent(args, kwargs)[source]
Feed audio samples to an ongoing streaming inference.
intermediateDecode(args, kwargs)[source]
Compute the intermediate decoding of an ongoing streaming inference. This is an expensive process as the decoder implementation isn’t currently capable of streaming, so it always starts from the beginning of the audio.

Huh well that's a flaw the neglected to mention! I guess you can't really do what you want yet.

reubenmorais11• 7 years ago

Huh well that's a flaw the neglected to mention! I guess you can't really do what you want yet.

The documentation is outdated. That is no longer true, intermediateDecode is cheap. Thanks for noticing, I'll fix it.

sillysaurusx• 7 years ago

So, funny story: I wanted #2 and did it myself. I was similarly frustrated with the lack of documentation.

It's been a little while since I got it running, but I basically got a siri clone working. If you want to test it out, I can try to answer questions / whatever problems pop up.

The code is here: https://github.com/shawwn/DeepSpeech/commit/01f5cf8d39c356ae...

As far as I know, you can simply run speech_to_text.sh. It will connect to your microphone and start dumping out transcribed audio to stdout.

It wasn't super easy, but once you spend a little time with the code you can sort of figure out ways to get it to do what you want.

EDIT: By the way, people will try to convince you that you're nuts and that the documentation is crystal clear and so on. Know this: It's not just you. I had the exact same experience. It's a recurring theme in AI programming.

The only thing to do is to either find someone else who feels similarly, or roll up your sleeves and dive into the code.

logicallee• 7 years ago

By the way, people will try to convince you that you're nuts and that the documentation is crystal clear and so on

No it really is simple: the original implementation had a base of prefabulated amulition grammar, this was surmounted by a malleable logarithmic function in such a way that the two main spurving vocabularies were in a direct line with the panametric grammar fields. The latter simply consists of marzlevanes fitted to the ambifacient morpheme waneshaft to eliminate side utterances. This main winding is of the normal lotus-o-deltoid type, just placed in panendermic semi-boloid stators. Basically every second conductor is connected by a nonreversible tremmie pipe to the differential grammeters.

All these words are in: https://en.wikipedia.org/wiki/Turboencabulator

travisporter• 7 years ago

Any recommendations on a good raspberry pi hat or microphone to test this?

jcims3• 7 years ago

This reminds me of something I would love to see happen but I don't have the skills to put it all together. I really think there's some potential merit to a reading coach app(lication) that listens to someone read and looks for weaknesses/disorders/etc compared to a trained model. It could provide those diagnostics to an educator, guide the content to focus on those, coach the reader directly, etc.

It all seems very doable based on what I see in the technology today, I just don't have the skills to do it.

Enginerrrd• 7 years ago

I'll be honest... that seems like a pretty tough sell to me. You could be looking at anything from a speech disorder to a learning disability to a perfectly healthy normal child with idiosyncrasies and those things are usually left to professionals with graduate degrees. The liability seems high, you've got both the potential stigma associated with incorrectly flagging a child, or worse, missing an otherwise obvious issue that means treatment is delayed during critical years.

jcims3• 7 years ago

I totally understand where you're coming from, but the reality is that there is a lot of imbalance and delay in this type of assessment and educational support already. My wife has been an English language arts teacher and instructional coach for almost 20 years. While they definitely have standard diagnostics and regular assessments to try to keep kids on track, it's essentially best effort. And when you move out of 4th/5th grade in many districts you start getting to 90:1 teacher-student ratios with 1-2 interns available to do 1:1 reading support across maybe 500 kids. And this is in a middle class school district.

There's definitely some ways it can go bad, but even basic fluency monitoring without any active remediation attempts would be a good addition.

nmstoker• 7 years ago

I've seen this referred to online as "Intelligibility Remediation" (a term I always forget when I'm trying to find the details months later!)

Certainly seems like an area with huge potential application.

mncharity4• 7 years ago

Perhaps there's an opportunity to create a demo webpage using tensorflow.js? Especially with the new TSLite support. To raise visibility and awareness. The posenet webcam demo[1] for example, seems to have ~2k links.

[1] https://storage.googleapis.com/tfjs-models/demos/posenet/cam...

testbed• 7 years ago

I can't seem to find pre-trained models for other languages (French). How long did you training take for those in English? Do you think it makes sense to start from the English one? Thank you!

maxbrunsfeld• 7 years ago

DeepSpeech v0.6 with TensorFlow Lite runs faster than real time on a single core of a Raspberry Pi 4

This is great news.

I'm not very familiar with the deep learning framework ecosystem. Does anyone know what the simplest way would be to incorporate a DeepSpeech model into a WebAssembly project?

For instance, is it straightforward to compile Tensorflow Lite with emscripten? If not, can this TensorFlow model be run with tensorflow.js? Or can it be converted to some other format that is easier to use with WASM?

z3t4• 7 years ago

I was very exited about this, then I tried it with the pre-trained model and I recorded "Hello this is a test message" 3 times and it was inferred as "a sassanian", "he is a paris" "he states that" ...

DeepSpeech 0.6

Related Stories