Meta AI announces Massive Multilingual Speech code, models for 1000+ languages

https://github.com/facebookresearch/fairseq/tree/main/examples/mms

by crakenzak • 3 years ago

705 222 3 years ago

github.com

crakenzak3• 3 years ago

Code: https://github.com/facebookresearch/fairseq/tree/main/exampl...

Blog Post: https://ai.facebook.com/blog/multilingual-model-speech-recog...

Paper: https://research.facebook.com/publications/scaling-speech-te...

Languages coverage: https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mm...

generalizations3• 3 years ago

Based on the availability of STT, TTS, and translation models available to download in that github repo, a real-life babelfish is 'only' some glue code away. Wild times we're living in.

bdidbdidhdidh• 3 years ago

hope that glue is better then the duct tapes google use on translate.g.c

it is still hopeless, and much worse than the dictionary based, at gender/nums/dicleasions in general.

i sometimes use it just to not thing about the grammar in some languages, and most times I'm doing a surprised double take of something that would be completely inappropriate or offensive instead of my simple phrases.

mach1ne• 3 years ago

Nah, full-on babelfish is simply not possible. The meaning of the beginning of a sentence can be modified retroactively by the end of the sentence. This means that the Babelfish must either be greatly delayed or awkwardly correct itself every once in a while.

littlestymaar• 3 years ago

AFAIK STT is still very bad without speaker-specific fine-tuning, so it's not going to be a literal babelfish (translating in the ear of the receiver), but it could make you speak many languages.

simonw4• 3 years ago

I loaded the language coverage into Datasette Lite and added some facets here:

https://lite.datasette.io/?json=https://gist.github.com/simo...

Here's how I did that: https://gist.github.com/simonw/63aa33ec827b093f9c6a2797df950...

Here are the top 20 represented language families:

    Niger-Congo 1,019
    Austronesian 609
    Sino-Tibetan 288
    Indo-European 278
    Afro-Asiatic 222
    Trans-New Guinea 219
    Otomanguean 149
    Nilo-Saharan 131
    Austro-Asiatic 100
    Dravidian 60
    Australian 51
    Creole 45
    Kra-Dai 43
    Uto-Aztecan 41
    Quechuan 36
    Language isolate 35
    Torricelli 32
    Maipurean 31
    Mayan 30
    Sepik 30

re5i5tor• 3 years ago

Thanks -- great way to visualize how massive this set of languages really is.

jimmySixDOF• 3 years ago

All this great work and the prior sota translations and Meta still only accepts Nort American english voice control in their VR equipment lol.

cleverwebble• 3 years ago

Wow, I didn't even know there was 7,000 documented languages in the world!

neom• 3 years ago

"According to the World Atlas of Languages' methodology, there are around 8324 languages, spoken or signed, documented by governments, public institutions and academic communities. Out of 8324, around 7000 languages are still in use."

https://en.wal.unesco.org/discover/languages

mkl• 3 years ago

Most are at risk of extinction.

"half of the languages spoken today have fewer than 10,000 speakers and that a quarter have fewer than 1,000 speakers" (https://en.wikipedia.org/wiki/Language_death).

"Today, on average, we lose one language in the world every six weeks. There are approximately 6800 languages. But four percent of the population speaks 96 percent of the languages, and 96 percent of the population speaks four percent of the languages. These four percent are spoken by large language groups and are therefore not at risk. But 96 percent of the languages we know are more or less at risk. You have to treat them like extinct species." (https://en.wikipedia.org/wiki/Language_preservation).

"Over the past century alone, around 400 languages – about one every three months – have gone extinct, and most linguists estimate that 50% of the world’s remaining 6,500 languages will be gone by the end of this century (some put that figure as high as 90%, however). Today, the top ten languages in the world claim around half of the world’s population." (https://www.bbc.com/future/article/20140606-why-we-must-save...).

_the_inflator• 3 years ago

Even Bavarian is covered... :D

echelon• 3 years ago

The MMS code and model weights are released under the CC-BY-NC 4.0 license.

Huge bummer. Prevents almost everyone from using this and recouping their costs.

I suppose motivated teams could reproduce the paper in a clean room, but that might also be subject to patents.

reaperman4• 3 years ago

Id imagine you could use inference from this as training for a commercial model, as that isn't currently protected under copyright.

EMIRELADERO• 3 years ago

I'd go a step further and say that the models themselves probably aren't copyrightable.

Etheryte• 3 years ago

I think this is a lot better than the other option, which would've been not releasing it at all. No company in the business of making money wants to give away their edge to would-be competitors for free.

jewel• 3 years ago

Trying to attach a license to model weights seems counter-productive to me. If you argue they are copyrightable then surely they are derivative works of the training data, which is unlikely to all be public domain. Machine learning enthusiasts are better off lobbying for model weights being non-copyrightable as it doesn't have any creative input and is the result of a purely mechanical process.

The copyright on the code, on the other hand, would definitely be copyrighted and would need a clean-room implementation, as you said. The community could pool its resources and do it once, and license it under the AGPL to keep it and further improvements available to everyone.

fulafel• 3 years ago

My layman's reading is that the license does seem to allow recouping of costs, would be interested if there is a more nuanced interpretation to the contrary.

(tangentially, like another comment briefly mentioned, are models actually copyrightable? Programs are, because they are human authored creative works.)

famouswaffles2• 3 years ago

Meta on a roll. any demo on how good the text to speech is ?

crakenzak3• 3 years ago

Doesn't seem like there's a demo set up by them yet, but you can just download the model weights and run the inference yourself and compare it to OpenAI Whisper or anything you have access to.

famouswaffles2• 3 years ago

yeah thanks. just saw that. fingers crossed it's really good for korean. i'm just glad they released it at least. the other sota TTS models by big companies won't be seeing the light of day.

dsrtslnd233• 3 years ago

reproducing the english TTS demo resulted in very underwhelming quality.

pruthvishetty• 3 years ago

This looks huge. Anyone know how this compares with Whisper in terms of quality and speed?

crakenzak3• 3 years ago

according to their blog post[1], MMS achieves ~half the error rate on words, while supporting 11x more languages. pretty impressive.

[1] https://ai.facebook.com/blog/multilingual-model-speech-recog...

youssefabdelm• 3 years ago

I wonder what the performance is on English specifically.

Edit: Just checked the paper, it seems to be worse[1][2] but feel free to correct me.

I feel like they should've just taken the Whipser architecture, scaled it, and scaled the dataset as they did.

[1] Page: https://i.imgur.com/bq15Tno.png

[2] Paper: https://scontent.fcai19-5.fna.fbcdn.net/v/t39.8562-6/3488279...

93po• 3 years ago

lack of labels on graph axes should be a crime

rvz2• 3 years ago

So many so-called overnight AI gurus hyping about their snake-oil product and screaming about 'Meta is dying' [0] and 'It is over for Meta' but little of them actually do research in AI and drive the field forward and this once again shows that Meta has always been a consistent contributor to AI research, especially in vision systems.

All we can just do is take, take, take the code. But this time, the code's license is CC-BY-NC 4.0. Which simply means:

Take it, but no grifting allowed.

[0] https://news.ycombinator.com/item?id=31832221

azinman2• 3 years ago

Unless you’re in a country that doesn’t care about the IP restrictions, then you do whatever you want.

mirekrusin• 3 years ago

Yes, they are so far more open than open<for business>ai.

reaperman4• 3 years ago

I assume this "competes" directly with https://sites.research.google/usm/ -- would be cool to see side-by-side benchmarks sometime! Maybe I should make those. I requested access to USM but have not been granted any access yet.

minhazm• 3 years ago

Is there any indication that USM will be open sourced though? This is more so competing with Whisper.

https://github.com/openai/whisper

freediver• 3 years ago

Come to think about it, Meta is a much better name for an AI company than a VR company.

Hamuko• 3 years ago

It's not VR, it's Metaverse; VR might involve experiences that are not bullshit.

idiotsecant• 3 years ago

The metaverse, if we define it as an information layer that exists in parallel with the physical 'stuff' universe and is a seamless, effortless, and essential part of what we experience as reality will be an enormous part of our future. Meta might just be a few centuries ahead of the curve, which is just as bad as being a few centuries late.

astrange• 3 years ago

If anything, it sounds like the start of a "Meta? I don't even know her" joke.

richard___• 3 years ago

meta is doing more for open ai than openai

nblgbg• 3 years ago

this is absolutely true, with pytorch, llama, ton of models in vision and now with this !

visarga• 3 years ago

We're in upside-down world, Meta is now cool; actually I always admired their open source projects, starting with React and PyTorch.

sabareesh• 3 years ago

What an irony !

s1k3s3• 3 years ago

That's really not hard to do, is it?

justapassenger• 3 years ago

For OpenAI, AI is the revenue source. For Meta, it's a tool used to build products.

If OpenAI gives their stuff away, they lose customers. If Meta does it, they can have community around it, and have joint effort about improving tools that then they'll use for their internal products.

OpenAI is modern (and most likely - very short lived) Microsoft in the AI space, while Meta tries to replicate Linux in the AI space.

jchw2• 3 years ago

I don't think anyone is too surprised by the fact that OpenAI went closed. As ironic as it may be, it was pretty obviously the only way they were going to continue to exist.

On the other hand, it definitely underscores:

- How blatantly companies exploit the "open source" concept with no remorse or even tacit acknowledgement; for many startups, open source is a great way to buy some goodwill and get some customers, then once they have what they want, close the doors and start asking for rent money. Nothing wrong with doing closed source or commercial software, but would OpenAI still have gained relevance if they hadn't started the way they did with the name they did?

- How little anyone gives a shit; we all watched it happen, but apparently nobody really cares enough. Investors, customers, the general public, apparently all is fair in making money.

I'm not suggesting OpenAI is especially evil, definitely not. In fact, the most depressing thing is that doing this sort of bait and switch mechanic is so commonplace and accepted now that it wasn't news or even interesting. It's just what we expect. Anything for a dollar.

But maybe I still seem like I'm just being whiny. Okay, fair enough. But look at what's happening now; OpenAI wants heavy regulation on AI, particularly they want to curtail and probably just ban open source models, through whatever proxy necessary, using whatever tactics are needed to scare people into it. They may or may not get what they want, but I'm going to guess that if they do get it, ~nobody will care, and OpenAI will be raking in record profits while open source AI technology gets pushed underground.

Oh I'd love to be wrong, but then again, it's not like there's anything particularly novel about this strategy. It's basically textbook at this point.

jokethrowaway2• 3 years ago

Because OSS doesn't work at scale, the economic incentives are not aligned.

Sure, it may work sometimes with the goodwill of someone who cares, but 90% of OSS code is a dead portfolio the authors built with the hope of landing a tech job or skipping some algorithm questions.

Sure, OSS allow people to experiment with crap for free (even though it's mostly big corps benefiting from OSS) but what about the negative effects OSS produce on small businesses?

How many developers could spend their life maintaining small parts of software instead of working in soulless corporations, if giving away your code for free (and without maintenance) wasn't so common? How much better would this code be compared to the wasteland of OSS projects? How much more secure could the entire ecosystem be? How many poor developers are working for free just in the hope of getting a job someday?

We need to stop dreaming the OSS dream and start making things fairer for the developers involved.

flangola72• 3 years ago

I don't think open source LLMs over a certain capability will be legal soon, in the same way an open source SDR transmitter would not be legal.

egberts14• 3 years ago

Does it do American Sign Language, US fifth largest language?

I didn’t think so.

ccooffee2• 3 years ago

American Sign Language, ISO-693-3 code 'ase', does not have a formal written or spoken grammar. Obviously, the list of MMS supported languages[0] does not include support for 'ase' because MMS is a tool for spoken language and, as an intermediate state, written language.

[0] https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mm...

crazygringo4• 3 years ago

Perhaps you could help by inventing a written version of ASL, since one doesn't currently exist. [EDIT: I was wrong, sorry, see my response below.] Seems like that would be a prerequisite for a model based on written language.

And of course if you could also create an entire corpus of training material in your written ASL?

egberts14• 3 years ago

It is called SignWriting for all you multiple naysayers.

https://en.m.wikipedia.org/wiki/SignWriting

nmfisher• 3 years ago

How would a speech model do sign language?

m3kw9• 3 years ago

The problem with all these model releases is they have no demos or even video of it working. It’s all just download it and run it, like it’s an app.

s1k3s3• 3 years ago

I'd argue that having a "download and run" approach is so much better than videos or demos. Why do you think this is a problem?

simonw4• 3 years ago

Because to download and run it you need to have a laptop nearby to download and run it on, with the correct operating system and often additional dependencies too.

I do most of my research reading on my phone. I want to be able to understand things without breaking out a laptop.

yjftsjthsd-h• 3 years ago

It's a lot bigger investment. It'd be nice to at least see a video; that's easier than expecting people to download and run something just to see it.

digging• 3 years ago

Why should we have any marketing materials, ever? Why do we show pictures of products? Sometimes people want to see the capabilities before they completely dive in and spend their time working on something.

crazygringo4• 3 years ago

They're intended for researchers/professionals not consumers, and I'm not sure how a video is going to be helpful?

And the issue with a live demo is that these are resource-intensive, they're not just webpages. It's an entire project to figure out how to host them, scale them to handle peaks, pay for them, implement rate-limiting, and so forth.

For the intended audience, download-and-run-it doesn't seem like an issue at all. I don't see how any questions are going to be answered by a video.

tmpz22• 3 years ago

Its so weird to me that they'd do 99% of the effort and just skip past the 1% of work to provide a dumbed-down summary and instructions for broader appeal. Clearly these are released in part for public-relations and industry clout.

Don't get me wrong they published this, it took a ton of work, they didn't have to do it. But its ultimately a form of gatekeeping that seems to come straight out of Academia. And honestly, that part of Academia sucks.

barking_biscuit2• 3 years ago

We're all research professionals now.

throwbackhere• 3 years ago

Even researchers and professionals appreciate a sample or two.

Ninjinka• 3 years ago

Here's the blog post with video of it working: https://ai.facebook.com/blog/multilingual-model-speech-recog...

leke• 3 years ago

I was kind of hoping for Interlingue, but was surprised to not even see Esperanto on the list.

samstave• 3 years ago

Random thought ; could Esperanto (as a crypto-language) be turned into any sort of programming language. Could one, conceivably program in Esperanto in any meaningful way?

felipesabino• 3 years ago

I think you are more likely to do so in a more logical and structured language like lojban

int_19h• 3 years ago

There's nothing particularly special about Esperanto in that regard compared to natural languages that it is modelled after.

explodingcamera• 3 years ago

Esperanto is supported in their larger model

armatav4• 3 years ago

Imagine if we used these types of models for like 500 years and it locked their vocabulary in time, disallowing any further language blending; then somehow the servers turned off and nobody could communicate across language barriers anymore.

Someone should write that down in some sort of short-story involving a really tall structure.

l33t233372• 3 years ago

Imagine if we used them everyday for 500 years but tune all the languages to be ever so slightly similar each day until there is one language.

armatav4• 3 years ago

Call it “Inglesh” or something phonetically similar

numpad0• 3 years ago

With ~1000 dialects.

hzay2• 3 years ago

Imagine if someone said this about dictionaries.

armatav4• 3 years ago

Imagine if your dictionary was constantly updated to modify your behavior

fennecfoxy2• 3 years ago

When a jet skier accidentally runs into a tree that's called a snow crash, aha

ripvanwinkle• 3 years ago

Anyone know what hardware it takes to run this? Asking as an enthusiastic newbie

detrites• 3 years ago

At a guess, based on model size, just about anything, even a Raspberry Pi 4 etc.

As comparison the GGML port of Whisper (OpenAI's equivalent), runs in the browser, via WASM: https://whisper.ggerganov.com/

qumpis2• 3 years ago

If this is the case, what hardware is needed for real-time tts?

EvgeniyZh• 3 years ago

Most of languages support only LID (language identification) task. Still impressive

qwertox• 3 years ago

I would like to use stuff like this as a side-project. Buy a Nvidia Geforce GPU and stick it into my 24/7 server and play around with it in my free time, to see what can be done.

The issue with all these AI models is that there's no information on which GPU is enough for which task. I'm absolutely clueless if a single RTX 4000 SFF with its 20GB VRAM and only 70W of max power usage will be a waste of money, or really something great to do experiments on. Like do some ASR with Whisper, images with Stable Diffusion or load a LLM onto it, or this project here from Facebook.

Renting a GPU in the cloud doesn't seem to be a solution for this use case, where you just want to let something run for a couple of days and see if it's useful for something.

nharada• 3 years ago

Wait why is renting a GPU in the cloud not a solution? You can even try multiple options and see which ones are capable enough for your use case.

Look into some barebones cloud GPU services, for example Lambda Labs which is significantly cheaper than AWS/GCP but offers basically nothing besides the machine with a GPU. You could even try something like Vast in which people rent out their personal GPU machines for cheap. Not something I'd use for uhhh...basically anything corporate, but for a personal project with no data security or uptime issues it would probably work great.

fnord77• 3 years ago

"but offers basically nothing besides the machine with a GPU"

they must offer distributed storage, that can accommodate massive models, though? how else would you have multiple GPUs working on a single training model?

itake4• 3 years ago

My annoyance was managing state. I’d have to spend hours installing tools, downloading data, updating code, then when I want to go to bed I have to package it up and store as much as I can on s3 before shutting off the $$ server.

nomel• 3 years ago

Besides some great tooling out there if you wanted to roll your own, you can literally rent windows/linux computers, with persistent disks. If you have good internet, you can even use it as a gaming PC, as I do.

ginko• 3 years ago

Having to connect to a gpu over the internet seems extremely cumbersome. Stuff like this should be as easy as running a local program with an accelerator.

chaxor2• 3 years ago

Cloud has a variable price (up to the whim of whatever they decide the price to be that day) so it's uncertain, but typically it is far" more expensive for this type of application. So when faced with 1) probably far more expensive, or 2) single price that will be cheaper, is always available, and has far more uses, I think most would choose 2) for self-hosting. Cloud is very rarely* a good option.

anonzzzies• 3 years ago

I have not seen many setups that wouldn’t pay itself back (including energy in my case) within a year (sometimes even 6 months) with buying vs renting. For something that pays itself back that fast, and that is without renting it out myself, just training with it, I cannot see how I would want to rent one.

Edit; on Lambda labs, the only exception seems to be the H100; it would be 1.5 years or so, but even 2 years would still fast enough. I have an A100 which paid itself back; thinking of getting another one.

ttt3ts4• 3 years ago

You can finetune whisper, stable diffusion, and LLM up to about 15B parameters with 24GB VRAM.

Which leads you to what hardware to get. Best bang for the $ right now is definitely a used 3090 at ~$700. If you want more than 24GB vram just rent the hardware as it will be cheaper.

If you're not willing to drop $700 don't buy anything just rent. I have had decent luck with vast.ai

NBJack2• 3 years ago

There is the world of used Nvidia Teslas, like the M40. Very cheap, but some assembly required.

aci_12• 3 years ago

Can we use 3090 or nvidia gpus in general with mac? do people generally have their windows dekstop for the gpus?

syntaxing• 3 years ago

I would recommend a 3090. It can handle everything a 4000 series can albeit slightly slower, has enough VRAM to handle most things for fun, and can be bought for around $700.

ra• 3 years ago

I think you just need to educate yourself a bit about the space. These models are very small (the large version is only 1B parameters) so should run on a 4GB gaming GPU.

slowmovintarget• 3 years ago

Here's a sheet I came across today...

https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_...

Granted, it's talking about quantized models, which use less memory. But you can see the 30B models taking 36 GB at 8-bit, and at least 20 GB at 4-bit.

The page even lists the recommended cards.

But as others have pointed out, you may get more bang "renting" as in purchasing cloud instances able to run these workloads. Buying a system costs about as much as buying instance time for one year. Theoretically, if you only run sporadic workloads when you're playing around it would cost less. If you're training... that's a different story.

bigdict• 3 years ago

A 3080 Ti is sufficient to run the largest Whisper model (just under 12 GB VRAM).

segmondy• 3 years ago

Just do it. Spend a few hours doing research and you will find out. With that said, buy as much memory as can. That makes 4090 king if you have the server that can carry it, plus the budget. For me, I settled for 3060, it's a nice compromise between cost and ram. Cheap, 12gb and 170TPW.

lhl• 3 years ago

The more VRAM the better if you'd like to run larger LLMs. Old Nvidia P40 (Pascal 24GB) cards are easily available for $200 or less and would be easy/cheap to play. Here's a recent writeup on the LLM performance you can expect for inferencing (training speeds I assume would be similar): https://www.reddit.com/r/LocalLLaMA/comments/13n8bqh/my_resu...

This repo lists very specific VRAM usage for various LLaMA models (w/ group size, and accounting for context window, which is often missing) - this are all 4-bit GPTQ quantized models: https://github.com/turboderp/exllama

Note the latest versions of llama.cpp now have decent GPU support and has both a memory tester and lets you load partial models (n-layers) into your GPU. It inferences about 2X slower than exllama from my testing on a RTX 4090, but still about 6X faster than my CPU (Ryzen 5950X).

Again this is inferencing. For training, pay attention to 4-bit bitsandbytes, coming soon: https://twitter.com/Tim_Dettmers/status/1657010039679512576

KRAKRISMOTT• 3 years ago

Use the A100 as a base reference and work your way down from there.

archon1410• 3 years ago

ASR: "Automatic Speech Recognition"; also known as "Speech to Text" (STT)

TTS: "Text to Speech"

LID: "Language Identification"

In case anyone else was confused about what the acronyms mean.

taneq• 3 years ago

TLA: “Text LLM Application.” ;)

lairv3• 3 years ago

I don't have much knowledge about TTS models, is it possible/affordable to fine-tune those models on your own voice ?

alienlid• 3 years ago

if I have an extra MBP 16" 2020 hanging around 16GB Ram, Quadcore i-7.... can I run this? I'd like to try TTS capabilities! LMK if you've got any guides or instructions online I can checkout:)

sigstoat• 3 years ago

can any of these models be coerced into just doing straight phonetic transcription? like spitting out IPA?

rllearneratwork• 3 years ago

Real Open AI lab.

rvz2• 3 years ago

Exactly.

vlugorilla• 3 years ago

Would it be possible to have this in somethin "a la" whisper.cpp?

TOMDM• 3 years ago

If it performs as well as the paper implies, then I certainly hope someone will pick it up and give it the same treatment.

stanislavb• 3 years ago

I still hate the name “meta”. You?

eigenvalue• 3 years ago

I just wanted to test out the TTS locally on a powerful Ubuntu 22.04 machine, but the process for setting it up seems pretty broken and poorly documented. After 20 minutes of trying I finally gave up since I couldn't get the VITS dependency to build (despite having a fully updated machine with all required compilers). It seems like they never really bother to see if the stuff works on a fresh machine starting from scratch. Somehow for my own projects I'm always able to start from a fresh git clone and then directly install everything using this block of code:

``` python3 -m venv venv source venv/bin/activate python3 -m pip install --upgrade pip python3 -m pip install wheel pip install -r requirements.txt ```

But whenever I try using these complicated ML models, it's usually an exercise in futility and endless mucking around with conda and other nonsense. It ends up not being worth it and I just move on. But it does feel like it doesn't need to be like this.

dragonwriter• 3 years ago

Yeah, a lot of the releases from researcg groups are woefully poorly documented.

Usually, some hero releases a friendly install system within a few days, though.

lairv3• 3 years ago

I wrote a colab to run TTS, it takes some time to install dependencies : https://colab.research.google.com/drive/11tZ1c7kQJ5uKImm9t-C...

Overall I got pretty poor results in english and french, I guess it would require some fine-tuning

deadmutex• 3 years ago

Thanks for this! Out of curiosity, how does vits play into this? I know nothing about the repo.

Also, I used the following to play the sound after it is generated:

from IPython.display import Audio, display display(Audio(output_file, autoplay=True))

ElectricalUnion3• 3 years ago

This thing uses pytorch, the bane of everything that attempts to make pypi installs sane.

As a human, you need to read https://pytorch.org/get-started/locally/ and install the correct version, depending on your pytorch-version/os/packaging-system/hardware-platform combo. It's bad. It's several conflicting index-urls messing with your requirements. It also doesn't work if you pick the wrong thing.

Or we need something like a "setup.py" that is turing-complete, to dynamically pick up the correct dependency for you.

`pip install -r requirements.txt` is not enough.

PufPufPuf2• 3 years ago

Research projects are notorious for this. The people behind them aren't very experienced in distributing software (many never heard of docker), the usual language of choice is Python which has poor packaging, native dependencies complicate things, and everything is on the bleeding edge by definition.

Usually, it needs a brave soul to reverse engineer what exact versions of dependencies are needed and make a Colab or Dockerfile that puts it all together.

evilhackerdude• 3 years ago

I've created this environment.yml:

  name: fairseq
  channels:
    - conda-forge
    - pytorch
  dependencies:
    - python=3.9
    - Cython==0.29.21
    - librosa==0.8.0
    - matplotlib=3.3
    - numpy=1.19
    - scipy==1.5.2
    - tensorboard==2.3.0
    - pytorch=1
    - torchvision=0
    - Unidecode==1.1.1
    - pip:
        - phonemizer==2.2.1

You can install it with micromamba (or conda):

  conda create -f environment.yml
  conda activate fairseq

You'll need to do this:

  cd path/to/vits/monotonic_align
  mkdir monotonic_align
  python setup.py build_ext --inplace

Then back to fairseq:

  cd path/to/fairseq
  PYTHONPATH=$PYTHONPATH:path/to/vits python examples/mms/tts/infer.py --model-dir checkpoints/eng --wav outputs/eng.wav --txt "As easy as pie"

(Note: On MacOS, I had to comment out several .cuda() calls in infer.py to make it work. But then it generates high-quality speech very efficiently. I'm impressed.)

sarabande• 3 years ago

I'm trying to use this on a 3M mp3 file to test ASR with language code deu, CPU only, and I keep getting this error -- are there limits to the MMS inference?

  File "fairseq/data/data_utils_fast.pyx", line 30, in fairseq.data.data_utils_fast.batch_by_size_vec
    assert max_tokens <= 0 or np.max(num_tokens_vec) <= max_tokens, (
  AssertionError: Sentences lengths should not exceed max_tokens=4000000
  Traceback (most recent call last):
  File "/home/xxx/fairseq/examples/mms/asr/infer/mms_infer.py", line 52, in <module>
    process(args)
  File "/home/xxx/fairseq/examples/mms/asr/infer/mms_infer.py", line 44, in process

willn• 3 years ago

Same error here, CPU only, haven't find out a way to run this on GPU (cuda) yet

sacnoradhq• 3 years ago

FYI: Another round of massive layoffs at Meta this Wednesday. Stay-at-home lockdown ensues.

gagabity• 3 years ago

I just want to translate a movie language audio from one to another, whats the easiest way to do this at home?

OkGoDoIt• 3 years ago

According to [1] on the accompanying blog post, this brings the Whisper 44.3 WER down to 18.7, although it’s unclear to me how much better this is at primarily English speech recognition. I’d love to see a full comparison of accuracy improvements as well as a proper writeup of how much more power it takes to run this in production or on mobile vs something like whisper.

[1] https://scontent-sjc3-1.xx.fbcdn.net/v/t39.8562-6/346801894_...

omneity2• 3 years ago

I was super excited at this, but digging through the release [0] one can see the following [1]. While using Bible translations is indeed better than nothing, I don't think the stylistic choices in the Bible are representative of how people actually speak the language, in any of the languages I can speak (i.e. that I am able to evaluate personally).

Religious recordings tend to be liturgical, so even the pronunciation might be different to the everyday language. They do address something related, although more from a vocabulary perspective to my understanding [2].

So one of their stated goals, to enable people to talk to AI in their preferred language [3], might be closer but certainly a stretch to achieve with their chosen dataset.

[0] https://about.fb.com/news/2023/05/ai-massively-multilingual-...

[1]: > These translations have publicly available audio recordings of people reading these texts in different languages. As part of the MMS project, we created a dataset of readings of the New Testament in more than 1,100 languages, which provided on average 32 hours of data per language. By considering unlabeled recordings of various other Christian religious readings, we increased the number of languages available to more than 4,000. While this data is from a specific domain and is often read by male speakers, our analysis shows that our models perform equally well for male and female voices. And while the content of the audio recordings is religious, our analysis shows that this doesn’t bias the model to produce more religious language.

[2]: > And while the content of the audio recordings is religious, our analysis shows that this doesn’t bias the model to produce more religious language.

[3]: > This kind of technology could be used for VR and AR applications in a person’s preferred language and that can understand everyone’s voice.

generalizations3• 3 years ago

This paragraph justifies the decision, IMHO.

Collecting audio data for thousands of languages was our first challenge because the largest existing speech datasets cover 100 languages at most. To overcome this, we turned to religious texts, such as the Bible, that have been translated in many different languages and whose translations have been widely studied for text-based language translation research.

I think the choice was just between having any data at all, or not being able to support that language.

omneity2• 3 years ago

Of course, my critique is less "This project shouldn't exist", but rather "It seems to me there are several biases that affect the performance of this project in the context it was presented in".

This is a great project and an important stepping stone in a multilingual AI future.

westurner2• 3 years ago

While AFAIU the UN: UDHR United Nations Universal Declaration of Human Rights is the most-translated document in the world, relative to the given training texts there likely hasn't been as much subjective translation analysis of UDHR.

Awesome-legal-nlp links to benchmarks like LexGLUE and FairLex but not yet LegalBench; in re: AI alignment and ethics / regional law https://github.com/maastrichtlawtech/awesome-legal-nlp#bench...

A "who hath done it" exercise:

[For each of these things, tell me whether God, Others, or You did it:] https://twitter.com/westurner/status/1641842843973976082?

"Did God do this?"

sangnoir2• 3 years ago

I don't think the stylistic choices in the Bible are representative of how people actually speak the language, in any of the languages I can speak

The cadence and intonation sounded a little weird, but I suspect fine-tuning can improve that by a lot. I am really excited to see some low-resource language finally get some mainstream TTS support at all.

kaycey20222• 3 years ago

Why is Meta open sourcing their AI work like this? Is it because they don't have a great reputation even among tech companies?

ryanar• 3 years ago

My speculation: the repo hasnt been in active development for at least a year, all the main contributors left to found Character.AI, so Meta is open sourcing it because it only rots sitting there and going open means goodwill, and maybe someone else picks it up and runs with it and contributes back to make it more useful.

ipsum2• 3 years ago

Meta has had a culture of open source, even before their reputation got bad (React, RocksDB, Thrift, folly, etc). They open source a lot of their AI models (instead of just source) because they don't have an enterprise cloud to sell these things, unlike Amazon/Google/Microsoft's offerings (which all offer speech transcription and text to speech as services).

fomine3• 3 years ago

This is an underrated point for Meta. They are good at opensourcing.

kaycey20222• 3 years ago

I understand that. I’m asking in the context that a number of business commentators have come out and said that Google open sourcing the transformer technology was in itself a big mistake, in hindsight. I think the business side in Meta would have taken a similar view.

throwme_123• 3 years ago

fairseq is a fairly missed naming opportunity. C-3PO would have been better.

woodson• 3 years ago

fairseq has existed for many years and is Fb’s (now Meta’s) AI research group’s sequence-to-sequence modeling toolkit. It’s not specific to this MMS model.

oars• 3 years ago

Some interesting links and tools in this thread, e.g. datasette.io

dsrtslnd233• 3 years ago

I tried the english TTS example and the result is quite underwhelming (compared to bark or polly/azure-tts ). It sounds like TTS systems one or two decades ago. Would those language-specific TTS models need to be finetuned?

Simon_O_Rourke• 3 years ago

Maybe one of those models has figured out a way to tell Zuck that the whole Metaverse concept is nonsense, hopefully it'll be graceful about letting him down.

PufPufPuf2• 3 years ago

1107 languages but no Czech or Slovak? Many languages with way fewer speakers made it to the list. I wonder what we did to Meta...

dsrtslnd233• 3 years ago

no chinese TTS either.

2Gkashmiri• 3 years ago

i checked the language coverage for "kashmiri"

https://www.ethnologue.com/language/kas/

"Himachal Pradesh state:"

this is obviously wrong so i dont know what else is wrong

neycoda• 3 years ago

Great, now the Terminators will be barking orders at each other in languages I can't understand.

feim_2022• 3 years ago

Wonder how this compares with deepgram offering. has anyone used/tried/compared or even read enough literature to compare. The WER rates showed in deepgram are still better than the largest MMS and the specific use case based fine-tuned models (zoom meetings, financial calls etc) probably make a bigger difference. WDYT ?

Meta AI announces Massive Multilingual Speech code, models for 1000+ languages

Related Stories