Mocking Bird – Realtime Voice Clone for Chinese

https://github.com/babysor/MockingBird

by axiomdata316 • 5 years ago

163 64 5 years ago

github.com

goblin896• 5 years ago

Supports only Mandarin, so description is slightly disingenuous.

fortran77• 5 years ago

12% of the world speaks Mandarin, so it's a good first choice!

goblin896• 5 years ago

This is not about some sort of language leaderboard. If you see the title (as phrased) written in %language%, you’d expect the proposed thing to work in that language.

Anyway, as someone else pointed out it’s a fork, so it’s understandable if docs weren’t fully updated from upstream.

drekk• 5 years ago

It's forked from an English version

goblin896• 5 years ago

Interesting, what’s the upstream? Must’ve missed the link.

Asraelite• 5 years ago

For anyone else wondering, the original title was "Clone a voice in 5 seconds to generate arbitrary speech in real-time"

goblin896• 5 years ago

Yes. I reckon “Mocking Bird – Realtime Voice Clone for Mandarin” might’ve been a more appropriate change, but it’s always easy to criticise after the fact…

echelon3• 5 years ago

I'm the author of FakeYou.com, so I have a little experience in this area. (We used to train GlowTTS models ourselves before turning it over to our users, which has had mixed results in terms of quality.)

This appears to be a repackaging of RealTimeVoiceCloning [1], albeit with a few additions, such as GSTs.

No matter what the repo claims, your results will depend on high quality data. Lots of it, and with ample fine tuning. Demo videos are absolutely cherry picked.

If you're picking this up for a project, HiFi-Gan is pretty much the best vocoder right now. Tacotron still produces great results, though there are lots of other interesting model architectures.

[1] https://github.com/CorentinJ/Real-Time-Voice-Cloning

weinzierl3• 5 years ago

By any chance, do you know which technology was used for the TikTok voice? While the voice itself is somewhat annoying I find the quality stunning. Any chance to reach this level with any of the models you mentioned?

Gigachad• 5 years ago

Is the quality that good? It seemed to me that text to speech was not a particularly hard problem when you can get someone to read out all the sounds of the language. And that the new advancement we have now is just being able to dump our normal speech audio and build a model based on that.

JohnHaugeland4• 5 years ago

i actively dislike the tiktok voice. i find the microsoft and google cloud voices to be much better.

pdamoc• 5 years ago

Long ago I found an approach to 3D modeling [1] that used a morphable model that was then morphed into the desired shape. Would something like this be possible for voice? A voice model obtained from a gigantic set of samples, that can be manually tuned to sound more masculine/feminine, higher/lower pitched and that can be morphed into the timbre of various samples.

- [1] https://www.youtube.com/watch?v=pSRA8GpWIrA

kragen5• 5 years ago

By "lots of [high quality data]" do you mean seconds, kiloseconds, or megaseconds of high-quality voice recordings?

oefrha2• 5 years ago

IIRC enterprise solutions from the big clouds usually ask for at least hours of studio quality voice recordings for a custom voice model.

iaml• 5 years ago

This paper[0] from this year seems to make do with a couple of minutes.

[0] https://davidyao.me/projects/text2vid/

echelon3• 5 years ago

Hours of audio comprised of clean spoken sentences, zero noise, and uniform microphone quality is ideal.

Some of the predominant base data sets used for transfer learning, such as LJSpeech [1], are unfortunately noisy and non-uniform.

[1] https://keithito.com/LJ-Speech-Dataset/

twistedcheeslet• 5 years ago

I'm very interested in this space. Forgive my ignorance, but what makes this fit for Chinese voices, while unfit for English voices?

oefrha2• 5 years ago

From README:

This repository is forked from Real-Time-Voice-Cloning which only support English.

https://github.com/CorentinJ/Real-Time-Voice-Cloning

JohnHaugeland4• 5 years ago

One of the big things about these projects is training sound that's paired to text.

The base project spoke English because it had been trained on English text paired with English recordings. This speaks Mandarin because it has been trained on paired Mandarin text and recordings.

Amusingly, if you take one trained on English text/recording pairs and feed it French text, it will speak French with an English accent.

Agingcoder2• 5 years ago

This is very interesting. People who have had a laryngectomy can't speak post-operation, and being able to say something with their 'real' voice would be psychologically helpful.

unbanned• 5 years ago

Would it? Unless the sound is produced from inside their body in the same manner as natural, no it will sound weird just like playing back your own voice.

Agingcoder2• 5 years ago

Not necessarily - people close to me were complaining about what they saw as losing a part of their identity ( and grandchildren definitely were not happy about the change).

Even for those who learn to speak with an oesophagal voice, being able to real time map it to something more natural would be great.

To be more specific, they consider themselves to be 'voice mutilated', even when able to speak, so I guess that anything closer to their original voice would be a net benefit.

JohnHaugeland4• 5 years ago

I suspect that it would be immensely better to sound sort of like yourself than like a standard text to speech engine, and those are used extensively

mdp20213• 5 years ago

Some may have missed the news in October (2021) about the 35 million dollars swindle, between UAE and Hong Kong, involving the bank director in HK receiving a call where the voice "of the company director" requested a transfer...

https://www.forbes.com/sites/thomasbrewster/2021/10/14/huge-...

https://www.unite.ai/deepfaked-voice-enabled-35-million-bank...

Not only the voice (simulated through machine learning) was trusted: also (even) emails had been unverified.

pulse7• 5 years ago

So it already started: a new generation of swindling using computer generated voice, video and other artifacts... It didn't take much time to misuse the newest tech...

mdp20213• 5 years ago

The issue is with the workers believing to live in a past distant century and sitting on the idea that current possibilities are "science fiction" - often criminally calibrating the risk on the abilities of the "man in the street", instead of the interested actor: the swindler. As in "But who could and would ever do it?"¹; the answer clearly being: the one really interested in doing it.

I have heard, literally: "Well, if I cannot trust one's email nor one's telephone number [than a scam is like fate]"... From a bank employee. This is dangerously placed ignorance, in subcultures where unduly laxity was accepted.

(¹ Again, an actual reply given to security experts showing systemic faults to managers.)

livinglist• 5 years ago

this is good

ximeng2• 5 years ago

https://www.hsbc.com.hk/ways-to-bank/phone/voice-id/ HSBC Hong Kong allows you to use your voice to log in to phone banking without a PIN. What could go wrong.

MrPatan2• 5 years ago

HSBC everywhere pushes it "for security". This has been insecure since the invention of the tape recorder. HSBC, not even once.

dillondoyle• 5 years ago

I refuse to accept the terms for Schwab's 'voice authentication' it asks me to every time I login.

aasasd2• 5 years ago

Well, with the previous voice-faking tech releases I kinda expected that the scams are gonna evolve from garbled “Hi ma, I've been in a road accident, need to pay off, please send money to this number”. But they escalated quicker than I could guess.

1010082• 5 years ago

I never thought about this scenario but now I am kind of scared. I guess that they would need a lot of data from my own voice to train a model like that. I am a kind of happy now that my podcast was taken down due to copyright infringment :)

lelandfe• 5 years ago

My 90 year old grandma had a fraudster call her last month claiming that her son had been in a car accident and needed money to pay for his emergency surgery

If that call had instead been a pitch-perfect replication of my dad’s voice - or my aunt - I guarantee you it would have worked.

aasasd2• 5 years ago

There was Adobe Voco, which seemed kind of a forerunner: https://www.youtube.com/watch?v=I3l4XLZ59iw, https://en.wikipedia.org/wiki/Adobe_Voco. It purportedly could edit speech like an audio editor and looked like a destroyer of authenticity. And then nothing was heard of it anymore.

(Edit: Wikipedia says that VoCo takes “approximately 20 minutes of the desired target's speech”, and that it was a research prototype.)

There was a thing called Tacotron from a team at Google, in 2018: https://google.github.io/tacotron/publications/speaker_adapt... (In fact, the OP repo and the original CorentinJ/Real-Time-Voice-Cloning apparently rely on Tacotron.)

And there was something from 2019: https://www.ohadf.com/projects/text-based-editing/, https://news.stanford.edu/2019/06/05/edit-video-editing-text...

The latter two seem to need more samples than pure real-time editing.

Overall, to me a layman, this space appears quieter than ‘deep-faking’ videos. Which makes me wonder if I haven't missed something.

meibo• 5 years ago

I imagine that it's inherently easier to create a fake voice that's convincing over a grainy phone line, than it is to make a convincing deep fake video.

Maybe big tech orgs(including Adobe) don't want to risk the liability/PR fallout.

bsenftner• 5 years ago

This is going to be a very abusive advertising technology. We already know the "cocktail party effect" where one can hear their name spoken by a familiar voice in a crowded space with lots of other voices. Expect to hear vaguely familiar voices using your name and personal information to craft advertising that pierces your awareness all the time, at odd times, and distracting you when you're busy.

champagnois• 5 years ago

Does this mean voice assistants and GPS will one day be made consumer friendly and support any voice/name we want them to have, rather than this gimicky brand driven marketing malware known as Alexa or Google or Siri?

VMtest• 5 years ago

I have checked the video, the result is still not satisfying

Cass• 5 years ago

Is there an easy way to make this work for a non-programmer, i.e. without installing a whole python environment? I'm less interested in cloning specific voices than in getting a high-quality text-to-speech program that'll read me arbitrary Mandarin input, which it seems like this should be able to do.

JohnHaugeland4• 5 years ago

ya, microsoft sells it saas under the azure label. you can easily play with their web version, then hit a URL to get it on your credit card.

https://azure.microsoft.com/en-us/services/cognitive-service...

Mocking Bird – Realtime Voice Clone for Chinese

Related Stories