Forget subtitles: YouTube now dubs videos with AI-generated voices

https://restofworld.org/2023/youtube-ai-dubbing-automated-translation/

39 46 3 years ago

superkuh• 3 years ago

It's cool and all but an awkward framing of the feature. Dubs don't really have the same functios as subtitle text. It's apples and oranges.

As for the impact on people who want money to perform audio to audio translation services it sounds a lot like the luddites vs the power-looms. It's too bad for the people in that service industry but I'm 100% happy auto-translation is making everyone more able to communicate.

vouaobrasil3• 3 years ago

I am wondering though: capitalism functions because of comparitive advantage. What happens when comparitive advantage vanishes in the realm of most people except the 0.1% in the tech companies.

It's almost as if AI/technology is reaching a phase where it can reap every advantage, causing a sort of heat death analogous to the heat death in thermodynamics, where trade will eventually cease because no one has any meaningful contribution since AI can do it better.

Maybe this will create a system where no one has to do any work and UBI will take care of everyone, but humans need a purpose and what happens to society when there is mass depression due to a widespread lack of purpose?

supriyo-biswas• 3 years ago

Thank you for this comment, sums up my thoughts perfectly.

I fear a regression of society to the days of zero-sum games. As an example, even though folks may say "open source models" are the answer, the only way to run said models at scale is to acquire physical resources such as electricity and hardware, and physical resources, by their very nature, are subject to zero-sum games.

falcor842• 3 years ago

It's another case of "it is easier to imagine an end to the world than an end to capitalism". I for one can absolutely imagine having a "purpose" without a "job".

Here are a few examples of activities that might give people a sense of purpose: art, traveling and exploring the world, lifelong learning, educating kids/adults about something you love, political activism, socializing in general, scientific inquiry, connecting with and helping animals, catching up on old literature/films/video games. And this is just me brainstorming for 2 minutes.

visarga• 3 years ago

AI/technology is reaching a phase where it can reap every advantage

No, the one who can reap every advantage of the new skills and knowledge learned by AI is actually you. They are all just one prompt away.

no one has any meaningful contribution since AI can do it better

It won't be human vs AI, but human+AI vs human+AI.

Has anyone noticed a trend? every AI task is fragile, they all need human oversight in critical scenarios. Looks like AI is only going to scale as much as we can scale oversight. It will have to be a human+AI team effort.

hfhdjdks2• 3 years ago

This is an old question in economics, but undoubtedly relevant now:

"Is this new technology change different or similar to other technology changes?"

In other technology changes, there are hundreds of jobs that disappear, but hundreds that appears...and, as far as I know, we haven't observed an increase in joblessness in the last, say 30-50 years. We went through the industrial revolution, the invention of personal computers and a big etc and it didn't happen.

Is this one different? I don't have an answer, but I don't think the answer is obvious.

throwaway2904• 3 years ago

If auto dub is as bad as auto generated subtitles, I am not holding my breath...

l3x4ur1n• 3 years ago

What are you on about? Youtube's auto generated subtitles are pretty good. granted, 5-10% of them are miss, but they usually convey at least enough to grasp the context. I would be lost without them (I'd have to be listening to really loud sound and/or focus more intensively to what is said)

throwaway2904• 3 years ago

Your estimate is way too optimistic. But also what it doesn't get is usually key subject content and what it does get is low hanging fruit guessable from context.

Sometimes I try subtitles on something I cannot understand (because English is my second language) and that has not helped ever, it's always gibberish.

I tried going subtitles only a few times and gave up because I have to rewind and turn on sound every few seconds.

Agentlien• 3 years ago

This has really not been my experience... until just now.

A few times before I've tried using auto generated subs and every time I turned them off within a minute because so much of it is nonsense. Last time was about a month ago.

It seemed to get a lot of simple sentences right but every few it completely misparsed. Additionally, it seemed like every uncommon name or technical term became something else.

In light of your comment I went to YouTube to evaluate it again and... it's a lot better than I remember! I watched a few minutes of a video analysing some music and everything, including music terms, was correct.

looperhacks• 3 years ago

Maybe for american English, but put an accent in there, or even just British English and it goes down, fast. I'm currently watching Taskmaster on YouTube and every time it's not crystal clear audio with no background noice, it's closer to 50% miss

pasttense01• 3 years ago

Some of us who are English speaking still read subtitles on English speaking media because of lack of clarity of the voices or we are hard of hearing.

almostnormal• 3 years ago

If the dubbing machine can understand, it could produce a more readable version in the original language, not only a different one.

hiccuphippo2• 3 years ago

If it's the same machine as the one that generates the captions, then it's not very reliable yet.

rolph• 3 years ago

subtitles can be anything you want if you have an editor with subtitle functionality. it could be the dialogue from the video; it could be "thought bubbles" for the characters, or just ascii strings of whatever code.

in a "good enough" sense end user subtitled video has potential use for steganographic methods.

jfernandezr• 3 years ago

Some of us use subtitles because we don't want to listen to the audio in situations where that would be disrespectful, like in the gym.

In Spain there is a big dubbing industry with very good professionals that are able to transmit the acting emotions nearly as good as the original (but never the same). But listening to automated AI dubbing would never reach that level of acting, sounds awkward. Even listening to "latin" spanish dubbing seems out of phase to most of us.

Anyway, YouTube should put some efforts on reviewing the (at least) iOS application localization in spanish, because it's impossible to see at a glance the age of a video in a video listing.

Moldoteck• 3 years ago

AI dubbing would never reach that level of acting, sounds awkward. - you should consider being more conservative in this direction)) I've said the same thing abt generating of images 5 years ago but the last progress of midjourney/dalle proved me wrong. We should not underestimate what hights human progress can achieve

danielbln2• 3 years ago

Op also clearly hasn't used Elevenlabs or similar tools. If you clone a professional narrator it already sounds incredibly good and effectively indistinguishable from a human. Giving acting directions to the model to steer the output (kind of like ControlNet does for Stable Diffusion) seems like a logical next step.

hiccuphippo2• 3 years ago

Funny, for me the Latin Spanish acting sounds better and the Spain ones sound stiff. And for video games, the acting from Japanese voice actors usually feels better than the English ones.

I guess it depends more on what you are used to than the actual acting.

eddd-ddde• 3 years ago

Spanish dubs are consistently the worse you can ever hear. Latin dubs are amazing sometimes even better than the original.

haunter• 3 years ago

Not Youtube but I saw EZDubs on Twitter recently and honestly the results are not that bad

Here is the Messi post match interview after his first game, mind you him and the interviewer not even speaking the same spanish dialect

https://twitter.com/MLSes/status/1682579623089930241

And this was the english dubbed version https://twitter.com/ezdubs_bot/status/1682612908604157952 There are some mistakes like free throw instead of free kick but apart from that it's not that bad

butz• 3 years ago

Great, in a few years Youtube will be able to generate new videos with some sort of LLM for video equivalent and stop paying creators. And include as much sponsorships and ads as possible too.

guerrilla2• 3 years ago

It is great because the demand for authentic content won't go away so this will create new networks which focus on promoting and ensuring that.

tonnydourado• 3 years ago

I think I'd have a hard time coming up with a feature I wanted less than this even if I tried.

guerrilla2• 3 years ago

I wonder if this could be used for language-learning. One of the biggest problem with learning certain small languages is that there's basically no interesting content produced in them (depending on your interests of course, but I ran into this problem multiple times.) For example, everything technical is done in English these days so you won't find too much advanced stuff on certain topics in Swedish, so you get stuck with repeating basics which gets boring after a while. There's also only certain countries which produce the kind of drama series or movies you want to watch. Sure, every country has some, but not always a lot, because again they import.

rcarmo• 3 years ago

As a native Continental Portuguese speaker, I find the Brazilian Portuguese translation jarring, and would never use it.

d3vmax• 3 years ago

While this may not be available to everyone right now, I am using a service from Finland https://www.hei.io/ to help me auto generate subtitles and "AI" voiceover.

Upload youtube link/video, Generate eng subs. Fix english subs manually, auto generate subs into any of the popular languages (we have req. for Asian langs.). Manually fix the auto-generated asian subtitle from minor errors. Reupload it and get perfect true "AI" Voiceover. All AI voice over services will have errors in translation/meaning. If you want quick job, this does it. If you want accurate job, manually fix the subs created by such services.

Anyways, what used to take us 2 weeks, takes us a day or two.

WirelessGigabit• 3 years ago

I want subs. I learned English through watching English TV with subtitles in my native language.

And the generation after me started watching dubbed kids tv. Large decline in their English language skills.

hfhdjdks2• 3 years ago

I grew up used to reading subtitles. Listening to the original voices and languages is super interesting to me. Besides some very specific cases (I can't look at the screen all the time) I don't have any use for dubbing.

mcpackieh3• 3 years ago

If they allow this to work with English -> English, I bet it will be very useful for lecture videos where the speaker does the "uh uh uh um um umm" thing (since those utterances don't get added to the subtitles.) I'm looking forward to trying this.

Also, I think some people are fixating on "forget subtitles" in the headline. Youtube obviously aren't going to take subtitles away. The article doesn't suggest that people stop using subtitles. It's just a meaningless figure of speech.

Fire-Dragon-DoL• 3 years ago

Subtitles are useful when you have children playing in the living room. Sometimes they will scream and you won't miss anything because of that

snehk• 3 years ago

Anyone outside of English speaking countries should get ready to be detaulted to crappy AI voices instead of the original English without any option to disable it. Happened with subtitles as well. I don't think there's any way to disable subtitles once and for all on all devices. They come back even if you turn them off everywhere. I constantly have to disable them manually. If that crap starts with sound as well where I cannnot set a default, I'll just use a third party way to watch the videos.

Forget subtitles: YouTube now dubs videos with AI-generated voices

Related Stories