Open-source tool translates and dubs videos into other languages using AI

https://github.com/jianchang512/pyvideotrans

by oldcai • 2 years ago

182 116 2 years ago

Unlike other methods of automation, AI is replacing human beings too fast. And before you say, "new jobs will be created" -- look at history. After the computer, new jobs have been created, but what kind of jobs? Every year, we are becoming more entwined in wage slavery as the wealth accumulates at the top and jobs become more meaningless.

So, no, new jobs will not be created, except the kind of jobs that crush the human spirit into oblivion so that the rich tech oligarchs can play God.

nickthegreek• 2 years ago

AI is replacing human beings too fast.

citation needed.

RIMR• 2 years ago

This is a subjective statement. If you disagree with it, counter it. Asking for a citation doesn't make sense here.

homarp2• 2 years ago

can't I with that creates video that have the world as potential users?

and the video can be the products, or tutoriala for another products. This allows me to do more, not less.

homarp2• 2 years ago

Kiro3• 2 years ago

Strange submission to post this comment on. It's not like translating and dubbing videos is the highest form of labor.

wongarsu• 2 years ago

It's not like it's mindless drudgery either. Losing as little as possible in translation while fitting within the constraints of subtitles (people can only read so fast) can be very challenging. And decent dubbing tries to approximately match the mouth movements, which puts further constraints on the translation and often requires creative text changes.

There is a point about industry size (not that many countries dub all their movies), but it is one of the intellectually more challenging professions.

insane_dreamer2• 2 years ago

It actually requires a fair bit of skill to do it correctly and accurately

surfingdino• 2 years ago

There is finesse to it. You are translating from one culture to another.

notkaiho• 2 years ago

Have you tried? It's a genuine "skilled labour" job, both the translation and the voice acting.

pizza2345• 2 years ago

Voice acting (dubbing) is acting, which is a "high form of labor". In some languages/countries, most of the dubbing is performed by a few extremely good voice actors.

codedokode2• 2 years ago

It is a form of art. I hate watching dubbed videos, because the voice actors who translate it almost always are worse than original actors.

insane_dreamer2• 2 years ago

The problem is that we are hurtling towards the unknown without a plan, driven by the “need” to make higher returns for shareholders and to capture the new market. It may be that some new types of employment magically appear that soak up the jobs lost, but you can be sure there is no one working on solving that problem since the goal is to eliminate labor not create it.

numpad02• 2 years ago

I don't think generative AI is replacing humans at all. It's like how SQL replaced software engineers, with added bonus of copyright doubts gatekeeping common folks from exploiting it. It's obviously killing open Internet fast and encouraging power concentration too. It's worst of couples of worlds.

IndySun2• 2 years ago

...new jobs will not be created, except the kind of jobs that crush the human spirit into oblivion...

AI certainly means everyone will be able to create 'art' and as a result we'll have more art than we know what to do with, music and images are already confetti, soon so will full length 'films/movies'. That leaves anyone who can actually sing, paint, play, dance, in prime position to take up those mantles.

amonith• 2 years ago

What do you mean by the last sentence? Isn't it the opposite? People who could do art stuff won't be able to sustain themselves using those skills anymore because those will be devalued a lot.

pizza2345• 2 years ago

This is a fallacy called "Lump of labour fallacy" (see https://en.wikipedia.org/wiki/Lump_of_labour_fallacy).

ZoomerCretin2• 2 years ago

Calling something a fallacy does not mean it truly is one. I have not seen strong evidence that there is not a "lump of labor" in extremely high-rent countries like the US.

For a sufficiently small period of time, there absolutely is a "lump of labor". If I were to go into a county with robots and rent them out to employers for 1/4 of the wage of their current employees, and they all fired their employees and accepted my robots, do you really believe that all of ex-employees would be able to find work again within 6 months? Or even 70% of them? What about their new wages? Do you think these new jobs would pay as well? I have a hard time believing that to be the case.

vouaobrasil6• 2 years ago

If it is such a fallacy, then why are there fewer and fewer options for those who don't want to be entrenched in technology?

lmm• 2 years ago

After the computer, new jobs have been created, but what kind of jobs? Every year, we are becoming more entwined in wage slavery as the wealth accumulates at the top and jobs become more meaningless.

What are you talking about? Many of us have tech jobs with much more comfort, creativity and autonomy than the jobs they displaced, and computerisation has made it much more practical for those who dare to strike out their own rather than needing wealthy family or friends before you can even begin to think of starting a business.

CaptainFever3• 2 years ago

I know, right. Jobs created by computers have by far been the best paying, and most comfortable jobs that exist in human history. We're privileged.

SoftTalker2• 2 years ago

After the computer, new jobs have been created, but what kind of jobs?

Um, jobs where someone under age 30 can be earning hundreds of thousands of dollars a year programming them?

vouaobrasil6• 2 years ago

Um, jobs where someone under age 30 can be earning hundreds of thousands of dollars a year programming them?

Not a very fulfilling job in my opinion, except for the money.

hatenberg• 2 years ago

Transformer based AI is basically printing machines for the knowledge economy from existing labor patterns.

Nobody currently can say which patterns it cannot extract, hence "we always figured out new jobs" is ... challenged

nsonha3• 2 years ago

so maybe don't cling on to "the job" so much and hoping it somehow can fulfil your life. If the job can be automated by a machine then isn't it already meaningless and mundance and bore you to death anyway?

I agree with the point about "wealth accumulates at the top" though. Maybe Karl Marx was right about a thing or 2. Maybe the distribution of wealth should not fall into the hand of non-elected corporations. Whatever it is, it should be determined by a democratic process and not some "market mechanism" that is actually just arbitrary algorithms optimized for metrics no actual human cares about.

fleischhauf• 2 years ago

the reaction is wrong. it shouldnt be "oh no, jobs are being removed" but "nice, less work more automatization, let's make sure we all benefit through less work and not only the rich with more profits"

pjc505• 2 years ago

let's make sure we all benefit through less work and not only the rich with more profits

It's fairly clear by now that that is not what happens and that the real AI risk is not the "grey goo" one of everything being converted to nanomush, but the age-old one of landlordism soaking up all the returns to economic activity.

cubbic• 2 years ago

Oh I made something similar but for Netflix!

https://github.com/cubbK/dubbing_ai_netflix_client

I want to learn swedish and because there are so few dubbed movies in Swedish I take the subtitles(Netflix is good at having subtitles in different languages) and text-to-speech it :)

sam_perez• 2 years ago

How would you rate the quality of the dubs?

Do you think it's usable for learning? Seems like you could end up with some quirky learnings.

woodson2• 2 years ago

Not OP, but I agree that this could lead to questionable learning outcomes, especially since Whisper isn’t that good for low-resource languages. It’s probably fine for languages like English/Spanish/Mandarin, though.

ainonsense44• 2 years ago

Wenn ist das nun Stück gitt und Schlottermeyer.

sunnybeetroot• 2 years ago

This is fantastic! Any direction on if I wanted to change the language to something else?

birktj• 2 years ago

Why not simply watch native Swedish content instead? There should be quite a lot globally available for free from SVT.

carlosjobim3• 2 years ago

You can only watch Nordic government channels within the Nordic countries. Unless you get a VPN for it, I guess.

ranger_danger2• 2 years ago

An English README would have been nice

pcarion• 2 years ago

there is one:

https://github.com/jianchang512/pyvideotrans/blob/main/docs/...

microflash• 2 years ago

Probably you missed it: https://github.com/jianchang512/pyvideotrans/blob/main/docs/...

ranger_danger2• 2 years ago

Yep I missed the "English" text right on the first line, sorry.

netsharc• 2 years ago

Huh, if it's good enough, movies/TV shows dubbed with AI-clone of the original voice would be great (if we can ignore the ethics of using the actor's voice and the loss of work for the dubbing companies and actors).

For example here's how weird Friends is in German: https://www.youtube.com/watch?v=nCoNSZV--z0 . Or Italian: https://www.youtube.com/watch?v=wO5qTzvyQ1s

Can AI detect the emotional tone of sentences yet, and recreate it in the target language?

programjames• 2 years ago

Yes, pop in a normalizing flow that removes tone and recreates it with a small audio sample as context.

https://arxiv.org/abs/2312.01479

woodson2• 2 years ago

What I found is that, for cross-language use-cases, this often just applies the intonation of the “context” sample to the created sample, which, if they are from different languages, usually gives the wrong result (in the sense that it sounds off).

devindotcom• 2 years ago

they're workin on it

yieldcrv• 2 years ago

Yes, have you used OpenAI’s voice model? It uses and reacts to tones

My favorite conversation has been getting it to tell me about marshmallow vs marshmellow spelling and pronunciation, it became very strict but patient with me

It can reply in other languages too, but I cant detect dialect as well to say

_moof• 2 years ago

(if we can ignore the ethics of using the actor's voice and the loss of work for the dubbing companies and actors)

This is a shocking parenthetical.

roenxi• 2 years ago

Optimistically, maybe we can replace the original actor with a synthetic voice as well. Then the issue is largely moot.

skeledrew6• 2 years ago

In my experience, human dubbing never captures the original tone anyway. Probably never can unless it's done by people fluent in both source and target languages and're also good at voice acting. And so I have a huge preference for subs so I can appreciate the nuance in the original voices.

tomw1808• 2 years ago

Sometimes, very rare, the dub is actually better (yes, subjective, but still) than the original. E.g. I do find Fear and Loathing in Las Vegas in German absolutely hilarious, start to finish. Then I watched the english version - was I surprised how serious this is in original voice. It's an entirely different movie in its original tone.

carlosjobim3• 2 years ago

When was the last time you experienced human dubbing? They do it with amazingly high quality in some languages today. I actually feel sorry for the dubbers who now have to be actors as well, making every grunt, sigh and laugh of the people they are dubbing.

They are even dubbing reality survival shows, so somebody has to sit in a studio and groan as if they are climbing a slippery hill in Alaska.

giorgiobalduino2• 2 years ago

For example here's how weird Friends is in German: https://www.youtube.com/watch?v=nCoNSZV--z0 . Or Italian: https://www.youtube.com/watch?v=wO5qTzvyQ1s

Can't talk for the German dubbing, but the Italian version sounds natural to us Italians while the original, English version, is hard to relate to and create a bond with. The dubbing makes it "close" to home if that makes sense. You might feel it's weird because you've grown accustomed to watching the original version while also immersed in everything that sitcom portrays.

randomgiy3142• 2 years ago

Because translations are copyrighted so it is complex to get legal rights for them.

CyberDildonics2• 2 years ago

Is this using an open source text to speech model or is it going out to some other internet service?

tourmalinetaco• 2 years ago

It can use Whisper which is open source, although optionally it can also use GoogleSpeech.

CyberDildonics2• 2 years ago

I took a look and that seems to be speech to text / speech recognition but not text to speech.

lossolo• 2 years ago

Based on english doc it seems it's not dubbing but voice over.

LeoPanthera• 2 years ago

For years I've wanted this for live TV. Even just subtitles would be amazing. I've always wanted to be able to watch news TV from other countries.

alphabetatheta• 2 years ago

Some of my friends have been using: https://app.vivalabs.ai/ as a more managed/paid service recently

cyberax• 2 years ago

I would pay a lot for a tool that removes the freaking laugh track from videos.

I just physically can't watch them. I wanted to watch the Blackadder series, but I couldn't even get through one episode.

deckar01• 2 years ago

https://github.com/facebookresearch/demucs

If you can train an instrument model on laugh tracks Demucs should do that.

mrtksn• 2 years ago

Back in high school, when I got my first PC a plumber came over to fix some stuff and when he saw the computer he got excited and asked some questions and one of the questions was “how do you translate the VCD with this, I have a movie to watch but hate subtitles”.

I was like “silly dude doesn’t know how computers work” but maybe I was the silly one who can’t dare to imagine how something like that can work.

codedokode2• 2 years ago

As I understand, it first extracts text from original video into subtitles, translates them using external LLM, and then converts text to speech. All of this is done using thrid-party solutions, and the project seems to be just a GUI app that allows to integrate them.

You obviously cannot use this to translate songs or movies because this method loses important information like voice, intonation, etc.

So it is still better to use subtitles.

ewuhic• 2 years ago

This one does dubbing, but is there an equivalent tool for subs?

fbnt• 2 years ago

Not open source, but https://fluen.ai does a good job at translating subs, while using your own or standards style guides for the target language - ie. reading speed, max characters, "chunking" sentences where it grammatically makes sense, re-adapting them etc.

cyanydeez• 2 years ago

So uh, what's with AI products throwing out the gold standard in testing these claims.

paulkon• 2 years ago

Is there an open source speech-to-speech model which retains intonation, cadence and delivery?

gagabity• 2 years ago

Cool what languages can it do?

Yandex browser does the most impressive version of this and for free but only to Russian I believe, its quite amazing it does appropriate different voices and follows the correct intonation for everyone, just takes a few seconds for a YT video.

nsonha3• 2 years ago

This could be useful in combating fake news. In many videos especially in political news, foreign languages are dub over with sometimes nuanced translation that can skew audiences to (mis)understand the content in certain ways.

kevingadd3• 2 years ago

A translation lacking nuance/precision (due to being the work of machine learning) can also cause significant misunderstandings, though. I'm not sure you win or lose in that regard by switching from humans to machines.

nsonha3• 2 years ago

the errors in ML translation would be systemic and less likely to be of human arbitration, which is less dangerous. Human translation could be deliberately misleading at a specific phrase in a specific interview.

pjc505• 2 years ago

The AI is quite capable of inserting its own translation errors.

skummetmaelk2• 2 years ago

Localization and dubbing is a sad endeavour. By trying to accomodate everyone's individual preference for information transmission we accomplish nothing more than reducing our ability to understand each other in the long run.

Having a Babelfish is all well and good. Until it stops working, and you realise no one can understand each other any more.

Ironically localization is often pushed by well meaning Americans who only speak one language. "Oh, you're in a French speaking region. You MUST want French language. Let me force it down your throat while I prance around virtue signalling about how inclusive we are"

written-beyond• 2 years ago

Interesting take, never really looked at it that way.

kevingadd3• 2 years ago

Are you saying it's a bad thing if the creator of a work decides they want it localized or dubbed into other languages? I don't understand why you want to take that choice out of their hands.

Which languages someone speaks isn't simply a matter of "individual preference". Learning a new language takes a lot of time and energy, and people only have the time to learn a handful of languages in most cases unless they can make a career out of linguistics.

i.e. I know a sprinkling of words in various languages, and I've started learning Japanese, but I simply don't have the time to also learn Mandarin, Korean, Cantonese, etc. So I appreciate it when authors of works in those languages offer localizations into a language I can speak, or when third parties spend their time translating stuff for free to make it available to a wider audience.

What's the advantage of closing knowledge and communication off from a wider audience?

Maybe I'm misunderstanding and you're just angry about Google Translate/DeepL etc (which I have a strong distaste for since they're Fake)

skeledrew6• 2 years ago

Nah the localization itself is fine. Where it becomes problematic is when there's no opt-out at the app level. Or perhaps it should even be opt-in.

pjc505• 2 years ago

This is a terrible take, and you should have at least included the "forced" dub disclaimer from your comment below. Without at least one of sub/dub, foreign (relative to your current location!) language content basically doesn't get consumed at all except by a very small minority of people who are very keen on the content anyway - or are speakers of the language anyway.

Now, as veterans of anime forum wars will know, subtitling is nearly always better than dubbing, and I hope this tech is capable of that as well. Most media systems let you put a whole load of subtitle tracks on and then pick one.

There's far, far too much content out there for more than a fraction of it to be ever professionally translated. While we should expect human translation review and a spot of localization for officially released works, most of the internet is just free content being given away for very little return. And that's where automatic translation is going to shine: release the non-English meme champions! Let us have a look in Bilibili!

skummetmaelk2• 2 years ago

A big reason for the Dutch and Nordic populations being better at English than the Germans, Italians, and Spaniards, is that they were simply forced to consume media in English. They had no other option. Turns out to be an advantage.

underdeserver• 2 years ago

https://deepdub.ai/ have been doing this commercially for a while.

hulitu• 2 years ago

Open-source tool translates and dubs videos into other languages using AI

Is there any assessment about how good the translation is ?

Open-source tool translates and dubs videos into other languages using AI

Related Stories