Serverless Video Transcription inspired by Cyberpunk 2077

https://github.com/elanmart/cbp-translate

by pierremenard • 4 years ago

774 80 4 years ago

chinchilla2020• 4 years ago

This is impressive and so is the writeup. How did you create the diagrams in the readme? I really like that visual format for diagrams.

super_linear• 4 years ago

For the hand-drawn style ones, I think it's https://excalidraw.com/

navanchauhan• 4 years ago

Pretty darn cool!

I almost want to try and implement this in OpenCVjs and see if it is possible to build a browser extension to do this in the browser itself

ArekDymalski• 4 years ago

Now this is both cool piece of tech and practically useful. Very nice!

vorpalhex• 4 years ago

This would be a killer app for a google glass type device. A near realtime closed captioning device with translation.

fennecfoxy3• 4 years ago

It made me so sad to see how Glass was neglected/spurned by the public. Especially because it had a camera.

I don't think people understand yet that this stuff is inevitable; whether it takes 20 years or 50 years eventually we (or the next generation) will _all_ be wearing these. So why not do it now?

For all the bad it could do, it could do so much more good/be useful. & if people are worried about the cameras on them; they should be worried about the camera that they _can't_ see, like the OmniVision OV6948.

totetsu2• 4 years ago

Matching faces to voices relies on simple co-occurence heuristic, and will not work in certain scenarios (e.g. if the whole conversation between two people is recorded from a single angle)

This seems like the really hard part. maybe if there was a way to find the time lips moves for a face. or guess gender and age of both face and voice.. or If the audio is a stereo mix, using relative position

photoGrant• 4 years ago

There's this that can differentiate speakers

https://github.com/CorentinJ/Real-Time-Voice-Cloning

oslem• 4 years ago

I think you are referring to GE2E [1], which is the speaker identification model used in that project.

[1] https://arxiv.org/abs/1710.10467

totetsu2• 4 years ago

But matching a speaker to a face is a different problem differentiating between speech in a recording

holler• 4 years ago

This is awesome, impressed you threw this together over a weekend!

What did you use to make that entity diagram?

edit: answered below

abidlabs• 4 years ago

Neat use of Gradio & Modal

ncr100• 4 years ago

FFMPEG is in there too, of course! Such a powerful tool.

https://youtu.be/9kaIXkImCAM humor

867-5309• 4 years ago

that was the first video I have watched start to finish without skipping in years. brilliant, thanks

daedalus2027• 4 years ago

It seems there is a niche market in this...

jacquesm2• 4 years ago

More than just a niche. Imagine a pair of glasses a tourist would wear that would translate in real time what that waiter just said.

Mizza• 4 years ago

I'm really hoping somebody comes along and puts this in a format that I can attach to my head inconspicuously.

shadowgovt2• 4 years ago

Google Glass has already been rejected by the public; I imagine anything more inconspicuous would arouse even deeper suspicion.

pyinstallwoes2• 4 years ago

It’s inevitable though. Is it not? Contact lenses, eye, and so on.

colordrops• 4 years ago

It was rejected because it didn't do anything useful. Certainly there are many really useful things we could do now that were not possible when it was first released, no?

ajuc• 4 years ago

It was ahead of its times (and wasn't that useful), in 10 years it will be perfectly acceptable.

fennecfoxy3• 4 years ago

What's funny is that the opposite is true. People will cry about the big obvious camera on the glass, but won't cry about a pair of thick framed but normal looking glasses because they have no idea that it has a camera on it.

As mentioned in another post chuck a OmniVision OV6948 or similar in the glasses and nobody will bat an eye because they simply won't know.

addandsubtract• 4 years ago

There's this[0], using NREAL Air glasses.

[0] https://www.youtube.com/watch?v=LauvOTnZMZg

jacquesm2• 4 years ago

Mindblowing that this was so easy to build.

Coming soon to a phone near you, and in realtime.

piyh• 4 years ago

People complain that the magic of programming has been lost because all we do is stitch together APIs. They're oversimplying the work done, those people are cynical, this is amazing.

coding123• 4 years ago

I think part of the magic is still lost, as we used to be very open source and free centric. These days it's all based on paid API calls.

ccn0p• 4 years ago

agree with parent re: stitching together -- but also, someone's gotta program the API!

johnathandos• 4 years ago

Someone is out there building and maintaining the APIs

fragmede2• 4 years ago

The magic is still there! In this case, the model for OpenAI's Whisper, which is arguably doing the bulk of the work here, is Open Source (under the MIT licence), and freely available for download at https://github.com/openai/whisper. You can run it wherever you want, though something with a GPU will let you do 5x realtime (or better!) transcription.

beecafe• 4 years ago

those APIs would be worthless if nobody wanted to stitch them together

s3000• 4 years ago

Compare it to cars. New cars are more complex but collectors pay the most for the old, simple cars with a bonus for beauty and speed.

madaxe_again• 4 years ago

On your old car, you can just take it apart and see how it works. Don’t have the part? Improvise. It’s heavily skill-based, and a little knowledge will take you a long way. As long as you know your fundamental toolchains, you can make any part, do any job.

On your new car, it’s a series of black boxes, cables, and tubes - you don’t know and don’t care what happens inside the boxes, just what goes in, and what comes out. It’s heavily knowledge based - you need to know an awful lot about the boxes and how to work with them - but nobody expects you to machine your own parts. If one isn’t made that suits your purposes, you have to go to an old car guy.

The skillsets are similar but different. To be clear, I really am talking about cars and mechanics here - you won’t find many who can fix up both a 1960 and a 2021 motor. You’ll find plenty who can do one or the other, however.

And yeah - the same applies to developers.

danso• 4 years ago

Clever hack/solution, and the thorough documentation is appreciated!

MetallicCloud• 4 years ago

Very cool. Zack Friedman made something similar to wear on a hoody. It wasn't perfect, but it worked. https://youtu.be/mTK8dIBJIqg

bee_rider3• 4 years ago

How easy is it to translate word by word like that? I was under the impression that generally it is hard because different languages have different word orders. Is it not necessary to have the whole sentence before starting? (Or maybe Polish is just conveniently similar to English in word order?)

ehnto2• 4 years ago

Yes it's not the best way to translate, and for some languages it won't make much sense since the word order is different, as in English to Japanese translation.

To say nothing of all the myriad nuances of language like context changing word meaning and colloquial meanings of words.

In the game Cyberpunk, when doing the inline translations between english and japanese it is actually swapping fully translated sentences out, so not word for word, the effect just looks similar.

shadowgovt2• 4 years ago

I really enjoy that aspect of the visuals because it's like a tiny reflection of the kernel of cyberpunk (the genre, not the game).

Here you have a technology that would work in a way totally different to the way people assume... But someone put the effort into making it look like it does something else because the skeuomorphic expectations of the human makes that inaccurate representation feel more correct.

That gap between the way tech works and the way our perceptions and psychology drive us to assume it works is very real, and it underpins piles of tech we use everyday, from the icons for making calls and hanging up to whether browsers blank the screen when navigating to another site.

pyinstallwoes2• 4 years ago

Just render reality at an offset to compensate for the latency. Hehe.

Dylan16807• 4 years ago

Where does it appear to be going word by word to you?

It's sweeping each sentence at a fixed number of milliseconds per character, set so that both languages finish simultaneously. Look at when "Donald Tusk" appears during the first line. Much sooner in English than in Polish, and both are happening before he actually says the name.

The example from the game is doing entire sentences too.

bee_rider3• 4 years ago

I got caught up in the sci-fi of it all and assumed for some reason that the intended application was a sort of “live” translation. That is, not looking ahead at the whole video. Which of course doesn’t really make sense.

NetOpWibby• 4 years ago

This is incredible

warangal• 4 years ago

Videos encode so much information, and it looks pretty cool when such projects extract higher level information to play with. And recent models like whisper and clip are amazing to help make sense of that information even for personal projects.

We are also trying to do something similar[0], but still a lot of work remaining. Idea is to allow real time processing of any video using just a CPU. [0] https://www.youtube.com/watch?v=E7UPj9blnWc

unraveller• 4 years ago

Nice demo, any chance this tech gets ported to an android library soon? Real-time face detection via CPU would be way better than constantly uploading your video feed to aws kinesis for ML processing.

sj88224• 4 years ago

This is utterly incredible. I have so many ideas after reading way too much scifi and watching Ridley Scott’s scifi series, Raised By Wolves (what if we could create a benevolent, kind and caring ai to help humans grow and navigate into this world, like Father in the series)?

I want to jump in, badly. How would one go about picking up the skills needed to create stuff like this? As pragmatically, concretely and efficiently as possible without getting sidetracked in overly theoretical distractions?

I’m a fullstack engineer and have an MS in CS and pretty good math chops, but I sadly only took 1 machine learning course in all of my formal education.

How do I get into this (gpt-3, chatgpt are also on my mind)? Please, any books, moocs, etc

DerSaidin• 4 years ago

I have so many ideas
I want to jump in, badly. How would one go about picking up the skills needed to create stuff like this?

Have you tried to make any of your ideas? If not, pick one and try to make it.

sj88224• 4 years ago

This is a great approach.

I want to build a benevolent, kind, caring and nurturing AI that one can go to for learning, understanding and support in navigating this world and being human.

I want to dump key knowledge into it that I have been fortunate enough to discern during my lifetime and knowledge from others that I regard highly. Kind of like what characterai is doing, but highly fined tuned and customized.

What skills do I need to do this? What tech do I need to learn? What books, moocs, etc. or even just words to look into? Just looking to be pointed in the right direction, I’ll take care of the rest.

Definitely not trying to reinvent gpt3 or anything like that, but looking to leverage these tools and customize and fine tune them highly

jamesblonde4• 4 years ago

This is a great example of serverless machine learning with modal.

ReFruity• 4 years ago

Nice! Also would be cool to see the text positioned above the face and translated just like in the cyberpunk gif.

flemhans• 4 years ago

It's actually incredible doing all this work and resist implementing the last few lines of code that would do that.

jamesblonde4• 4 years ago

I developed an online course on serverless machine learning, where you can learn some of the principles of refactoring ML systems into separate feature/training/inference pipelines: https://github.com/featurestoreorg/serverless-ml-course

Some of the students have built similar systems, for example chaining Whisper and ChatGPT or translation or sentiment analysis of transcribed text, such as here (transcribe Swedish and tell me the sentiment of the text): https://huggingface.co/spaces/Chrysoula/voice_to_text_swedis...

SiempreViernes• 4 years ago

Why is this type of cloud based computing called serverless?

To me "putting all the work on remote machines you don't manage" seems pretty cloudy, and I naively expected "serverless" to mean something like runs locally, not "uses a shit town of someone else's servers".

jamesblonde4• 4 years ago

Great question. I think of it as follows - if I can just write some operational Python code and "deploy" it, without needing to install/upgrade/maintain systems software (k8s/DB/feature-store/etc), then I have built a serverless system. What's cool now is lots of these services have great free tier and you can run serverless ML systems without paying anything.

jwr• 4 years ago

Good question. For me it's in the same category as "hoverboards" that don't actually hover.

rospaya• 4 years ago

A good analogy I heard is that serverless uses servers as much as wireless uses wires. It does, but it's not part of your responsibility.

pyfan878• 4 years ago

Thanks for sharing! will check it out later, but this looks great.

fareesh2• 4 years ago

Does anyone have a link to something similar for real-time captions (audio only - (.g. Google Meet, Twitter Spaces)? Or is that typically done on the device itself

input_sh• 4 years ago

If I understood you correctly, there's a Chrome accessibility feature that you should check out: https://support.google.com/chrome/answer/10538231

Think of it as YouTube's auto-generated captions, but on a browser level.

fareesh2• 4 years ago

I was mainly interested in studying implementations of real time transcription

schizo89• 4 years ago

This is cool. I guess it's possible to this all in single multi-task NN

slowmotiony• 4 years ago

Is it possible to have this tool run locally and use it for myself? I don't see any instructions on the Readme, all I see are ways to set it up for development and ways to deploy it to some 3rd party cloud solutions.

EchoReflection• 4 years ago

So I've taken to saving many, many posts from HN with https://web.archive.org/save/ and this is the amusing result I got when trying to save this one: "This host has been already captured 100,091.0 times today. Please try again tomorrow. Please email us at "info@archive.org" if you would like to discuss this more." Haha.

yDogbone• 4 years ago

I would like to point out: As someone who is a newbie coder, I had a ton of fun learning how your code works with the help of ChatGPT. Even learning about unfamiliar topics like serverless apps, nlp transformers, or yaml vs json lol

Serverless Video Transcription inspired by Cyberpunk 2077

Related Stories