Background Features in Google Meet, Powered by Web ML

https://ai.googleblog.com/2020/10/background-features-in-google-meet.html

by Marat_Dukhan • 6 years ago

427 246 6 years ago

ai.googleblog.com

Happy to see ML become mainstream. In the future, I don't think ML will be a separate field of programming. It'll just be "programming," the same way webdev is.

There's a tendency to think of ML as "not programming," or something other than just plain programming. But as the tooling matures, that'll go away.

(Lisp used to be considered "AI programming," till it became useful in many other contexts.)

sltEvas• 6 years ago

ML will become a library. It has about as much to do with programming as a compiler. You don't need to know what it does, you just need to know how to make it do things. The problem with ML currently is that nobody really knows how to do things and that you have a million parameters that need tuning and most algorithms need continuous improvement and fine tuning to the use case. There is nothing "mainstream" about ML at this point, except that everyone wants to use it.

In maybe a decade, it might be found in standard libraries of programming languages and on top of things like `Math.abs`, we will have `ML.textToSpeech("Hello world")`, or `ML.isCat(image)`, etc. However, the problem I see with that is that no matter how far we wind the clock forward, we will only be able to put the most simplistic use cases into a library. `ML.isCat()` could be one of those, since most humans will be able to image categorization, it stands to reason that you could put this into a library. However, most industry application involved highly customized ML algorithms that are optimized for a very specific use-case. So there will always be a need for a research team in big companies at least. Maybe smaller companies will try to build their stuff by chaining libraries together.

fhaodhdms• 6 years ago

Fwiw macs have had an equivalent functionality for both text to speech and speech to text for at least 17 years to my memory. The quality is poor compared to today's server-driven approaches, of course, but the functionality has been there if you're willing to articulate yourself clearly.

virgilp3• 6 years ago

There's never going to be a `ML.isCat(image)`, just like there isn't a `Math.solveProblem(hypothesis)`. Yes you do have `Math.abs` and you're going to have stuff `model.fit()` and `layers.dense()` - but something like `ML.isCat` is too specific to be used in a library

m00x• 6 years ago

AI is learning existing patterns from input/outputs. Programming is setting up patterns to turn your inputs into desired outputs. Most often it's just plumbing data around with some transformations.

What you're talking about is using AI as programming tools. It's still programming, but using pre-trained models as part of the plumbing.

jtokoph• 6 years ago

The example video clips in the post look nothing like me and my team's view when using the new feature. Most of the time half of our hair gets blurred or replaced and hand gestures will cause either our hands or head to disappear.

samtheprogram2• 6 years ago

I can vouch for this. I haven’t really needed the background blur feature personally, but I’ve tried it and both myself, colleagues, and friends — pretty much everyone I’ve talked to that has used it — loathe Google Meet’s background blur, and prefer Zoom’s by far.

In my experience, it doesn’t completely cover the background most of the time, and if you move at all, as you point out, it can’t keep up.

Kind of funny to see Google engineering blogging about it when it feels extremely half baked.

This makes me sad, because in all other areas, I think Meet excels well beyond the competition.

EDIT: removed my general sentiment on Google

judge2020• 6 years ago

"Turns out our training set was comprised entirely of backdrops from Google HQ. Sorry everyone else!"

_pastel• 6 years ago

"Half baked" misrepresents the difficulty this task. Yes, Zoom does it better, but it's _still_ an excellent and interesting engineering accomplishment.

I've always wondered what proportion of modern real-time video effects rely on ML vs. classical image processing; this not only answers that question, but provides details down to the level of model architecture and the final latency and IOU benchmarks.

Of course I'd be more interested to read how Zoom manages to do even better, but I'm not holding my breath for them to publish those details.

madars• 6 years ago

At least for background blur the latency there is enough to make it almost unusable: easily over 100ms. This is with latest stable Chrome on a relatively recent Ryzen/Nvidia system. Maybe background replacement will do better once it rolls down to regular Google Meet (too lazy to log into my Google <del>Apps</del> <del>Suite</del> Workspace) :-) However, everything else about Google Meet is great and I wish I could make all my Zoom friends switch.

riffraff• 6 years ago

However, everything else about Google Meet is great and I wish I could make all my Zoom friends switch.

is it _better_ than zoom tho? I my experience, I don't see much of an improvement worth switching.

chrischen• 6 years ago

I'm wondering why they didn't just use standard CV techniques like background subtraction? Does their technique work with a dynamic background as well?

objclxt• 6 years ago

I’ve done some work in this space - subtraction doesn’t perform well when other motion is present, whereas if you use pose / body detection you can ignore other bodies in view (i.e, the toddler running across the room).

gundmc2• 6 years ago

It seems to have gotten a little better recently, but my experience matches yours. It really struggles when I wear over-ear headphones - they sort of phase in and out of existence.

The other thing I've noticed is the background blur absolutely annihilates my CPU. To the point where I would rather just turn off my camera if I don't want my background visible.

cameldrv• 6 years ago

They have their example video clips, but they also provide data. They say that in their better model, They get an IoU of 93.8% This means 6.2% of pixels are misclassified. Either it's your hair getting cut off or the background is leaking through. 6.2% of an image is a fair bit considering your head is probably 30% of the frame.

chdjakdkgb• 6 years ago

Wtf happened to hangouts? How many video products does google have?

samtheprogram2• 6 years ago

Meet is business oriented and offers features that Hangouts does not, e.g. dialing in via phone. It also requires a G Suite account (or did before COVID, IIRC).

Polylactic_acid• 6 years ago

Didn't hangouts have a business version as well?

senectus1• 6 years ago

Hangouts is EOL :-(

We're being herded into the new more useless products.

toper-centage2• 6 years ago

Hangouts was always terrible and bloated. I miss Google Talk. Instead, we get something more terrible and bloated than Hangouts.

hoveringhen• 6 years ago

Hangouts is being replaced by Chat and Meet.

easytiger• 6 years ago

And Duo.

vinhboy• 6 years ago

Blur is awesome. Way better, less distracting, than backgrounds. Everyone who try it uses it permanently because it works so well.

I also think it makes the subject look better for some reason.

probably_wrong• 6 years ago

Here's a tip: take a picture of your real, actual background from the POV of your webcam, and set that as your meeting background.

Advantages: it looks natural, it covers whatever is going on behind you (in case you are not alone and people walks by, or if your living room is messy), and it blends better than fake backgrounds (because it's the same image behind it). I have a picture of my office that I use both at home and at my real office, and most people can't tell. And since I took the picture with my phone which has better resolution, my video feed looks better for cheap.

dharma12• 6 years ago

Sony released some software a couple of months ago that lets you use most of their DSLRs as webcams with USB. My goodness, paired with a fast lens, what a difference to my MacBook webcam, even with these ml blurred backgrounds!

It's only 720p and around 15fps but real shallow dof, very little sensor noise, autofocus works. Well worth trying if you have a Sony camera from the last few years.

Sensor size and good optics still wins. Having said that,the effort and detail gone into this feature is very impressive, enjoyed the blog post. Also webassembly SIMD looks super cool, looking forward to a new class of webapps using wasm.

kenhwang2• 6 years ago

Canon did too! Definitely a huge upgrade over a typical webcam.

I'm using my old T1i which can be had for less than $50 these days, plus you can pick up a 18-55mm kit lens for like $20 and the video quality blows away any webcam, especially for the same price. Also recommend battery->power adapter.

silly-silly• 6 years ago

What is the canon software called ?

callmeal• 6 years ago

The T1i is not listed on their compatibility page. Which version of the utility did you get for it?

jmarcher• 6 years ago

Canon and Nikon do too. In practice, the quality bump is nice, but we are still talking of a fairly low res/bit rate when it gets through Zoom so the end result is fairly underwhelming. As far as what the other people see on their wnd.

dharma12• 6 years ago

Yeah.. both Zoom and Google Meet have >720p video but the bitrate especially on Zoom is a travesty, 600kbps/1.2mbps stream with all the different resolutions in the same stream.

The codec situation with h264/HEVC/vp9/AV1 software/hardware encoding is a mess. Hopefully we'll get wide hardware support for AV1, although it might take a while.

piquadrat2• 6 years ago

I recently tried to get a setup similar to this with a Fujifilm X-T20 I had lying around, remembering that Fujifilm announced similar software. Alas, that software only works with their higher end models.

I ended up getting a $10 HDMI USB capture stick from Aliexpress. I get a perfect 1080p/60fps signal, and at least on Linux it worked out of the box with Zoom.

The only problem now is that most of my meetings start with "wow, why do you look like you're on TV?"

notimetorelax• 6 years ago

This is interesting, could you link to the device you used? Do you know if it would work on Chrome OS?

brainless• 6 years ago

I use my Android (Redmi Note 8 Pro) primary cam (720p I think) using Droidcam and it works like a charm on Linux.

I also tried gPhoto2/ffmpeg and virtual cam driver with Nikon D5200 (USB) on Linux but I prefer the Redmi since I do not have a decent low light lens for my DSLR.

Quarrel• 6 years ago

Woot. Thanks for pointing this out - I looked for a solution a while back and it seemed like I had to get a separate capture card to connect my Sony DSLR. Will go check this out now.

(I ended up having to buy a little logitech webcam, which has been fine, but being able to pick my lens etc is awesome!)

mdoms• 6 years ago

Honestly it's the worst implementation I've seen of this technology yet. Just absolute and total garbage.

yjftsjthsd-h• 6 years ago

Why? What makes this one worse?

neilpanchal2• 6 years ago

Aside: Imagine you’re driving down the road and you need to make a right turn. Well, for some reason the steering wheel is stowed away and disappeared! You need to hover your hand around the center console in a specific area to be able to expose it. Out comes the steering wheel and now you can make a right turn.

Google UX/UI team: Please fucking make the mute/unmute button visible at all times.

askvictor• 6 years ago

More important than the button is the status indicator - I need to know if the call is muted or not. Even better, promote it to an OS-level icon/badge/overlay. If my mic is actively in use, please make it blindingly obvious.

captn3m0• 6 years ago

The color indicator for mic on/off is terribly for Meet.

neilpanchal2• 6 years ago

It's a toggle switch. The action and its current status is combined.

odiroot• 6 years ago

I already have it on my Thinkpad with KDE Plasma.

Physical button to block the microphone, LED on the button itself and a tray icon with the microphone status displayed.

dawnerd• 6 years ago

Also stop telling me my camera is disabled. Like, I know it is, I disabled it for a reason.

baskire• 6 years ago

COVID taught me that video conferencing is so much more demanding than phone conference bridges.

The best conferencing solutions I’ve used to shame those not using video

sundvor5• 6 years ago

And stop telling me I'm using an input different than the output. I have a condenser microphone on an audio interface with RTX Voice; no, it's not going to transmit an echo.

helpfulgoogler• 6 years ago

Not what you asked for, but you can mute/unmute quickly with the keyboard shortcut: ⌘/Ctrl + d

TheChaplain• 6 years ago

The blessed workaround as we say in software engineering, which allows us to move the real issue waay down the backlog indefinitely. :)

tobr2• 6 years ago

Ah, yes, of course. D as in mute.

ehsankia3• 6 years ago

And ctrl+e for video

raverbashing• 6 years ago

Zoom at least uses Cmd+Shift-A for Audio and V for Video

But as the recent Google Icon kerfuffle, UI/UX is not their strength (probably because of opinionated technical people that think you need to A/B shades of blue)

systemvoltage• 6 years ago

Isn't this sort of a Fizz Buzz for a UX/UI design professional? I don't mean to demean anyone, but I see this sort of a thing literally everywhere. Hiding important and absolutely crucial information (that can make or break your product) in the name of minimalism. Coming out of a company that has one of the highest hiring bars for software engineering, and yet, their products have such an awful UX/UI. This isn't an exception, it is a pattern.

atoav2• 6 years ago

I worked as a freelance graphic artist/web designer once and while I wasn't bad at the job, I really hated one aspect of it: Everybody and their kid thought they knew better than I did. When I said: "Yeah but this should really be visible, because accessibility", they would say: " But it looks better if..."

People in high paid position certainly want "has taste" and "knows what looks good" to be part of their self image. Many fails in design and architecture happen for that reason alone.

I then ended up programming and working in film sound, because very few people in both fields tell you what to do when they have no idea what's going on.

JumpCrisscross• 6 years ago

in the name of minimalism

Ironically forgetting that visual minimalism produced by hiding things isn’t really minimalism.

It would be like me throwing all my things in the garage and advertising my house as Spartan. No, it’s not, it’s a mess. The mess is just hidden until I need to do something.

hoseja• 6 years ago

It's the absolute disdain for the user, the aim for lowest possible common denominator.

mcv• 6 years ago

Exactly. Everybody does this. In anything using video, UI elements apparently need to be hidden as much as possible. In virtual meetings, Youtube, and it's often an option in games.

And sometimes it's great, because you get to focus on the content, and sometimes it's not, because you lose control. It's something that should be optional or configurable. It's great to have shortcuts for the most common commands (like space for pause in youtube), and I guess it would make a lot of sense if video conferencing tools also had such a shortcut for mute/unmute.

But again, give people more control over their UI. There are too many applications that mess this up one way or another.

drevil-v2• 6 years ago

their products have such an awful UX/UI

This is true. I find Android UI so offensive that if I did not have iOS as an alternate I probably would carry a dumb phone and live like a monk. I can’t stand the miles of white space and brightly coloured tiny UI controls.

Evokes such a visceral reaction in me that even I am startled at times haha

tssva2• 6 years ago

"Hiding important and absolutely crucial information"

If we want to give awards for this my vote would go to Apple. I find their products to be horrific when it comes to completely undiscoverable features. iOS is bad on its own but the Apple TV is a total train wreck. I couldn't get rid of that thing with its awful interface and remote fast enough.

leeoniya• 6 years ago

https://en.m.wikipedia.org/wiki/Mystery_meat_navigation

TacoSteemers2• 6 years ago

I have similar problems with Zoom.

The mute/unmute changes position and can be hidden in a top bar that slides out. In some fullscreen situations there is no button to get out of fullscreen. Sometimes double-click works, sometimes it doesn't. Recently I could not even alt-tab away, basically my computer got 'locked' by zoom.

https://tacosteemers.com/articles/2020-10-16-ux-anti-pattern...

jcims4• 6 years ago

I imagine most know this by now but the space bar works as a push to talk button in Zoom (as long as it has focus of course).

I really think there is a market for a physical video conference controller. If I could get a hefty slab of something with quality buttons to enable/disable video, push to talk/mute/unmute, bring to foreground, ‘on air’ light and end call, I’d easily pay $100 for it.

mr_mitm2• 6 years ago

Zoom does this well on the iOS app. They call it "safe driving mode" [1] and half your screen essentially becomes the must/unmute button. You can either tap it or swipe left to unmute.

[1] https://support.zoom.us/hc/en-us/articles/201362973-What-Is-...

aaronharnly2• 6 years ago

Every time I go into Present mode in Zoom, it’s panic-inducing. Where the hell did everyone else go? How do I mute? It’s frustrating.

adrr• 6 years ago

It’s even worse on touch devices. You have to touch the bottom screen to get the controls to appear. Accident touch twice in the wrong location and you can hang up.

Spare_account• 6 years ago

I've often thought that on a touch screen device the OS should ignore touches on buttons/popups that have been on screen for less time than a human could reasonably have observed it and chosen to interact with it. If I touch the screen 0.05 seconds after a button appears, I was probably _not_ aiming for that button.

In fact, now I think about it, this has happened many times over the years with traditional mouse drive interfaces too.

I'm sure some power users would like to shorten the 'reaction time delay' or even remove it entirely so I guess that should be an option as well.

wdr1• 6 years ago

Cmd-D will mute/unmute. I find it much easier than using the mouse.

chedabob2• 6 years ago

I find more often than not I end up bookmarking the page with that shortcut.

vosper• 6 years ago

https://www.meetenhancementsuite.com/

Not that you should have to install an extension to get basic UX

ehsankia3• 6 years ago

To be fair a lot of sites do need it, especially for more power user level UX. See BetterTTV, RES, etc. Sites generally don't target power users, understandably.

chedabob2• 6 years ago

Also don't put that damn bar over the subtitles.

Angostura• 6 years ago

MS Teams has finally changed this on their video calls. Ah the hours I spent telling colleagues 'If you move your mouse around, you should see a black bar appear somewhere near the middle llof the screen'.

gogopuppygogo• 6 years ago

Zoom has the same issue.

I bought an external microphone for my laptop with a hardware mute button.

nurettin• 6 years ago

We are in the era of three seashells. There is no turning back from this. Soon you won't be able to find the power button for anything tech industry related.

teddyh• 6 years ago

“Off switches are illegal.”

vagrantJin• 6 years ago

I looked like an idiot trying to find a power button for a meeting a few weeks back. Literally stood there feeling up the TV for a good few minutes.

I'm a 28 yr old software developer.

rplnt4• 6 years ago

Out comes the steering wheel and now you can make a right turn.

But you will hit a dog probably, because the steering wheel suddenly blocks your view too.

amf124• 6 years ago

When did you recently use Meet? I just used it yesterday with a gaming session with friends and the console for the mute / unmute was visible at all times. I even just tried it right now.

tester342• 6 years ago

The only software that gets vide-co right is probably Discord

I used MS Teams and zoom and both are decent (ms teams works fine for school)

but it's insanely unbelievable that this kind of software lacks of features that gaming communities had probably 20 years ago

PUSH TO TALK is probably one of the most important features of any voice software. The lack of it is big WTF.

It gives you 100% control over when you're talking and you don't have to alt-tab between programs in order to "mute" yourself.

You can bind it to e.g MOUSE3 (scroll-push) and it works fine with other programs, games and stuff. Switching between muted/unmuted is different thing.

From somebody who uses/used ventrilo, mumble, teamspeak and nowadays discord for like last 12 years for hours per day, almost everyday.

Orphis2• 6 years ago

For push to talk to work, you need to have access to keys even when you're not in focus.

That's not something doable today on the web for obvious security reasons, but it's possible for Discord that has a separate app, would be doable for Zoom too I guess.

himinlomax• 6 years ago

While you're at it, always display a vu-meter. It gives feedback on what is transmitted and thus can alert a user whether they are being heard or not. It's the most basic of sound recording tools, and was a standard part of recording equipment for over half a century for good reason.

And if you need minimalism, offer a toggle for that. But I think most people should have it forced on them, would save anyone a lot of trouble -- just think about all the aggregate time lost talking into a muted mike by all users.

sundvor5• 6 years ago

If you're on Windows, "Digital Level Meter" is an absolute gem: https://www.darkwooddesigns.co.uk/pc2/meters.html

I did donate a contribution to say thank you.

tjpnz2• 6 years ago

Speaking of mute/unmute I've not yet found a way to get Google Hangouts (same thing as Meet?) to play nice in situations where simultaneous interpretation is involved. Our company works in Japanese and English and we typically have a second meeting running in parallel for interpretation. This setup almost works, I say almost because I've yet to find a way of muting the audio in one meeting so I can properly listen to the other. I can't leave the first meeting either because often I'll also want to see the presentation slides. Currently I'm working around this by muting my MacBook and joining the second meeting on my phone.

Perhaps I'm missing something obvious (or a Chrome plugin that will allow me to mute based on the page URL rather than site). In the unlikely event that a Googler is reading this I'm not asking for yet another product or complicated new piece of functionality aimed at this specific use case. Just a mute button for audio. Thanks!

lima3• 6 years ago

Google Hangouts (same thing as Meet?)

No, vastly different products. Hangouts is the legacy thing and never worked quite right for me. Meet is much better.

emartinelli• 6 years ago

Right click on the tab -> "Mute site".

It works for me for Chromium on Ubuntu.

eugmill• 6 years ago

cmd+D will mute/unmute (ctrl+D on windows?)

I still can't stand the bottom popping up and down and not being able to tell if I'm muted.

swiley2• 6 years ago

At this point the bad UI in google products feels intentional. You're supposed to feel helpless on computers and just do whatever they want you to do.

sundvor5• 6 years ago

A major motivation why I got a StreamDeck was to be able to put a big fat mute button that "physically" kills the microhone level at the source.

It renders a big cross through the microphone when muted.

Simple, yet insanely effective UI (#).

Best thing ever.

#) Especially when compared to the mess that is Google Meet. My favourite "feature" of theirs is how when someone is presenting, it's impossible to view the presentation as just another stream - no they have to make it dominate everything, meaning it's so hard to see the other team members.

And it can be extremely hard to see who's talking when viewing a lot of cameras at the same time. And for whatever reason the quality turns to a blurry mess a far cry from 720p just way too often. (I have fibre internet).

howlgarnish• 6 years ago

Press Ctrl-D to mute/unmute.

Doesn't excuse the UI, but at least this lets you avoid using it!

nobleach• 6 years ago

Which is so odd as CTRL-D is also the bookmark shortcut in Google Chrome. So, say for example, my team has a goto channel where we have our ad-hoc meetings. It's a pain to bookmark it for later use without jumping through the gui.

three_seagrass• 6 years ago

There are so many unnecessary clicks in the Google Meet U.I.

When I leave a meeting, can you please stop asking me for feedback every time and just take me back to the main meet screen?

It would be so easy just to put that small dialogue box on the main meet screen rather than prompt me to click the button to return.

folmar4• 6 years ago

I guess not a lot of people use the main screen at all. It only hass a minicalendar and you probably have another calendar application anyway.

MarkyC42• 6 years ago

Also: Consider another shortcut for mute/unmute (cmd + d or ctrl + d is the bookmark shortcut in like every browser... maybe this is intentional?)

on_and_off• 6 years ago

unpopular opinion it seems : in personals 1:1 calls, I love seeing the UI disappear and just have a full screen video of my SO/family member.

obilgic• 6 years ago

You all notice that this is a PR piece to get tech people interested in using google meet instead of zoom right.

toper-centage2• 6 years ago

No, because tech people want software that works, has good UX etc. This is a PR piece for people that prefer software that has cutsie little backgrounds.

hota_mazi• 6 years ago

Google finally catching up to where Zoom was two years ago.

Can we get a mute button visible at all times before 2024?

amf124• 6 years ago

Can we get a mute button visible at all times before 2024?

Is it just me or is the button visible at all times? I could see the button visible on the bottom of the screen at all times I used meet during a session with friends. I even tried it right now to make sure.

eCa• 6 years ago

It’s not. Move the pointer outside of the window and the bottom bar should hide.

loosescrews2• 6 years ago

Too bad it doesn't seem to be supported in Firefox.

spurgu3• 6 years ago

Not yet at least.

Jitsi also has background blur but it's only ok-ish on Chrome and unusably slow on Firefox.

nostromo• 6 years ago

I wish my coworkers would stop using background blur.

It sucks and it’s distracting.

Your hair and hands pop in and out of blur. Sometimes part of your face will blur.

I don’t care if your workspace is messy or your kid walks in the room. I do care that we’re all being distracted by your weirdly blurred hair and hands.

janekm2• 6 years ago

Your co-workers have a reasonable expectation of privacy regarding their home life and family members.

Given that many had to start WfH with short notice meaning they couldn't relocate to circumstances enabling a dedicated home office space blurry hair and hands are a very reasonable compromise.

josalhor2• 6 years ago

Your co-workers have a reasonable expectation of privacy regarding their home life and family members.

I think you are overthinking it. I've seen people use it when it provides no real material benefit other than the placebo effect on the user to believe that the blur makes other people focus on their face.

fsloth• 6 years ago

Yeah, this is why I use the background blur. I have my wife's and my hobbystuff behind me. Can't really align the video/pc setup any other way. Rather provide a blur than a confusion of guitars, sewing kit and such.

swiley2• 6 years ago

Is it really that hard to set up a greenscreen for this? I can look out accross the street and see a number of people who have done this in their tiny apartments. If I cared about people being able to see the room behind my WFH set up I would do it too. Thankfully for me it just points at the wall I use as a projector screen so there's nothing to see. Plus my team seems to have just given up on video anyway.

josalhor2• 6 years ago

I find background blur even more distracting than background replacement. It's like my mind tries to picture the person that I am seeing in a particular environment and blur makes that process messy.

But that's not always true tho, I have seen background replacement all over people's face (and yes, I seem to be the only one who thinks that's wrong).

tziki• 6 years ago

I don't think anyone is being distracted by blurred hair or hands. If your coworkers don't feel comfortable even turning on the camera, it shouldn't matter to you. Aside from edge cases like a modelling agency looking for fresh faces, you have zero right to demand how people choose to potray themselves in a VC call.

0xffff22• 6 years ago

I have no problem with people choosing to leave their cameras off (I rarely turn my own camera on in meetings). I still think the complaint about poorly implemented background blur/background replacement is at least partially valid. It is very distracting to me compared to either a raw camera or no camera at all.

arketyp• 6 years ago

They mention SIMD support, but It's unclear to me in what capacity the GPU is leveraged. The hair segmentation example on the MediaPipe webpage suggests it's evaluating the graph on the GPU though.

jonex2• 6 years ago

The "Rendering Effects" section describes it in some detail: "Once segmentation is complete, we use OpenGL shaders for video processing and effect rendering" and some info on what that covers. (OpenGL parts runs on GPU)

kerng• 6 years ago

Interesting that this post made it to #1. It seems like Google marketing trick.

Anyone who uses the blue realizes that it's far lacking in quality from other offerings and Google Meet UI is very bad also.

Zoom, Teams, even WebEx are superior quality and usability wise.

rplnt4• 6 years ago

Google Meet UX is the worst. Except WebEx which is somehow even worse. But WebEx is at least "complicated" and bad, Meet is just bad.

Out of these I'm really surprised how "not as horrible" MS Teams are. Loads of functionality and the UX is bearable.

tantalor• 6 years ago

Can you elaborate your criticism? Meet seems fine to me.

lima3• 6 years ago

Curious to hear why? Google Meet is the only web-based videoconferencing product that works well for us without audio dropouts or random issues.

Zoom's web client is particularly terrible, and we can't install the desktop client for security reasons.

And the new background noise cancellation feature is magic.

tortasaur• 6 years ago

We haven't run into issues with a self-hosted instance of Jitsi Meet. Might be worth a look.

sundvor5• 6 years ago

I am going to admit that Nvidia Broadcast looks absolutely amazing to me. It's likely to be the reason why my next GPU won't be AMD's new, even though it appears to deliver much more bang for the buck.

I already have RTX Voice now and it's the best thing ever.

https://www.nvidia.com/en-au/geforce/news/nvidia-broadcast-a...

thinkloop• 6 years ago

Zoom, Teams, even WebEx are superior quality

Are they able to change the bg in the browser?

alblue• 6 years ago

Badly.

wdroz• 6 years ago

If you have a Windows computer with a RTX graphic card, you can use nvidia broadcast to get similar perks. It creates a virtual camera that you can select in whatever conference apps/browsers you are using.

There are some works on OBS to get the green screen AI working, so I hope we will get that on GNU/Linux one day.

amq• 6 years ago

The single biggest missing feature compared to Zoom for my team is background noise cancellation. It's an unfortunate decision to limit it to Enterprise users.

hrktb2• 6 years ago

I don’t understand this part:

In the current version, model inference is executed on the client’s CPU for low power consumption and widest device coverage.

Naively I would think model inference done server side would have the lower CPU power (from the client point of view) and widest device coverage (client does nothing more), what am I missing ?

blauditore• 6 years ago

Probably increased latency due to additional round trips to the server. Perhaps image streams are sent between clients directly, but I'm not sure.

jonex2• 6 years ago

It is done on the CPU instead of the GPU. GPU would seem like the natural choice for a convolution heavy model but was not used here for the mentioned reasons.

hrktb2• 6 years ago

Thanks, it makes sense

Orphis2• 6 years ago

Some work needs to happen locally to show you a preview of what you're going to transmit, as it should for most video related work.

If the segmentation is done server-side, then you need to sync it to the sender and reflect that quickly in the preview. It's probably not a great experience, at least for a launch.

mft_• 6 years ago

As an aside, this example looks faked:

https://1.bp.blogspot.com/-viEA4OY0sxA/X5s7IBwoXOI/AAAAAAAAG...

As in, the blurred background looks totally different (light:dark, shapes, etc.) to the unblurred background.

(I get that they’d need to do something funky to show blurred and unblurred backgrounds with the same foreground video, and faking it is likely easier than doing it programmatically, but this is just odd/sloppy.)

kmisiunas• 6 years ago

From the image caption "Background blur and background replacement, powered by MediaPipe on the web.".

Jyaif• 6 years ago

The fact that you think that the blurred background example is fake is a testament to how well background replacement work.

professor_v• 6 years ago

Have you actually used it? In practice I wouldn't say it works at all really. The artifacts are terrible.

germandude123• 6 years ago

The left clip is an example of background blur.

The right clip is an example of background replacement.

This is why the blurred background on the left does not look anything like the unblurred background on the right.

jcims4• 6 years ago

It would be nice if there was a webcam on the market that took actual lenses so you could get free, legit depth of field. Paying $700 for a used DSLR that has a clean hdmi out is not appealing, especially when I have a mirrorless from the same company that could probably do the same with a firmware update (that will never come)

nucleardog• 6 years ago

I think a cheaper solution would probably just be a depth sensing camera. Even a developer targeted Intel RealSense kit is only like $150. Consumer hardware could be much cheaper I imagine.

Once you have depth information integrated with a camera, then it should be pretty trivial to do background removal.

Whereas a 35mm f1.8 from Nikon is like $200 and whatever you mount it to is still going to need to do auto focusing and a bunch of other camera-y stuff to make it accessible to non photo geeks and then you’re going to need an off camera microphone so the entire call isn’t listening to your autofocus motor and...

mike_kamau• 6 years ago

Why are people replacing their backgrounds?

I thought the whole point of having a video call is to see who you are talking to, and their environment to further enhance the effectiveness of the conversation.

If you are in your kitchen, or under a tree, I definitely would like to see that because that environment will have an effect on how we communicate.

gerbler• 6 years ago

Sometimes people may not be comfortable sharing their backgrounds, and may not have convenient alternatives. For example, if you have a bed in the background it can be awkward and you might want to blur that out.

adwww• 6 years ago

I don't bother, but then I live in my own home and my background is an empty study.

I have coworkers who are in house shares with 5 other adults all trying to work from home around tiny desks. Background blur for them is a nice way to hide some of the chaos of their living arrangements.

spurgu3• 6 years ago

If the apartment is a mess in general. Table full of empty cans of beer. A dildo on a chair. Your wife randomly walking by in her underwear (not sure whether this would be unblurred?).

In the above scenarios, if I'm not certain there aren't going to be ackward things in behind me, I'd want to blur or set a custom background. Back against a wall also works which is what a lot of people seem to be doing.

0xffff22• 6 years ago

Why not just turn off your camera? The blurring tech doesn't seem nearly reliable enough for me to trust it if my "office" was that much of a catastrophe.

rkagerer• 6 years ago

Very awesome!

Although there's a lot of blurring on the shoulder of the guy at the beach: https://i.imgur.com/D5ueGUh.png

sercand• 6 years ago

I guess this is why when I open Google Meet my fan starts spinning and making noise.

adioe3• 6 years ago

Not supported in Firefox or Safari.

madeofpalk• 6 years ago

I wish one of the lovely people in the examples were wearing headphones.

kevingadd• 6 years ago

The listed CPU usage / elapsed time for the features in this article is obscene. Only 62FPS = maxing out at least one core on a 60hz display, just to replace/blur a background. Kiss your laptop's battery goodbye. How is this worth it?

The_rationalist• 6 years ago

What would be the inference time like on a modern smartphone?

acdha2• 6 years ago

It’s funny how Google pours time into things like this but the last person I know who uses a Google chat product just stopped because it’s less reliable than Zoom. Losing 15 minutes with someone trying to get the sound working counts more than a gimmick many people never notice, not to mention now even normal people don’t want to yet install another app because they expect it to be cancelled soon.

sjs7007• 6 years ago

Most corporate employees probably don't even have a choice of which app they can use though. They are tied in by whatever their company chose.

acdha2• 6 years ago

This is only true to the extent that the IT department has complete control and is insulated from user opinion. That tends to result in choices like WebEx or the ever popular shadow IT option. This is especially true now that everyone is doing this and the odds approach certainty that if someone is having problems they’ll suggest switching to a different product they know works.

Given the number of IT people I’ve heard express concerns about UI quality and eventual cancellation even for enterprise purchases, it’s also far from a given that the IT department is just blindly pushing a product.

tsycho• 6 years ago

Having used both Zoom and Meet extensively now for the past 6 months, my experience is:

1/ Your internet connection, especially upload bandwidth and latency matter a lot.

2/ Zoom's desktop app performs very well, but its web version is atrocious. Not just because of the dark patterns they use to force you to install the desktop app, but also its performance is terrible compared to its desktop version, as well as worse than almost everything else. Unfortunately, I don't trust them and refuse to use their desktop app on anything but my iPad.

3/ Meet used to be bad like Zoom on web 6 months ago, but has improved a lot and is slowly approaching Zoom desktop in performance. I have noticed that Meet on my work GSuite calls at work perform much better than on my personal account. This might be explained by #1 above I.e. my family has worse internet connections than my coworkers, but I am not sure if all improvements have been rolled out to personal accounts.

izacus3• 6 years ago

Meet certanly rolls out improvements for GSuite before public ones. I think there's even a GSuite setting of "release channel" where you can control how early you get these improvements.

boulos• 6 years ago

Disclosure: I work on Google Cloud (but not Meet).

For the last item, personal accounts (only?) default to send and receive video at lower resolution (360p). So if you meant that the quality is lower, you can set it on both sides to 720p.

Edit: I don’t think Meet remembers those settings though, so you have to do it every time (and show your family members how to do so).

kevincox• 6 years ago

It doesn't remember them, it is frustrating.

As a legacy free GApps user it is even more confusing because the admin page gives me an option to default to higher quality video but that doesn't do anything.

ss3000• 6 years ago

This is maddening...

Why does Google, with all the resources at its disposal, choose to cheap out like this when competitors in the video chat space (from tiny startups to gigantic corporation of similar size) have offered near native resolution video chat for ages?

Are they even _trying_ to compete?

k__• 6 years ago

I used Meet and Zoom in the last weeks.

Meet was much worse than Zoom, even when I take the bad web interface of Zoom into account.

I ain't a fan of either, though.

lotsofpulp• 6 years ago

1/ Your internet connection, especially upload bandwidth and latency matter a lot.

It grates me when people claim DSL/cable qualifies as sufficiently good broadband in the US because of the lack of upload bandwidth and slow latency (can add packet loss in here too). The situation is so bad that you can't even find how much upload bandwidth so called "broadband" cable ISPs offer.

The experience on symmetric fiber connections is noticeably improved, and we can have a house with a whole group of people streaming video up and down simultaneously without a hiccup. Such as in times of work from home and school from home.

wffurr2• 6 years ago

1/ Your internet connection, especially upload bandwidth and latency matter a lot.

I moved to a new house, and the quality of my video calls dropped dramatically. Constant freezing and dropouts. It was extremely frustrating to try to participate in a meeting. I could receive fine, but anytime I spoke out, I would drop out within minutes.

Speed tests showed plenty of bandwidth, but my modem statistics showed high upstream power levels, occasionally out of the allowed range, and lots of "uncorrectable" packets.

I finally got a Comcast technician in to look at it (yay for business-class support), and they replaced the cable from the pole all the way to the first splitter in the basement, and since then it's been flawless. 100/15 Megabit service has been totally adequate for our needs, so long as it's reliable and the latency is low enough.

It kills me that our city isn't putting in conduits or fiber while doing utility work, though. The whole time that was happening, there were gas contractors opening the street and running new supply lines to every house, but not putting in any extra conduits or dark fiber. The construction sounds were almost like being back in the office...

Ntrails• 6 years ago

Today I took a flawless webex meeting on a laptop tethered to my mobile phone, that same tether also allowed me to work without issue over rdp or whatever.

My mobile internet is really fucking good, and often outperforms my sodding wired connection

zamalek• 6 years ago

2

I refuse to install Zoom. They have removed the dark pattern, and the "join via browser" option is almost immediately available. If you have it installed, now is a good time to uninstall it.

daxfohl• 6 years ago

From the title I thought this was two distinct features running in google's background that used Web ML to figure out how to work together.

lern_too_spel2• 6 years ago

Why isn't Mediapipe built on gstreamer? Nvidia gets this right. If you're slinging frame buffers around, use an API that there is already an ecosystem for.

Liskni_si• 6 years ago

A few people commented that the foreground/background detection cannot keep up with movements fast enough. Here's an idea that might help, although I'm not sure if it can realistically be done:

When the video is encoded, the codec does motion estimation (among other things) to reduce the bandwidth required. So why don't we use the motion vectors from the video codec to modify the foreground/background mask in real time? Obviously this is going to create weird artifacts pretty soon, but it might just be good enough for a few frames before the ML model produces another accurate mask.

supernova87a2• 6 years ago

I have a different issue with Google Meet.

I have observed in the last couple months that whenever I create a Google Calendar invite with others, Google has started inserting a Google Meet conference as the location to meet.

It was one thing to ask/offer this as an option if you'd like to use it, but now Google is positioning it as if you had chosen that. So if you left it empty, because you usually use some other understood method with your friends/colleagues, now your participants are confused and think you wanted to use Google Meet.

I think that's going too far to get people to adopt your product.

fx32s• 6 years ago

I was able to change this default in the settings.

supernova87a2• 6 years ago

Ah, thanks for that!

It's still shady that they turned that on by default to get people to use it...

hongalex• 6 years ago

I noticed this too, but I actually got a tooltip popup that notes this is something that can be disabled in the calendar settings. The specific checkbox is "Automatically add Google Meet video conferences to events I create"

Disclaimer: I work at Google but not on these products.

Edit: it seems the tooltip only appears the first time you try to add Meet. After that it doesn't appear and you have to go into settings.

Nimitz14• 6 years ago

I was going to point out that xnnpack was basically created by a single guy who also created qnnpack, and how amazing it is for the work of a single guy to have so much impact, then I realized he posted it! Congratz dude!

Background Features in Google Meet, Powered by Web ML

Related Stories