MLow: Meta's low bitrate audio codec

https://engineering.fb.com/2024/06/13/web/mlow-metas-low-bitrate-audio-codec/

by mikece • 2 years ago

572 197 2 years ago

engineering.fb.com

PaulHoule• 2 years ago

Sometimes it sounds great but there are moments I think I'm listening to a harp and not somebody's voice.

plus• 2 years ago

It's not exactly reasonable to expect super high fidelity audio at the bitrate constraints they're targeting here, and it certainly sounds a lot better than the Opus examples they're comparing against.

cobbal• 2 years ago

The more complicated the codec, the more fascinating the failure modes. I love watching a digital TV with a bad signal, because the motion tracking in the codec causes people to wear previous, glitched frames as a skin while they move.

77pt772• 2 years ago

Are they comparing against opus using nolace?

Because that makes all the difference!

Tostino• 2 years ago

That is a marked improvement compared to the other examples provided. Nice to see it also has less compute resources required for that higher quality output.

zekica3• 2 years ago

Honest question: why do we need to optimize for <10kbps? It's really impressive what they are able to achieve at 6kbps, but LTE already supports >32kbps and there we have AMR-WB or Opus (Opus even has in-band FEC at these bitrates so packet loss is not that catastrophic). Maybe it's useful in satellite direct-to-phone use-cases?

ThrowawayTestr• 2 years ago

why do we need to optimize for <10kbps?

Because some people have really slow internet

hokumguru• 2 years ago

There exist a few billion people without LTE. Meta doesn’t only operate in the western world.

noprocrasted2• 2 years ago

Are there really many situations where a 10kbps connection would actually be stable enough to be usable? Usually when you get these kinds of speeds it means the underlying connection is well and truly compromised, and any kind of real-time audio would fail anyway because you're drowning in a sea of packet loss and retransmissions.

Even in cases where you do get a stable 10kbps connection from upstream, how are you going to manage getting any usable traffic through it when everything nowadays wastes bandwidth and competes with you (just look at any iOS device's background network activity - and that's before running any apps which usually embed dozens of malicious SDKs all competing for bandwidth)?

treflop• 2 years ago

Even in the Western world, you can appreciate low bandwidth apps even you are a music festival or traveling through relative wilderness.

gorkish6• 2 years ago

It's useful.

AMBE currently has a stranglehold in this area and by any and every measurable metric, AMBE is terrible and should be burned in the deepest fires of hell and obliterated from all of history.

londons_explore• 2 years ago

Internet connectivity tends to have a throughput vs latency curve.

If you need reliable low latency, as you want for a phone call, you get very little throughput.

Examples of such connections are wifi near the end of the range, or LTE connections with only one signal bar.

In those cases, a speedtest might say you have multiple megabits available, but you probably only have kilobits of bandwidth if you want reliable low latency.

zekica3• 2 years ago

Yes, but it doesn't have to be. Have you looked into Dave Taht's crusade against buffers?

lxgr10• 2 years ago

Load ratios of > 0.5 are definitely achievable without entering Bufferbloat territory, and even more is possible using standing queue aware schedulers such as CoDel.

Also, Bufferbloat is usually not (only) caused by you, but by people sharing the same chokepoint as you in either or both directions. But if you're lucky, the router owning the chokepoint has at least some rudimentary per-flow or per-IP fair scheduler, in which case sending less yourself can indeed help.

Still, to have that effect result in a usable data rate of kilobits on a connection that can otherwise push megabits (disregarding queueing delay), the chokepoint would have to be severely overprovisioned and/or extremely poorly scheduled.

cornstalks• 2 years ago

There’s a section (“Our motivation for building a new codec”) in the article that directly addresses this. Assuming you have >32 kbps bandwidth available is a bad assumption.

nicce2• 2 years ago

The best assumption would be that you either have connection available or not available.

Then, if it is available, what is the minimal data rate for connections which are available in general? If we do statistical analysis for that, is it lower that 32 kbps? How significantly?

For some reason, I would assume that if you have connection, it is faster than 2G these days.

hateful• 2 years ago

It's not only about the end that's receiving, it's also the end that's transmitting 10kbps * thousands of users.

sogen• 2 years ago

I'm assuming they'll just re-encode everything, for every user, to a lower bitrate using this codec.

So, with their huge user base they'll be saving a gazillion terabytes hourly, that's what I concluded from their "2 years in the making" announcement.

ajb• 2 years ago

If you mean for storage, real time codecs are actually pretty inefficient for that use case because they don't get much use of temporal redundancy. Although I'm not actually aware of a non-real time audio codec specialised for voice. They probably exist in Cheltenham and Maryland but for Meta this likely doesn't make a big enough part of their storage costs to bother

lxgr10• 2 years ago

Meta's use case are OTT applications on the Internet, which are usually billed per byte transmitted. Reducing the bitrate for the audio codec used lets people talk longer per month on the same data plan.

That said, returns are diminishing in that space due to the overhead of RTP, UDP and IP; see my other comment for details on that.

evandrofisico• 2 years ago

More than that, in developing countries, such as my own, Meta has peering agreements with telephony companies which allow said companies to offer basic plans where traffic to Meta applications (mostly whatsapp) is not billed. This would certainly reduce their costs immensely, considering that people use whatsapp as THE communications service.

kylehotchkiss• 2 years ago

Maybe something like this would be helpful for Apple to implement voice messages over satellite. Also a LOT of people in developing countries use WhatsApp voice messages with slow network speeds or expensive data. It's too easy to forget how big an audience Meta has outside the western world

gorkish6• 2 years ago

The lack of any reference or comparison to Codec2 immediately leads me to question the real value and motivation of this work. The world doesn't need another IP-encumbered audio codec in this space.

muizelaar• 2 years ago

They also don't compare with Lyra (https://github.com/google/lyra)

cvg• 2 years ago

Nice. Google's soundstream already has some great quality. Some examples at 6kbps here: https://google-research.github.io/seanet/soundstream/example...

gorkish6• 2 years ago

Or speex narrowband or others. I think the tendency to pick Opus is just because it has a newer date on it -- its design goals were not necessarily to optimize for low bitrate; Opus just happened to still sound OK when the knob was turned down that far.

One other point I intended to make that is not reflected in many listening/comparison tests offered by these presentations -- in the typical applications of low bitrate codecs, they absolutely must be able to gracefully degrade. We see Mlow performing at 6kbps here; how does it perform with 5% bit errors? Can it be tuned for lower bitrates like 3kpbs? A codec with a 6kbps floor that garbles into nonsense with a single bit flip would be dead-on-arrival for most real world applications. If you have to double the bitrate with FEC to make it reliable, have you really designed a low bitrate codec? The only example we heard of mlow was 30% loss on a 14kbps stream = 9.8kbps. Getting 6kbps through such a channel is a trivial exercise.

doodlesdev• 2 years ago

Lyra goes against the design goals of MLow though, by using machine learning techniques, and thus possibly not running on the devices that Meta is targeting with MLow.

Google claims SoundStream can run on low-end devices, though, so indeed I would like Meta to show that it still doesn't work well enough for their usecase. Specifically, I would like to know if it's possible to get SoundStream running in very old Android versions in low end devices, before a lot of APIs related to NNs in Android came around.

Dwedit• 2 years ago

There's also the LPCNet Codec (2019), which does wideband speech at 1.6kb/s by using a recurrent neural network.

https://jmvalin.ca/demo/lpcnet_codec/

nickels• 2 years ago

Could it be used for voice over satellite, ie Emergency SOS via satellite on iPhones?

lxgr10• 2 years ago

iPhones use Globalstar, which theoretically supports voice bitrates of (I believe) 9.6 kbps, although only using dedicated satphones with large, external antennas.

Apple's current solution requires several seconds to transmit a location stamp of only a handful of bytes, so I think we're some either iPhone or satellite upgrades away from real-time voice communication over that.

Starlink has demonstrated a direct-to-device video call already, though, so we seem to be quickly approaching that point! My strong suspicion is that Apple has bigger plans for Globalstar than just text messaging.

zekica3• 2 years ago

Starlink is in a better position as their satellites are in a low earth orbit - 30 times closer than geostationary. It correlates to 1000 times (30dB) stronger signal on both sides.

chronogram• 2 years ago

No mention of NoLACE make the comparison samples a bit less useful: https://opus-codec.org/demo/opus-1.5/

jamal-kumar• 2 years ago

That does sound very nice

sitkack2• 2 years ago

This is really cool and I very very very much appreciate that xiph puts so much work into standardization. https://datatracker.ietf.org/wg/mlcodec/documents/

It would be nice if Meta donated this to the world so we have less anchors for patent trolls and can transition the future we deserve.

77pt772• 2 years ago

Where is the source code?

lxgr10• 2 years ago

All these new, low-bitrate codecs are amazing, but ironically I suspect that they won't actually be very useful in most of the scenarios Meta is using them:

To keep latency low in real-time communications, the packet rate needs to be relatively high, and at some point the overhead of UDP, IP, and lower layers starts dominating over the actual payload.

As an example, consider (S)RTP (over UDP and IP): RTP adds at least 12 bytes of overhead (let's ignore the SRTP authentication tag for now); UDP adds 8 byte, and IPv4 adds 20, for a total of 40. At at typical packet rate of 50 per second (for a serialization delay of 1/50 = 20ms), that's 16 kbps of overhead alone!

It might still be acceptable to reduce the packet rate to 25 per second, which would cut this in half for an overhead of 8 kbps, but the overhead would still be dominating the total transmission rate.

Where codecs like this can really shine, though, is circuit-switched communication (some satphones use bitrates of around 2 kbps, which currently sound awful!), or protocol-aware VoIP systems that can employ header compression such as that used by LTE and 5G in IMS (most of the 40 bytes per frame are extremely predictable).

tgtweak4• 2 years ago

I think this is likely incorrect based on how much voice/audio distribution meta does today with facebook (and facebook live), instagram and whatsapp - moreso with whatsapp voice message and calling given it's considerable market share in countries with intermittent and low-reliability network connectivity. The fact it is more packet-loss robust and jitter-robust means that you can rely on protocols that have less error correction, segmenting and receive-reply overhead as well.

I don't think it's unreasonable to assume this could reduce their total audio-sourced bandwidth consumption by a considerable amount while maintaining/improving reliability and perceived "quality".

Looking at wireshark review of whatsapp on an active call there was around 380 UDP packets sent from source to recipient during a 1 minute call, and a handful of TCP packets to whatsapp's servers. That would yield a transmission overhead of about 2.2kbps.

quick edit to clarify why this is: you can see starting ptime (audio size per packet) set to 20ms here, but maxptime set to 150ms, which the clients can/will use opportunistically to reduce the number of packets being sent taking into consideration the latency between parties and bandwidth available.

(image): https://www.twilio.com/content/dam/twilio-com/global/en/blog...

lxgr10• 2 years ago

What part of that calculation is incorrect in your view?

380 UDP packets sent from source to recipient during a 1 minute call, and a handful of TCP packets to whatsapp's servers. That would yield a transmission overhead of about 2.2kbps.

That sounds like way too many packets! 380 packets per second, at 40 bytes of overhead per packet, would be almost 120 kbps.

My calculation only assumes just 50, and that’s already at a quite high packet rate.

you can rely on protocols that have less error correction

You could, but there's no way to get a regular smartphone IP stack running over Wi-Fi or mobile data to actually expose that capability to you. Even just getting the OS's UDP stack (to say nothing of middleboxes) to ignore UDP checksums and let you use those extra four bytes for data can be tricky.

Non-IP protocols, or even just IP or UDP header compression, are completely out of reach for an OTT application. (Networks might transparently do it; I'm pretty sure they'd still charge based on the gross data rate though, and as soon as the traffic leaves their core network, it'll be back to regular RTP over UDP over IP).

What they could do (and I suspect they might already be doing) is to compress RTP headers (or use something other than RTP) and/or pick even lower packet rates.

I don't think it's unreasonable to assume this could reduce their total audio-sourced bandwidth consumption by a considerable amount while maintaining/improving reliability and perceived "quality".

I definitely don't agree on the latter assertion – packet loss resilience is a huge deal for perceived quality! I'm just a bit more pessimistic on the former, unless they do the other optimizations mentioned above.

vel0city• 2 years ago

Another interesting use case for these kinds of ultra-low bitrate voice compression systems are digital radio systems. AMBE+2 and similar common voice codecs used on radio systems sound pretty miserable and don't handle dropped packets nearly as gracefully as compared to these newer codecs.

toast03• 2 years ago

Latency is the mind killer, but if available bandwidth is low, you save a ton of overhead by bundling 2-5 of your 20ms samples. Enough that the codec savings start to make sense, even though 100ms packets adds a ton of latency. Fancier systems can adapt codecs and samples per packet based on current conditions. The one I work on is a static codec and 60 ms of audio per packet, which isn't ideal, but allows us to run in low bandwidth much better than 20 ms per packet.

Edit to add: Meta can also afford to add a bit more sampling delay, because they've got very wide distribution of forwarding servers (they can do forwarding in their content appliances embedded in many ISPs), which reduces network delay vs competing services that have limited ability to host forwarding around the globe. Peer to peer doesn't always work and isn't always lower delay than going through a nearby forwarding server.

akira25012• 2 years ago

you save a ton of overhead by bundling 2-5 of your 20ms samples.

You pay that price back when the packets are dropped. We use a lot of advanced codec managers for broadcast, and while they do combining like this, they also offer the ability to repeat frames within subsequent packets. So you may get a packet with frames [1,2,3] then a packet with frames [2,3,4].

The best codecs actively monitor the connection and adjust all these parameters in real time for you.

lukevp• 2 years ago

Why would you need 50 packets per second vs 10? Is 100ms not acceptable but 20ms is?

tgtweak4• 2 years ago

Default configuration for SIP used to be 20ms, the rationale behind it was actually sourced in the fact that most SIP was done on LANs and inter-campus WAN which had generally high bitrate connectivity and low latency. The lower the packet time window the sooner the recipient could "hear" your voice, and if there were to be packet loss, there would be less of an impact if that packet were dropped - you'd only lose 20ms of audio vs 100ms. The same applies for high bitrate but high latency (3g for example) connectivity - you want to take advantage of the bandwidth to mitigate some of the network level latency that would impact the audio delay - being "wasteful" to ensure lower latency and higher packet loss tolerance.

Pointedly - if you had a 75ms of one-way latency (150ms RTT) between two parties, and you used a 150ms audio segment length (ptime) you'd be getting close to the 250ms generally accepted max audio delay for smooth two-way communication. the recipient is hearing your first millisecond of audio 226ms later at best. If any packet does get lost, the recipient would lose 150ms of your message vs 20ms.

Modern voice apps and voip use dynamic ptime (usually via "maxptime" which specifies the highest/worst case) in their protocol for this reason - it allows the clients to optimize for all combinations of high/low bandwidth, high/low latency and high/low packet loss in realtime - as network conditions can often change during the course of a call especially while driving around or roaming between wifi and cellular.

crazygringo• 2 years ago

Is 100ms not acceptable but 20ms is?

Yup pretty much. Doubling it for round-trip, 200 ms is a fifth of a second which is definitely noticeable in conversation.

40 ms is a twenty-fifth of a second, or approximately a single frame of a motion picture. That's not going to be noticeable in conversation all.

Of course both of these are on top of other sources of latencies, too.

NavinF• 2 years ago

Yes 100ms feels horrible. People constantly interrupting each other because they start talking at around the same time and then both say "you go first". Discord has decent latency and IMO it's a major reason behind their success

dilyevsky• 2 years ago

Anything over 40ms will be noticeable. Just to give you an idea how sensitive our ears are - there’s a max distance certain instruments can sit away from each other in an orchestra booth or they start falling out of sync due to speed of sound delay

yalok• 2 years ago

this codec is for RTC comms - it supports 20ms frame rate. They did mention it's launched in their calling products:

"We have already fully launched MLow to all Instagram and Messenger calls and are actively rolling it out on WhatsApp—and we’ve already seen incredible improvement in user engagement driven by better audio quality."

saurik• 2 years ago

I don't know of any setups which would support muxing in exactly the way I am thinking, but another interesting use case is if you have multiple incoming audio streams which you don't want to be mixed by the server -- potentially because they are end-to-end encrypted -- and so a single packet can contain the data from multiple streams. Doing end-to-end encrypted audio calls is finally becoming pretty widespread, and I could see Facebook being in a good position for their products to do custom muxing.

sjacob• 2 years ago

What skills/concepts would one need to acquire in order to fully grasp all that you've detailed here?

dan-robertson• 2 years ago

I’m not totally certain about your argument for the specific amount of overhead (if the receiver/sender are on mobile networks, maybe something happens to the packet headers for the first/last legs before the real internet). But doesn’t the OP already give an example where the low bit-rate codec is good: if you can compress things more then you have more of an opportunity to add forward error correction, which greatly improves the quality of calls on lossy connections. I wonder if smaller packets are less likely to be lost, and if there are important cases where multiple streams may be sent like group-calls.

woah• 2 years ago

According to the blog post, this is all practical research that Meta is doing to improve their services. Maybe that is BS, but I kind of doubt that given that Meta is in fact one of the biggest providers of voice and video calls on low bandwidth devices.

What makes you think that they have somehow just been misleading themselves the whole time?

mmmBacon• 2 years ago

They do this to optimize user experience for 3G and 2G networks not LTE or 5G. A huge number of people are on those networks.

jhugo• 2 years ago

Having traveled widely in three of the four most populous countries I’m a bit dubious of this claim. Having decent 4G in the middle of nowhere in India or Indonesia is pretty standard these days. I haven’t been to Africa, but a quick search suggests more than two thirds of the population of the continent has 4G coverage, and more than 80% has 3G. Where are these huge numbers of people on 2G?

kragen3• 2 years ago

what you're saying is that tcp/ip sucks at 9600 baud and below, nothing to do with voice really. this is a well-known fact, and solutions to the problem include cslip, vj header compression in ppp, and csp, as well as the lte and 5g stuff you mention

trustmebroo• 2 years ago

webrtc engineer here...

You are right that OPUS already hits the ball for most audio applications. 20kbps is pretty good, and 30 kbps is almost perfect.

Meta has tons of Indian and South Asian users with insanely poor network. It's already using 60ms/120ms packets (16.6/8.3 packet rate) for those people, but still, some calls are still bad.

What this new codec does is to further make the call accessible by bringing down the bitrate, which seems to help their DAU.

Nitpick: measuring packet rate on your device doesn't usually work. I think meta (as well as most other voip providers) is using DTX, so when user doesn't talk, packet rate is 2/sec.

dgmdoug• 2 years ago

They also don't do a comparison with Pied Piper.

mig39• 2 years ago

It might have a Weissman score in the fives, but I haven't seen a real-world implementation. Does it use middle-out compression?

byteknight• 2 years ago

MIDDLE OUT!

barbazoo• 2 years ago

That's great, now they can reach even more developing countries and do damage the way they did for example in Myanmar [1].

[1] https://www.amnesty.org/en/latest/news/2022/09/myanmar-faceb...

thrtythreeforty• 2 years ago

Are they releasing this or is this just engineering braggadocio? I can't find any other references to MLow other than this blog post.

Facebook/Meta AI Research does cool stuff, and releases a substantial portion of it (I dislike Facebook but I can admit they are highly innovative in the AI space).

sllabres• 2 years ago

If you think about 'implementing then algorithm in a product' it seems so: (From the article) "We are really excited about what we have accomplished in just the last two years—from developing a new codec to successfully shipping it to billions of users around the globe"

animanoir• 2 years ago

Nice technology, tho Opus adds that warm sound I love...

aidenn02• 2 years ago

Only slightly OT:

ELI5: Why is a typical phone call today less intelligible than a 8kHz 8-bit μ-law with ADPCM from the '90s did?

[edit]

s/sound worse/less intelligible/

skygazer• 2 years ago

Does decrease in intelligibility correlate with the instance count of concert seats in front of the loud speakers back in the oughts?

toast03• 2 years ago

Depends on your call; u-law has poor frequency response and reasonable dynamic range. Not great for music, but ok enough for voice, and it's very consistent. 90s calls were almost all circuit switched in the last mile, and multiplexed per sample on digital lines (T1 and up). This means very low latency and zero jitter; there would be a measurable but actually imperceptible delay versus an end to end analog circuit switched call; but digital sampling near the ends means there would be a lot less noise. Circuit switching also means you'd never get dropped samples --- the connection is made or its not, although sometimes only one-way.

Modern calls are typically using 20 ms samples, over packet switched networks, so you're adding sampling delay, and jitter and jitter buffers. The codecs themselves have encode/decode delay, because they're doing more than a ADC/DAC with a logarithm. Most of the codecs are using significantly fewer bits for the samples than u-law, and that's not for free either.

HD Voice (g.722.2 AMR-Wide Band) has a much larger frequency pass band, and sounds much better than GSM or OPUS or most of these other low bandwidth codecs. There's still delay though; even if people will tell you 20-100ms delay is imperceptible, give someone an a/b call with 0 and 20 ms delay and they'll tell you the 0 ms delay call is better.

rylittle2• 2 years ago

could you explain a little more in a more ELI5 please?

Dylan168072• 2 years ago

HD Voice (g.722.2 AMR-Wide Band) has a much larger frequency pass band, and sounds much better than GSM or OPUS or most of these other low bandwidth codecs.

At what bitrate, for the comparison to Opus?

And is this Opus using LACE/NoLACE as introduced in version 1.5?

...and is Meta using it in their comparison? It makes a huge difference.

akira25012• 2 years ago

and multiplexed per sample on digital lines (T1 and up).

Technically even on ISDN because you had channel bonding there. Although it's all still circuit switched. The timeslot within the channel group is reserved entirely for your use at a fixed bandwidth and has full setup and tear down that is coordinated out of band.

sva_• 2 years ago

Hearing ability deteriorates with age.

aidenn02• 2 years ago

Yes, but it doesn't deteriorate in such a way as to cause someone speaking to sound like gibberish and/or random medium-frequency tones, which happens in nearly every single cell phone conversation I have that lasts more than 5 minutes.

My experience is that phone calls nowadays alternate between a much wider-band (and thus often better sounding) experience and "WTF was that just now?"

hot_gril2• 2 years ago

Phone ear speakers are quieter than they used to be, so if the other person isn't talking clearly into the mic, you can't crank it up. I switched from a flip phone to an iPhone in 2013, huge difference. I had to immediately switch to using earbuds or speakerphone. Was in my teens at the time.

kragen3• 2 years ago

packet switching drops packets, circuit switching drops attempted calls (all circuits are busy)

most 90s calls didn't use adpcm, just pcm. assuming you got confused there

they also didn't use radio; solid wire from your microphone to my receiver. radio (cellphones, wifi, cordless phones) also is inherently unreliable

old phones had sidetone, but many voip apps don't

finally, speakerphone use is widespread now, and it is incompatible with sidetone and adds a lot of audio multipath fading

iamnotsure• 2 years ago

Please stop lossy compression.

cheema332• 2 years ago

Lossy compression has its practical uses. Under ideal circumstances nobody is going to stop you from using FLAC.

GrantMoyer2• 2 years ago

Have you ever looked at the size of losslessly compressed video? It's huge. Lossy compression is the only practical way to store and stream video, since it's typically less than 1% of the size of uncompressed video. Lossless compression typically only gets down to about 50% of the size. It's amazing how much information you can throw out from a video, and barely be able to tell the difference.

aembleton• 2 years ago

Surely all digital audio is lossy to a certain extent as you have a finite number of quantization levels and a finite sampling frequency.

GrantMoyer2• 2 years ago

It depends on what you consider lossy.

For time discretization, the Nyquist–Shannon sampling theorem[1] says a band-limited, continuous-time signal can be perfectly reconstructed from a discrete-time signal with a sufficiently high (but finite) sampling frequency. Human hearing is naturally band limited to about the range 20Hz to 20kHz, and audio recordings typically use sufficient sampling frequency to losslessly recontruct thos bandwidth.

For quantization, any recorded analog signal is a true signal plus some level of measurement noise. Quantization of a signals can also be thought of as adding some noise to a continuous amplitude signal[2]. If the "quantization noise" is much smaller than the measurement noise of the signal, then the discretization is effectively lossless. Typical audio formats have a maximum quantization error far, far smaller than typical audio recording hardware's measurment error (and the human ear's).

So typical quantized, discrete-time audio formats can be considered lossless representations of the anolog sound signals humans can hear (assuming proper capture and processing). On the other hand, no quantized or discretized signal can losslessly represent a non-band-limited, zero-noise audio signal.

[1] https://en.wikipedia.org/w/index.php?title=Nyquist%E2%80%93S...

[2] https://en.wikipedia.org/wiki/Quantization_(signal_processin...

amelius2• 2 years ago

Can't we have an audio codec that first sends a model of the particular voice, and then starts to send bits corresponding to the actual speech?

roywiggins• 2 years ago

You need a bunch of bandwidth upfront for that, which you might not have, and enough compute available at the other end to reconstruct it, which you really might not have.

amelius2• 2 years ago

Regarding your first point, how about reserving a small percentage of the bandwidth for a model that improves incrementally?

neckro23• 2 years ago

This is actually an old idea, minus the AI angle (1930s). It’s what voders and vocoders were originally designed for, before Kraftwerk et al. found out you can use them to make cool robot voices.

hubraumhugo• 2 years ago

Is it just my perception or has Meta become cool again by sharing a ton of research and open source (or open weights) work?

Facebook's reputation was at the bottom, but now it seems like they made up for it.

mrguyorama• 2 years ago

How the hell does releasing one audio codec undo years and years of privacy nightmare, being a willing bystander in an actual genocide, experimenting with the emotions of depressed kids, and collusion to depress wages?

pt_PT_guy• 2 years ago

they also did release LLM models, and zstd, and mold, and and and... a lot of stuff

risho• 2 years ago

it isn't just one audio codec. they also released and continue to release the best self hostable large language model weights, they have authored many open source projects that are staples today such as zstandard, react, pytorch

stuxnet79• 2 years ago

You will need to provide citations on the last point as Facebook are widely known to have broken the gentleman's agreement between Apple and Google that was suppressing tech pay in the early 2010s.

yard20102• 2 years ago

You forgot selling the 2016 US elections to Putin for 100k[0]

Good luck undoing that releasing codecs haha

[0] https://time.com/4930532/facebook-russian-accounts-2016-elec...

nine_k• 2 years ago

I have the same impression.

Facebook the social network reputation may be not shiny, but Meta the engineering company reputation is pretty high, to my mind.

It's somehow similar to IBM, who may look not stellar as a hardware or software solutions provider, but still have quite cool research and microelectronics branches.

danuker• 2 years ago

I don't think they made up for it. They are training AIs off of personal data. The open stuff are a desperate red herring.

https://www.theregister.com/2024/06/10/meta_ai_training/

jhallenworld• 2 years ago

So this is one argument- another is that I'm impressed that they got their own LLM running and integrated into facebook messenger.

I ran across this interesting graphic recently:

https://www.theverge.com/2023/12/4/23987953/the-gpu-haves-an...

Suddenly facebook is useful as a search engine..

Also interesting is that Meta AI is much faster than ChatGPT from end-user's point of view, but results not quite as good. Here is a comparison:

https://www.youtube.com/watch?v=1vLvLN5wxS0

jen729w2• 2 years ago

a desperate red herring.

Or you can recognise that ‘Meta’ isn’t a conscious entity, and that it’s perfectly likely that there are some people over there doing amazing open-source work, and different people over there making ethically dubious decisions when building their LLMs.

RIMR• 2 years ago

I would say that I have a very favorable opinion of Meta in terms of how they share their research and open source their software.

I would say that I have a very unfavorable opinion of Meta in terms of their commitment to privacy, security, and social responsibility.

lucb1e4• 2 years ago

research department != product department

Microsoft Research also puts out some really cool stuff, but that does not mean the "same" Microsoft can't show ads in their OS' start menu for people's constant enjoyment. I noticed this interesting discrepancy in Microsoft some years ago as a teenager; it does not surprise me at all that Facebook has a division doing cool things (zstandard, etc.) and a completely separate set of people working towards completely different goals simultaneously. Probably most companies larger than a couple hundred people have such departmental discrepancies

callalex2• 2 years ago

I racked my brain and couldn’t think of a single example of something released by Microsoft Research offhand, so I proceeded to visit their website. It can be found at https://www.microsoft.com/en-us/research/

I found exactly zero things that they have provided to the world. Literally none, and I really came in with an open mind. All I found was marketer drivel telling me how I should feel about the impact of the marketing copy I was currently reading. No concrete examples after fanning out 3 links deep into every article posted on that site. I must be missing something so can you point me to at least one example of work that’s come out of Microsoft Research on the same scale as Meta’s LLM models or ReactJS?

yard20102• 2 years ago

Don't fall for it. They will find a way to let some powerful actors exploit the users for dimes.

chefandy• 2 years ago

The marketing is def working. I'm sure we'd be pretty depressed by some of the projects that didn't make the blog cut.

rldjbpin• 2 years ago

they've had a proven history of open-sourcing systems they engineered to power their services. cassandradb and (py)torch comes to mind.

thenoblesunfish• 2 years ago

Is there a peer reviewed paper associated with this, or just a blog post?

theoperagoer2• 2 years ago

Was hoping this would have a GitHub link ...

mcoliver• 2 years ago

Maybe SiriusXM can pick this up. The audio quality is generally awful but especially on news/talk channels like Bloomberg and CNBC. There is no depth or richness to the voices.

tgtweak4• 2 years ago

It actually comes down to the SiriusXM receiver that is being used - I've witnessed the built-in sirius/xm on on the latest GM platform (a $100,000+ Cadillac) sounding like AM radio to immediately sitting in a better-than-apple-streaming quality rendition of the exact same channel on an older lexus a few minutes apart...

The mobile xm receivers (like ipods) that they used to sell also had very good quality and I never noticed any quality shortcomings even with good headphones.

I think the "high" quality stream is 256kbps/16k which is fairly high compared to most streaming services that come in around 128/160.

sitkack2• 2 years ago

I am archiving some music at 40kbps using Opus and the quality is pretty amazing. I think once things get over 20+kbps all the codecs start sounding pretty good (relative to the these low bitrates).

I still prefer flac if possible.

wildzzz2• 2 years ago

My old Sirius portable receiver sounded like garbage despite the marketing material saying "Crystal Clear". My 2006 Infiniti Sirius receiver didn't sound any better despite being a massive space heater in the trunk. The later cars I've used it in sound good, at least good enough to sound as clear as FM radio or even HD radio. I think some of the channels are still in lower bitrates like the news channels for example, they've always sounded bad. There's something I've read about SiriusXM using terrestrial transmitters which may improve the signal whereas the satellite link may be of lower bandwidth.

victorp13• 2 years ago

Does anyone happen to know if ChatGPT's voice feature uses audio compression similar to Opus? Especially the "heavy 30 percent receiver-side packet loss" example sounds a LOT like the experience I have sometimes.

dsign• 2 years ago

Can I use this to make music?

A little bit on a tangent, a technique called Linear Predictive Coding, which was developed by telecoms in the sixties and seventies, has a calculated bandwidth of 2.5 kbit/s. The sound quality is not any good, and telephone companies of the time didn't use it for calls, but the paper I read describing the technique says the decoded speech is "understandable". LPC found its way into musical production, in a set of instruments called "vocoders" used to distort a singer's voce. There are, for example, variations of it in something called "Orange Vocoder IV".

So, now I'm wondering, can MLow be used to purposefully introduce interesting distortions in speech? Or even change a singer's voice color?

WalterSear• 2 years ago

Just use Digitalis :)

https://www.youtube.com/watch?v=bA23ysR2hAo

dbcurtis• 2 years ago

What is the license? I searched but could not find anything.

varenc• 2 years ago

It appears to be entirely closed source at the moment. This is just the announcement of development of in-house tech. Nothing public yet that could even be licensed.

annoyingnoob• 2 years ago

I wonder how it sounds compared to G.729.

I worked for a company 20 years ago that had a modified G.729 codec that could go below 8kbps but sounded decent. We used this for VoIP over dial-up Internet, talk about low bandwidth.

Turns out some of the more interesting bits were in the jitter buffer and ways to manage the buffer. Scrappy connections deliver packets when they can and there is an art to managing the difference between the network experience and the user experience. For communications, you really need to manage the user experience.

eelioss• 2 years ago

I am curious about encoding times comparing with standard codecs using ffmpeg

lucb1e4• 2 years ago

They claim it's 10% lower than Opus, specifically for the decoder iirc but since they speak of the 10-year-old hardware used to make million of WhatsApp calls daily, the encoder can't be computationally complex either

But, yeah, some actual data (if they're not willing to provide running code) would have been a welcome addition to this PR overview

therealmarv• 2 years ago

Somebody knows if this is better compared to whatever Google Meet is using? With choppy near unusable slow Internet Google Meet still fulfils its purpose on Audio Calls where all other competitors fail (tested e.g. while being in Philippines on a remote island with very bad internet). However Google Meet's tech is not published anywhere afaik.

lucb1e4• 2 years ago

Can hardly try that out if this PR piece does not contain any code. We can judge it as well as you can from the couple examples they showed off

mckirk• 2 years ago

Maybe it's just me (or maybe I've invested too much money into headphones), but I actually liked the Opus sound better at 6 kbps. The MLow samples had these... harsh and unnatural artifacts, whereas the Opus sound (though sounding like it came from a tin-can-and-string telephone and lacked all top-end) at least was 'smooth'. But I'm pretty sure that's because they are demonstrating here the very edge of what their codec can do, at higher bitrates the choice would probably be a lot clearer.

FrostKiwi• 2 years ago

Not sure of the timing, is this what Meta researched when fighting audio latency for Virtual Meetings during the development of the Meta Quest 2?

andrewstuart• 2 years ago

This is amazing.

So you could have about 9 voice streams over a 56K modem.

Incredible.

Great for stuffing audio recordings into little devices like greeting card audio players and microcontrollers.

justinclift• 2 years ago

Doesn't seem to have any mention of what they're going to do with this patent wise.

The one of the biggest points for Opus was it being patent free (and "good enough" audio wise), so anyone and everyone was able to use it legally.

arendtio• 2 years ago

Does someone know if we can expect to see similar improvements for video streaming in the near future?

snvzz• 2 years ago

Looks (and sounds) cool.

What is the license?

jacobgorm• 2 years ago

Will the source code be available?

megamix• 2 years ago

Meta..

sidcool• 2 years ago

Meta engineering really rocks. All other controversies aside.

yobid20• 2 years ago

Is it free and open source? Any plans on making it available for native webrtc in the browser?

Rakshith• 2 years ago

they should all jump straight to vvc with lcevc to enhance it

vagab0nd• 2 years ago

How low can we go? Ultimately we just need to send "who's speaking" at the beginning and the content as text. So about 20 bps. Plenty of room at the bottom.

MLow: Meta's low bitrate audio codec

Related Stories