kixelated

Yeah that's a really good way of framing the argument, I wish I wrote that. The way robots listen/respond is bounded by compute, not time. Buffering audio isn't a great experience for humans but definitely works for robots.

Close, but that's a minimum latency. We want a maximum latency knob.

To clarify, I meant waiting an extra 200ms if the alternative was dropping part of the prompt. During periods of zero congestion, the latency would be the same.

HELLO MR SEAN,

1. Of course users want lower latency, but they also want fewer instances where the LLM "misheard" them. It would be amazing to run A/B experiments on the trade-off between latency vs quality, but WebRTC makes that knob difficult to turn.

2. I'm obviously not an TTS expert, but what benefit is there to trickling out the result? The silicon doesn't care how quickly the time number increments?

3. Yeah, sometimes the client is aware when their IP changes and can do an ICE renegotiation. But often they aren't aware, and normally would rely on the server detecting the change, but that's not possible with your LB setup. It's not a big deal, just unfortunate given how many hoops you have to jump through already.

4. Okay, that draft means 7 RTTs instead of 8 RTTs? Again some can be pipelined so the real number is a bit lower. But like the real issue is the mandatory signaling server which causes a double TLS handshake just in case P2P is being used.

5. Of course WebRTC is easier for a new developer because it's a black box conferencing app. But for a large company like OpenAI, that black box starts to cause problems that really could be fixed with lower level primitives.

I absolutely think you should mess around with RTP over QUIC and would love to help. If you're worried about code size, the browser (and one day the OS) provides the QUIC library. And if you switch to something closer to MoQ, QUIC handles fragmentation, retransmissions, congestion control, etc. Your application ends up being surprisingly small.

The main shortcoming with RoQ/MoQ is that we can't implement GCC because QUIC is congestion controlled (including datagrams). We're stuck with cubic/BBR when sending from the browser for now.

Hello Mr Author here. Apologies that my comment replies aren't as funny.

Every low-latency application has to decide the user experience trade-off between quality and latency. Congestion causes queuing (aka latency) and to avoid that, something needs to be skipped (lower quality).

The WebRTC latency vs. quality knob is fixed. It's great at minimizing latency, but suffers from a lack of flexibility. We still (try to) use WebRTC anyway, because like you implied, browser support has made it one of the only options.

Until now of course! WebTransport means you can achieve WebRTC-like behavior via a generic protocol. Choose how long you want to wait before dropping/resetting a stream, instead of that decision being made for you.

And yeah my point in the blog is that often the user wants streaming, but not dropping. Obviously you can stream audio input/output without WebRTC. The application should be able to decide when audio packets are lost forever... is it 50ms or 500ms or 5000ms? My argument is that voice AI shouldn't pick the 50ms option.

QUIC libraries work by looping over pending streams (in priority order) to determine which UDP packet to send next. If there's more stream data than available congestion control, the data will send there in the stream send buffer.

Either side can abort a stream if it's taking too long, clearing the send buffer and officially dropping the data. It's a lot more flexible than opaque UDP send buffers and random packet loss.

FEC would make the most sense at the QUIC level because random packet loss is primarily hop-by-hop. But I'm not aware of any serious efforts to do that. There's a lot of ideas out there, but TBH MoQ is too young to have the production usage required to evaluate a FEC scheme.

Yep, it's similar to multicast but L7.

But a huge difference is that there's a plan for congestion. We heavily rely on QUIC to drain network queues and prioritize/queue media based on importance. It's doable with multicast+unicast, but complicated.

Yeah for Safari support I'm using polyfills; it sucks.

- libav.js for AudioEncoder/AudioDecoder. - QMux over WebSockets for WebTransport.

Both are NPM packages if you want to use them. @kixelated/libavjs-webcodecs-polyfill and @moq/qmux

26.4 removes the need for both so there's hope!

Absolutely agree.

You can convert any push-based protocol into a pull-based one with a custom protocol to toggle sources on/off. But it's a non-standard solution, and soon enough you have to control the entire stack.

The goal of MoQ is to split WebRTC into 3-4 standard layers for reusability. You can use QUIC for networking, moq-lite/moq-transport for pub/sub, hang/msf for media, etc. Or don't! The composability depends on your use case.

And yeah lemme know if you want some help/advice on your QUIC-based solution. Join the discord and DM @kixelated.

Hey lewq, 40Mbps is an absolutely ridiculous bitrate. For context, Twitch maxes out around 8.5Mb/s for 1440p60. Your encoder was poorly configured, that's it. Also, it sounds like your mostly static content would greatly benefit from VBR; you could get the bitrate down to 1Mb/s or something for screen sharing.

And yeah, the usual approach is to adapt your bitrate to network conditions, but it's also common to modify the frame rate. There's actually no requirement for a fixed frame rate with video codecs. It also you could do the same "encode on demand" approach with a codec like H.264, provided you're okay with it being low FPS on high RTT connections (poor Australians).

Overall, using keyframes only is a very bad idea. It's how the low quality animated GIFs used to work before they were secretly replaced with video files. Video codecs are extremely efficient because of delta encoding.

But I totally agree with ditching WebRTC. WebSockets + WebCodecs is fine provided you have a plan for bufferbloat (ex. adaptive bitrate, ABR, GoP skipping).

Yeah, technically it's SCTP over DTLS for data channels. Only the media layer gets to use raw UDP, limiting the scope.

QUIC has a much better alternative to FORWARD-TSN, either via RESET_STREAM or QUIC datagrams.

I've implemented SCTP before to hack in "datagram" support by spamming FORWARD-TSN. Fun fact: you can't use FORWARD-TSN if there's still reliable data outstanding. TSN is sequential after all, you have to drop all or nothing.

QUIC as a protocol is significantly better than SCTP. I really recommend the RFC

For sure, if you want an ordered/reliable stream then WebSocket is ideal. WebTransport is useful when you also want prioritization and semi-reliable networking, similar in concept to WebRTC data channels.

I like to frame WebTransport as multiple WebSocket connections to the same host, but using a shared handshake.

It's common to multiplex a WebSocket connection but you don't need that with WebTransport, while also avoiding head-of-line-blocking.

But yeah I wish WebTransport had a better TCP fallback. I still use WebSocket for that.

I maintain https://github.com/kixelated/web-transport

But yeah the HTTP/3 integration definitely makes WebTransport harder to support. The QUIC connection needs to be shared between HTTP/3 and WebTransport.

There's no probing in any QUIC implementation but it's possible. There's a QUIC extension in the IETF similar to transport-wide-cc but it would still be up to the browser to use it for any upload CC.

SCTP and by extension, WebRTC data channels, are supposed to use the same congestion control algorithms as TCP/QUIC. But I don't know which CC libsctp does these days.

WebTransport in Chrome currently uses CUBIC but the Google folks want to turn on BBR everywhere. It uses the same QUIC implementation as HTTP/3 so it's going to be more battle hardened.

I had a 30 minute intro call and got a rejection a few days later. It was VERY timely (Sep 21 email, Sep 23 meeting, Sep 26 rejection).

We both weren't sure if my project was a good fit for the program. It was still a positive experience, and they were nice enough to offer me an intro to a more relevant team within OpenAI.

I couldn't quite figure out the goal of Grove. The line about "pre-idea" individuals, and of course the referral offer, made me feel that it's more of a hiring pipeline and not a traditional incubator. But we'll see when they announce the cohort.

Thanks!

You don't need multicast! CDNs effectively implement multicast, with caching, in L7 instead of relying on routers and ISPs to implement it in L3. That's actually what I did at Twitch for 5 years.

In theory, multicast could reduce the traffic from CDN edge to ISP, but only for the largest broadcasts of the year (ex. Superbowl). A lot of CDNs are getting around this by putting CDN edges within ISPs. The smaller events don't benefit because of the low probability of two viewers sharing the same path.

There's other issues with multicast, namely congestion control and encryption. Not unsolvable but the federated nature of multicast makes things more difficult to fix.

Multicast would benefit P2P the most. I just don't see it catching on given how huge CDNs have become. Even WebRTC, which would benefit from multicast the most and uses RTP (designed with multicast in mind) has shown no interest in supporting it. But I did hear a rumor that Google was using multicast for Meets within their network so maaaybe?

QUIC has support for preferred address, where anycast is used for the QUIC handshake then the connection migrates to a unicast address. It still has issues but it's nice to have sticky established connections and avoid flapping mid connection.

Oh and the autoplay restrictions for <video> don't apply when muted.

It's all hardware accelerated, assuming the VideoDecoder has hardware support for the codec. VideoFrame is available in WebGL and WebGPU as a texture or gpu-buffer. We're only rendering after a`requestAnimationFrame` callback so decoded frames may get automatically skipped based on the display frame rate.

I don't think the performance would be any worse than the <video> tag. The only exception would be browser bugs. It definitely sounds like the black bars are a browser rendering bug given it's fine when recorded.

I answered in another reply, but client -> server protocols like TCP and QUIC don't have an issue traversing NATs. The biggest problem you'll run into are corporate firewalls blocking UDP, but hopefully HTTP/3 adoption helps that (UDP :443, same as WebTransport).

It works well right now because there's only one edge per continent. But if I had traffic, anycast is definitely the way to go.

My fault, I trying too hard to avoid rehashing previous blog posts: https://moq.dev/blog/replacing-webrtc/

And you're right that MoQ primarily benefits developers, not end users. It makes it a lot easier to scale and implement features; indirect benefits.

I am bad at CSS.

Yeah, and CMAF is just a fancy word for fMP4. The f in fMP4 meaning an MP4 file that has been split into fragments, usually at keyframe boundaries, but fragments can be as small as 1 frame if you're willing to put up with the overhead.

The Big Buck Bunny example on the website is actually streamed using CMAF -> MoQ code I wrote.

https://caniuse.com/webtransport https://caniuse.com/webcodecs

Technically, WebCodecs is not required since you can use MSE to render, but it's more work and higher latency.

Working on a WebSocket fallback and OPUS encoder in WASM to support Safari.

Horizontal black lines? Dunno what that could be about, we render to a <canvas> element which is resized to match the source video and then resized again to match the window with CSS.

EDIT: Sorry I just noticed this was directed to Cloudflare. They're using the same architecture as Cloudflare Realtime, their WebRTC offering.

`relay.moq.dev` currently uses GeoDNS to route to the closest edge. I'd like to use anycast like Cloudflare (and QUIC's preferred_address), but cloud offerings for anycast + UDP are limited.

The relays nodes currently form a mesh network and gossip origins between themselves. I used to work at Twitch on the CDN team so I'd like to eventually add tiers, but it's overkill with near zero users.

The moq-relay and terraform code is all open source if you're super curious.