pthatcherg

I'm excited to see a major use of WebTransport!

Why don't you like libwebrtc? There's a lot of good stuff in there.

This is just an initial proposal, and probably will reduce down to one with developer feedback and implementation experience, which is normal for API development.

Whether it's one shape or the other doesn't take affect the network protocol underneath.

If you have constructive ideas of how to simplify the protocol underneath or the API, you can share them with the authors. The people that work on that APIs are always looking for constructive feedback

I'm not sure what you mean. It does work without a domain name and SSL cert.

I did something similar a while back: https://github.com/pthatcherg/stunl

WebRTC (with p2p) was done within the W3C, as was a p2p QuicTransport. And this new API is similar to the open screen APIs that have been done in the W3C.

So I'm pretty sure work like this can be done in the W3C.

I was one of the original authors of the p2p QuicTansport API and the WebTransport API which is based on it.

I've also recently discussed this new local p2p API with the authors. And I'm in the favor of it, at least in principle.

The difference is that local p2p does not require signaling like, say the p2p done by ICE, WebRTC, or p2p QuicTransport. So for some uses cases, this would be better. And, if done right, this would fit well into a cohesive set of APIs that cover both kinds of p2p.

I'm also in favor of making APIs low level, but I'm not sure if raw sockets is going to work. Keeping congestion control working is pretty important.

The article's information about WebRTC is not accurate. You can do client/server WebRTC without a "signaling server". Just make the server do the signaling. It takes a few extra round trips, but it doesn't need to be an extra server. And WebRTC data channels work quite well as a replacement for WebSockets or SSE, especially if you want to avoid head-of-line blocking. And there are many libraries that will do pretty much all of the work for you, like Pion or str0m.

I also think calling the WebTransport API complex is overblown. If you don't want the more advanced things, you can ignore them. If you want to use it like a WebSocket, just open one bidirectional stream and you're basically done. If you want to avoid head-of-line blocking, just open a stream for every message. It's a little more complex, but it's not the kind of thing you need a library for. Github Copilot will probably write the code for you. It's true there aren't as many server libraries out there yet, since WebTransport is still maturing. And we're waiting for Safari to add support.

There's also WebTransport

From a server or a native client, you can send whatever RTP packets you want, but you cannot send whatever RTP packets you want from a web client, and you cannot have access to the RTP packets from a web client and do whatever you want with them, at least not very easily. We are working on an extension to WebRTC called RtpTransport that would allow for just that, but it's in early stages of design and standardization.

It's shame it looks like web clients won't be able to connect to each other p2p. If it supported the use of ICE+DTLS+SCTP or even ICE+QUIC, then it web clients could join in.

Or am I missing something in the docs?

So WebTransport does what you were proposing to do?

It started in the W3C and the IETF group was added later. But thank you for pointing out that a lot of effort is put into making these standards a joint effort across the industry.

(I was one of the original authors in the W3C but left shortly after the BoF happened).

How would one add datagrams to WebSockets?

You can use ICE and QUIC together. That's what the original version of WebTransport did (back when it was called QuicTransport).

There's nothing stopping QUIC from being p2p. In fact, the original version of WebTransport was called QuicTransport and was p2p, but there was more interest in client/server use cases, so we switched focus to that.

(I'm one of the original authors of the WebTransort spec and when I worked at Google years ago, some excellent engineers on my team shipped the original QuicTransport behind a field trial).

It's been almost 3 years since I first presented on WebTransport + WebCodecs: https://youtu.be/VD5GBLBiSxo

Live streaming was a motivating example for both of those, as you can tell from the video. And both of them grew out of our efforts to make WebRTC better for live streaming.

That's correct. You can't mix without decrypting. Signal does not mix at the server because it can't.

More info is available at my blog post about it: https://signal.org/blog/how-to-build-encrypted-group-calls/

Hey, I'm the author of the Signal blog post about SFUs. I have a few questions/comments:

1. I don't think there are many good open source SFUs to choose from. I know of 2, maybe 3 (including our new one). There may be many, but few have good rate control. But maybe I just don't know about them? I'd be happy to learn of more good ones.

2. Echo cancellation is certainly a hard problem, but it doesn't conflict with E2EE unless you do it on the server, which isn't necessary. So perhaps it may be a somewhat harder problem because you close off one possible approach (doing it server-side), but many (most?) echo cancellation solutions are done client-side.

3. You may not be completely dependent on WebRTC's echo cancellation any more because of the new MediaStreamTrackProcessor and MediaStreamTrackGenerator APIs. I don't know if it will work for echo cancellation, but it might.

If you're main focus is to mostly receive 1-2 video streams while sending 0, then, yeah, I guess bundle vs non-bundle doesn't matter much. I'm pleasantly surprised to hear that 25 PeerConnections work well for you.

About simulcast: why would a sender not support simulcast? Something to do with VP8 hardware encoders?

I'm still interested to hear what you have learned about AQM and the like.

Oh, that's interesting. That's not as bad as I thought, although I am surprised that's the approach that was taken (unbundled).

In that case, if you're relying on REMB from WebRTC with multiple PeerConnections, then, yes, each will be handled separately and you'll likely not end up with loss-base congestion control but will be using the latency-based congestion control WebRTC is going.

Some potential problems I can think of with this approach:

- It won't scale to a large number of streams. I'm not sure at what point it would cause problems, but I'm guessing it would work OK for... 20? 40? Most of the time, the extra ICE traffic and per-connection RTCP is probably small. A long time ago, there were issues with per-PeerConnection memory usage, but I think we (back when I worked on the WebRTC team) fixed them to a reasonable degree, so it should work for 20-40. You have to do a bunch of DTLS handshakes, but you could get around that by using SDES. You'll opening a lot of NAT bindings, potentially at the same time. Again, with a smaller number of streams, this might be fine. But at some point, you may cause issues with consumer-grade NATs.

- The WebRTC receive-side congestion controllers (sending back REMB) will effectively be competing with one another. I'm not sure how well they work with a large number doing so (or even a small number.) I'd be interested to hear how many you can have running at the same time before you notice problems.

- You can't easily prioritize one stream over the others if you want an "active speaker" view. I suppose you could something where you "steal" bits from one PeerConnection and give it to another, but that will probably contradict what you're trying to do with playing nicely with traffic shapers and AQM.

I'd be very interested to hear your findings with how traffic shapers and AQM respond to non-bundled traffic vs bundled traffic. What characteristics do you seek work better or worse? Higher rates? Lower latencies? Lower jitter? Less loss? And how do you test (given that network behavior can vary so widely one network and the next)?

If you'd like to talk more directly I'm "peter at signal.org".

Yes, you are right that there is also a simple loss-based congestion control mechanism (https://github.com/jech/galene/blob/e8fbfcb9ba532f733405b1c5...) and a min() between it and the REMB. I missed that part. However, that appears to still be immature, only in a different way than I thought.

If I'm right that only one server->client video stream (called "down track" in the Galene code) is receiving the REMB message, then only one will use the REMB value and the rest will fallback to loss-based congestion control. If I'm wrong and all the server->client video streams are receiving the REMB message, then all of them will use that same value which will be higher than a calculated by the loss-based congestion control for each stream independently, so in effect they will all be falling back to loss-based congestion control (when there is more than 1 video stream; for 1 video stream it probably works fine).

Either way, it appears that each server->client video stream is independently running a loss-based congestion controller, all of which will be battling each other (like N TCP streams do). That can work, I guess, but it's better to run one congestion controller and then divide that bitrate among the various video streams, which is what I meant by "bitrate allocation".

In other words, selecting video layers to send is exactly what I mean by bitrate allocation. Sorry for being unclear about that. The code you linked to is estimating the client->server bitrate for a given video stream. What I was looking for is code that will take a bitrate (from one congestion control mechanism, whatever it may be) and then divides that between the various video streams that flow server->client by selecting which layers to forward. I couldn't find that, and now I see why: because each video stream has its own congestion controller, and they apparently compete with each other where they are likely all loss-based in practice.

Loss-based congestion control for video conferencing isn't as good as latency-based congestion control because it will cause more latency.

Thanks for pointing out what I was missing. Now that I understand it better, I can see that it would work fine for 1 video stream (when there are only 2 clients in the call), but then likely falls back to loss-based congestion control for more than 2 clients, which will work, but not as well.

If this is accurate, then I'd make a few suggestions for Galene: 1. Use one "maxBitrate" calculation per-"rtpConn" instead of per-"downTrack" and then divide that bitrate between those downTracks rather than doing N such independent calculations. This will avoid the problems from having congestion controllers competing with each other. 2. Feed the REMB value from the receiving client into the unified calculation. Then you'll get the benefit of latency-based congestion control (assuming the client is doing latency-based congestion control). 3. Switch the loss-based mechanism in the server to a latency-based system using something like transport-cc (which I think Pion supports).

I was curious and looked through the code of Galene briefly and found the following, which may partially answer your question. For context, I am familiar with the Jitsi code and have written a calling server (and written about it: https://signal.org/blog/how-to-build-encrypted-group-calls/).

Galene appears to be less mature than Jitsi. For example, it uses REMB feedback messages from the client to calculate allowable bitrates rather than calculating the bitrates itself (as Jitsi and Signal's SFU do). Worse, it appears that what it does with that information is erroneous. I could be wrong, but it looks like the bitrate allocation code (see https://github.com/jech/galene/blob/e8fbfcb9ba532f733405b1c5...) only allocates the bitrate for one of the video streams, not all of them. Perhaps the author did not realize that there is one REMB sent back for all the video streams by WebRTC rather than one per stream (see, for example, here: https://source.chromium.org/chromium/chromium/src/+/main:thi...). Further, I find the spatial layer switching code to be strange. For examples, it doesn't go down a layer unless it's 150% over the estimated allowable bitrate, which gives a lot of opportunity for inducing latency.

In short, I think Galene has a ways to go before it works as well as Jitsi (Videobridge), and thus Pyrite group calls are unlikely to work as well as Jitsi group calls (for 1:1 calls, I don't know; I didn't look into that).

Oh, and just a reminder, the SFU we use for Signal group calls is also open source: https://github.com/signalapp/Signal-Calling-Service.

Yes, that's called "signaling". In the case of Signal, it's done through Signal messages, which pass through a server. But the server doesn't know they are calling related messages. They could be any message.

Yes, it knows the IP addresses of the clients.

The other thing it knows is that you have provided a proof that you are a member of a the group conversation associated with the group call (basically a proof that you are allowed to join the call). And that proof is based on Signal's "zkgroup" system.

That's it. Once the server has the proof, it forwards packets for you. And it doesn't learning about you from then proof (other than what the proof proves: that you can join).

We don't have a percentage because for 1:1 calls, we don't even know how many calls there are (it's completely p2p), but I've ofte heard quoted somewhere between 5-10% of calls.

Yes, TURN is only used if needed in those cases, and, yes, you could use a forwarding server instead of a TURN server (although with different tradeoffs).

I'll add the full mesh option to our list of options for future work that we discuss regularly.

It's open source. No objection from me.

CPU usage was so very high for calls over 5-6 participants. I'm not sure what more there is too it.

Author here.

We are considering making 2-person group calls switch to p2p under the hood. I don't know if that that will ever expand to 3 or more.

Author here.

What is in the article is a very simplified version of the code. The code that runs in production uses many threads.

See here: https://github.com/signalapp/Signal-Calling-Service/blob/mai...