Sean-Der

Write up about the architecture is here https://openai.com/index/delivering-low-latency-voice-ai-at-...

What's not easy about it? Would love to hear about the gaps in software/education that could make it easier to use.

Thanks Woodrow :)

Accepting 'Big Changes' from people is VERY frustrating. These thoughts run through my head.

* Idea is usually good! Even if I don't understand it could help lots of others users.

* The contributor is very focused on just getting their feature in. The impact on the larger project isn't as much a concern.

* New contributors often don't have the grit to see it out. They will disappear before things are done. So I am left picking up the pieces (which is harder then doing it all myself)

----

What I try and remember is that their happiness/experience matters more then any code. I try to help the contributor learn/grow as much as possible and even see some career benefits out of it. Pion will cease to matter eventually, so I hope to help as many programmers with it as possible.

NLNet is a wonderful organization. They have supported two Pion projects!

I am grateful the code got written, but even better people got careers out of it/learned new stuff. If you are on the fence about taking on a project I encourage you to do it!

Does WebRTC not work inside/outside of the browser anywhere?

This should be fixed!

I added this in Pion here[0] and I remember testing against Chrome + FireFox and it seemed to work great!

[0] https://github.com/pion/webrtc/commit/e4ff415b2bff31382bdb80...

Amazing debugging, I loved reading that. HN doesn't get enough good posts like this anymore :)

If https://github.com/pion/sctp/issues/12 had happened (not just in Pion but across all implementations) this could have been fixed years ago. The hardcoding we all settle for is tragic.

I work on WHIP in OBS and an open source project Broadcast Box[0]. I made this site/video to talk about the reasons why https://webrtcforthestreamer.com

I hope this site can convince more people to check this stuff out. If you are curious or have feedback I would love to hear.

[0] https://github.com/glimesh/broadcast-box

I work on WHIP in OBS and an open source project Broadcast Box. I see a future where streaming is better.

I hope this site can convince more people to check this stuff out. If you are curious or have feedback I would love to hear

I have been working on making streaming cheaper/more private/lower latency (via WebRTC) in OBS. Been working on this site to get people excited https://webrtcforthestreamer.com/ after it is done want to record a YouTube video for it.

After that I want to spend the weekend just closing out Pion bugs/relaxing :)

I can add it! What format would like to see?

The spirit is alive with https://github.com/m1k1o/neko

The 'co-watching/co-streaming' is the best. I love watching a crappy horror movie with friends and bantering.

You run one encoder for all the viewers. CPU usage won't scale up from 1 -> 15 viewers.

I could get it lower by encoding once and then syncing to keyframes. It would make the code more complicated though. If someone asks for it/gets excited would love to do it though :)

The 'Seek' is done server side. So if you go to `n` seconds it is done server side and done for everyone.

I should reword the README though. If someone is savy enough they could totally grab the video. For most users it is like a Google Meet though. If you click `Show Controls` you can pause the video that is it.

With things like Insertable Streams[0] you can totally grab the video.

[0] https://developer.mozilla.org/en-US/docs/Web/API/Insertable_...

That is 100% controllable! By setting Playout Delay Header[0] you can pick between 'drop everything to stay live' or buffering up to ~40 seconds!

In this project I don't set anything though.

[0] https://webrtc.googlesource.com/src/+/refs/heads/main/docs/n...

1.) Latency vs quality doesn't come up enough to make people want to A/B test it unfortunately. At work I would say ~5 people care about WebRTC vs QUIC vs X. All effort is around the models (how can I provide tools to be support those doing that work)

2.) The model isn't processing just text anymore. Also taking into account breathing/emotion etc... not just spitting out big responses anymore. As it generates them it is taking into account the users response.

3.) It works with the LB setup today. Clients are sending ICE traffic, if it roams we lookup the ufrag and route appropriately.

4.) With DTLS 1.3 it is 1 RTT with SNAP[0] for WebRTC session. SCTP info goes in Offer/Answer, DTLS is packed into ICE. You are totally right about signaling though! [1] was my answer for doing WebRTC without signaling, couldn't get anyone to care though.

5.) I don't have anything that I need to tune. If I want to increase (or decrease) latency [3] is something I put into Transceiver. Otherwise I can't think of any 'change this WebRTC behavior' that has been asked by users/developers.

[0] https://datatracker.ietf.org/doc/draft-hancke-tsvwg-snap/

[1] https://github.com/pion/offline-browser-communication

[3] https://webrtc.googlesource.com/src/+/refs/heads/main/docs/n...

That is not the case. See get-realtime-translate[0 that's doing it as a trickle instead (not turn based).

[0] https://developers.openai.com/api/docs/models/gpt-realtime-t...

Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though.

…but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate

This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the magic. You also don't want to send faster than real-time. If the user interrupts the model you just wasted a bunch of bandwidth sending 3 minutes of audio (but only played 10 seconds)

TTS is faster than real-time

https://research.nvidia.com/labs/adlr/personaplex/ Voice AI for the latest/aspirational is moving away from what the author describes. It is trickled in/out at 20ms

We really hope the user’s source IP/port never changes, because we broke that functionality.

That is supported. When new IP for ufrag comes in its supported

It takes a minimum of 8* round trips (RTT)

That's wrong. https://datatracker.ietf.org/doc/draft-hancke-webrtc-sped/

I’d just stream audio over WebSockets

You lose stuff like AEC. You also push complexity on clients. The simplicity of WebRTC (createOffer -> setRemoteDescription) is what lets people onboard easily. Lots of developers struggled with Realtime API + web sockets (lots of code and having to do stuff by hand)

----

I think if I had my choice I would pick Offer/Answer model and then doing QUIC instead of DTLS+SCTP. Maybe do RTP over QUIC? I personally don't feel strongly about the protocol itself. I don't know how to ship code to multiple clients (and customers clients) with a much large code footprint.

I believe Gemini is Websockets? I have the same experience with heavy/custom applications that try to roll their own media stuff.

You run into issues around AudioContext and resumption etc... it's a PITA to have to handle all those corner cases :(

What platforms were you targeting that you found it painful! Sorry it was frustrating.

I hope it’s getting better with education/more libraries. It’s also amazing how easy Codex etc… can burn through it now

I am excited for VAD to go away. PersonaPlex totally seems like the future.

However things like 'Call center helpline' turn based actually seems better! You don't want to be interrupted when giving information back and forth (I think?)

Thanks for reading it!

You can't beat Websockets :) Especially since you have so much tooling/existing stuff that works with HTTP.

I have been trying to get a website off the ground that does Datachannels + SQlite in the browser and then users sync between each other. I have gotten distracted so many times though.

Thanks for using it :)

I think It’s a case of you improve what you own. The owners of WebRTC servers were aggressively improving their part. They don’t own the inference servers.

Did you use libwebrtc on the backend? When you say `libwebrtc` is the only game in town are you talking about clients or servers?

Even for clients you have things like libpeer that libwebrtc can't hit.

Check out [0]. You can do 'Voice AI' on small/cheap hardware. It's the most fun you can have in the space ATM :) It's been a while, but posted a demo here [1]

[0] https://github.com/pipecat-ai/pipecat-esp32

[1] https://www.youtube.com/watch?v=6f0sUEUuruw

It doesn't today, but you could with sometime like this [0]. You can save/suspend all WebRTC state and bring it back with the next process.

[0] https://github.com/pion/webrtc-zero-downtime-restart

Very grateful that OpenAI published the article/publicized their usage of Pion[0] a library I work on. If you aren't familiar with WebRTC it's a super fun space. I work on a book WebRTC for the Curious [1] that details how it works.

[0] https://github.com/pion/webrtc

[1] https://webrtcforthecurious.com

What USB Tuners do people like? I have a Hauppauge WinTV-dualHD Dual Tuner - 955D and I have to restart server every 3 days because it deadlocks.

Do any alternatives to TvHeadEnd exist? I looked a bit and seems like it’s the best. So much customization it would be hard to reach it’s quality

use WebRTC and deploy selective forwarding units, which are going to be something custom

Would you mind explaining more? If you are doing WHIP/WHEP you should be able to drop in Broadcast Box/MediaMTX etc... and switch out servers and no one should notice. You can use browser/mobile/ffmpeg/OBS etc... get the same behavior. I care a lot about the broadcast space, want to learn about other problems.

subtly speed up audio/video to keep everything in sync

You can use https://webrtc.googlesource.com/src/+/refs/heads/main/docs/n... to add more delay (if you want to force more buffering). Or if you don't link the media together (via MediaStream) you don't get the behavior you describe either!

capture each participant's audio individually

That's a neat problem. I haven't solved this one myself, I wonder if it's easier with RtpTransport or insertable streams?

Huge fan of BigBlueButton! It’s been cool to watch the project go through multiple big tech changes and still keep going.

Never give up! Free/Open Source software especially in schools is so important