kwindla

We started getting customer pings at 6:15 PT. This is impacting some large ISPs (AT&T) in the US and globally.

This is the topic of almost every conversation I'm having with friends who are programmers right now. The compiler analogy (we basically never read compiler output) is the thing I keep thinking about.

Yes, but also ... the analogy to assembly is pretty good. We're moving pretty quickly towards a world where we will almost never read the code.

You may read all the assembly that your compiler produces. (Which, awesome! Sounds like you have a fun job.) But I don't. I know how to read assembly and occasionally do it. But I do it rarely enough that I have to re-learn a bunch of stuff to solve the hairy bug or learn the interesting system-level thing that I'm trying to track down if I'm reading the output of the compiler. And mostly even when I have a bug down at the level where reading assembly might help, I'm using other tools at one or two removes to understand the code at that level.

I think it's pretty clear that "reading the code" is going to go the way of reading compiler output. And quite quickly. Even for critical production systems. LLMs are getting better at writing code very fast, and there's no obvious reason we'll hit a ceiling on that progress any time soon.

In a world where the LLMs are not just pretty good at writing some kinds of code, but very good at writing almost all kinds of code, it will be the same kind of waste of time to read source code as it is, today, to read assembly code.

Happy to answer questions about this (or work with people on further optimizing the open source inference code here). NVIDIA has more inference tooling coming, but it's also fun to hack on the PyTorch/etc stuff they've released so far.

The Twilio integration made me smile. (For context, the default way to use this is a WebRTC connection from your mobile device to talk to Claude Code, but any network transport supported by Pipecat works.)

I'm trying to come up with a use case where I need to call my Claude on the phone, rather than connect to it in a more modern way.

I'm a big Gatsby fan and recommend this recent Wesley Morris podcast about the book.

https://podcasts.apple.com/us/podcast/our-last-chance-to-tal...

Morris has written some of my favorite long-form New York Times pieces, ("Willie Nelson's Long Encore", "Aretha Franklin Had Power. Did We Truly Respect It?", ), and he has the novelist Min Jin Lee and Gilbert Cruz, editor of The New York Times Book Review, as guests on the podcast to talk about why they regularly re-read Gatsby.

Min Jin Lee talks about how amazing the craft of the novel is (beyond the obvious greatness of the sentence-by-sentence writing).

A couple of months ago I was in New York and found a new cyberpunk re-telling of Gatsby called Local Heavens. A fun read if you like Fitzgerald, or re-imaginings of famous novels, or cyberpunk.

https://kmfajardo.com/local-heavens/

It depends on what you mean by replacing.

The integrated developer experience is much better on Vapi, etc.

The goal of the Pipecat project is to provide state of the art building blocks if you want to control every part of the multimodal, realtime agent processing flow and tech stack. There are thousands of companies with Pipecat voice agents deployed at scale in production, including some of the world's largest e-commerce, financial services, and healthtech companies. The Smart Turn model benchmarks better than any of the proprietary turn detection models. Companies like Modal have great info about how to build agents with sub-second voice-to-voice latency.[1] Most of the next-generation video avatar companies are building on Pipecat.[2] NVIDIA built the ACE Controller robot operating system on Pipecat.[3]

[1] https://modal.com/blog/low-latency-voice-bot - [2] https://lemonslice.com/ = [3] https://github.com/NVIDIA/ace-controller/

One easy way to build voice agents and connect them to Twilio is the Pipecat open source framework. Pipecat supports a wide variety of network transports, including the Twilio MediaStream WebSocket protocol so you don't have to bounce through a SIP server. Here's a getting started doc.[1]

(If you do need SIP, this Asterisk project looks really great.)

Pipecat has 90 or so integrations with all the models/services people use for voice AI these days. NVIDIA, AWS, all the foundation labs, all the voice AI labs, most of the video AI labs, and lots of other people use/contribute to Pipecat. And there's lots of interesting stuff in the ecosystem, like the open source, open data, open training code Smart Turn audio turn detection model [2], and the Pipecat Flows state machine library [3].

[1] https://docs.pipecat.ai/guides/telephony/twilio-websockets [2] - https://github.com/pipecat-ai/pipecat-flows/ [3] - https://github.com/pipecat-ai/smart-turn

Disclaimer: I spend a lot of my time working on Pipecat. Also writing about both voice AI in general and Pipecat in particular. For example: https://voiceaiandvoiceagents.com/

Sounds like it. Dude you can be honest here.

I'm going to politely weigh in here and say things Sean won't say about himself.

You're talking to someone who has spent the last ten years building open source WebRTC software that many, many, many people use and that he's never tried to commercialize. He works tirelessly to make the Pion community welcoming to everyone, from engineers with a ton of networking/video experience to brand new contributors. He wrote the guide that should be everyone's first read about WebRTC.[] All of it as a labor of love.

He's being honest.

https://webrtcforthecurious.com/

I honestly can't tell if this is trolling. LEGO bricks are pretty new technology, in the scheme of things. The original LEGO company "binding brick" was created in the late 1940s.

Of course you don't "need" an LLM to have a great toy. You also don't "need" injection-molded plastic. But if you have access to one or both, that can be pretty great!

Source: I wrote the spec for the first version of the LEGO Mindstorms programming language. These days I build a lot of voice+LLM stuff, some of it for big companies, some of it for myself and my kid.

I've done a fair amount of fine-tuning for conversational voice use cases. Smaller models can do a really good job on a few things: routing to bigger models, constrained scenarios (think ordering food items from a specific and known menu), and focused tool use.

But medium-sized and small models never hit that sweet spot between open-ended conversation and reasonably on-the-rails responsiveness to what the user has just said. We don't know yet know how to build models <100B parameters that do that, yet. Seems pretty clear that we'll get there, given the pace of improvement. But we're not there yet.

Now maybe you could argue that a kid is going to be happy with a model that you train to be relatively limited and predictable. And given that kids will talk for hours to a stuffie that doesn't talk back at all, on some level this is a fair point! But you can also argue the other side: kids are the very best open-ended conversationalists in the world. They'll take a conversation anywhere! So giving them an 8B parameter, 4-bit quantized Santa would be a shame.

I 100% agree with Sean that the computer is an exploration machine. There are lots of net positive things for kids (and non-kids) that LLMs make possible. Just like there were lots of net positive things that an Internet connection makes possible.

Of course there are things technologies can do that are bad. For kids. For adults. For societies. But I build this kind of voice+LLM stuff, too, and have a kid, and the exploration, play, and learning opportunities here are really, really amazing.

For example, we are within reach of giving every child in the world a personalized, infinitely patient tutor that can cover any subject at the right level for that child. This doesn't replace classroom teachers. It augments what you can do in school, and what kids will be able to do outside of school hours.

This repo is one possible starting point for tinkering with local agents on macOS. I've got versions of this for NVIDIA platforms but I tend to gravitate to using LLMs that are too big to fit on most NVIDIA consumer cards.

https://github.com/kwindla/macos-local-voice-agents

As someone who spends a lot of time looking at timestamped log lines to debug Pipecat pipelines, I'm a big fan of this work from Aleix.

In general, I have three pain points with debugging realtime, multi-model, multi-modal AI stuff. 1. where's the latency creeping in? 2. What context actually got passed to the models. 3. Did the model/processor get data in the format it expected.

For 1 and 3, Whisker is a big step forward. For 2, something like LangFuse (Open Telementry) is very helpful.

https://www.slate.auto/en

The configurator is fun:

https://www.slate.auto/en/personalization

In general, for realtime voice AI you don't want this model to support multiple speakers because you have a separate voice input stream for each participant in a session.

We're not doing "speaker diarization" from a single audio track, here. We're streaming the input from each participant.

If there are multiple participants in a session, we still process each stream separately either as it comes in from that user's microphone (locally) or as it arrives over the network (server-side).

I've talked about this a lot with friends.

Endpoint detection (and phrase endpointing, and end of utterance) are terms from the academic literature about this, and related, problems.

Very few people who are doing "AI Engineering" or even "Machine Learning" today know these terms. In the past, I argued that we should use the existing academic language rather than invent new terms.

But then OpenAI released the Realtime API and called this "turn detection" in their docs. And that was that. It no longer made sense to use any other verbiage.

A couple of interesting updates today:

- 100ms inference using CoreML: https://x.com/maxxrubin_/status/1897864136698347857

- An LSTM model (1/7th the size) trained on a subset of the data: https://github.com/pipecat-ai/smart-turn/issues/1

It takes about 45 minutes to do the current training run on an L4 GPU with these settings:

    # Training parameters
    "learning_rate": 5e-5,
    "num_epochs": 10,
    "train_batch_size": 12,
    "eval_batch_size": 32,
    "warmup_ratio": 0.2,
    "weight_decay": 0.05,

    # Evaluation parameters
    "eval_steps": 50,
    "save_steps": 50,
    "logging_steps": 5,

    # Model architecture parameters
    "num_frozen_layers": 20

The current data set size is ~8,000 samples.

Turn detection is deciding when a person has finished talking and expects the other party in a conversation to respond. In this case, the other party in the conversation is an LLM!

Can you say more? There's not much open source work in this domain, that I've been able to find.

I'm particularly interested in architecture variations, approaches to the classification head design and loss function, etc.

580M parameters. More info about the model architecture: https://github.com/pipecat-ai/smart-turn?tab=readme-ov-file#...

The Multimodal Live API is free while the model/API is in preview. My guess is that they will be pretty aggressive with pricing when it's in GA, given the 1.5 Flash multimodal pricing.

If you're interested in this stuff, here's a full chat app for the new Gemini 2 API's with text, audio, image, camera video and screen video. This shows how to use both the WebSocket API and to route through WebRTC infrastructure.

https://github.com/pipecat-ai/gemini-multimodal-live-demo

We've helped a number of Pipecat users hook into a variety of content moderation systems or use LLMs as judges.

The most common approach is to use a `ParallelPipeline` to evaluate the output of the LLM at the same time as the TTS inference is running, then to cancel the output and call a function if a moderation condition is triggered.

Other people have written custom frame processors to make use of the content moderation scoring in the Google and Azure APIs.

If you're interested in building a Pipecat integration for your employer's tech, happy to support that. Feel free to DM me on Twitter.

There's a really nice implementation of phrase endpointing here:

  https://github.com/pipecat-ai/pipecat/blob/d378e699d23029e8ca7cea7fb675577becd5ebfb/src/pipecat/vad/vad_analyzer.py

Silence isn't literally silence -- or shouldn't be. Any "voice activity detection" library can be plugged into this code. Most people use Silero VAD. Silence is "non-speech" time.

Speech confidence also can come from either the VAD or another model (like a model providing transcription, or an LLM doing native audio input).

Audio level should be relative to background noise, as in this code. The VAD model should actually be pretty good at factoring out non-speech background noise, so the utility here is mostly speaker isolation. You want to trigger on speech end from the loudest of the simultaneous voices. (There are, of course, specialized models just for speaker isolation. The commercial ones from Krisp are quite good.)

One interesting thing about processing audio for AI phrase endpointing is that you don't actually care about human legibility. So you don't need traditional background noise reduction, in theory. Though, in practice, the way current transcription and speech models are trained, there's a lot of overlap with audio that has been recorded for humans to listen to!

Hi!

What? No. That’s crazy. (I believe you. I’ve just … never heard of giving up IP rights because you participated in a hackathon.)

This is about community and building fun things. I can’t speak for all the sponsors, but what I want is to show people the Open Source tooling we work on at Daily, and see/hear what other people interested in real-time AI are thinking about and working on.

If you're interested in low-latency, multi-modal AI, Tavus is sponsoring a hackathon Oct 19th-20th in SF. (I'm helping to organize it.) There will also be a remote track for people who aren't in SF, so feel free to sign up wherever you are in the world.

https://x.com/kwindla/status/1839767364981920246

"Generative" AI/ML is moving so fast in so many directions that keeping up is a challenge even if you're trying really hard to stay current!

I'm part of a team building developer tools for real-time AI use cases (voice and video). I feel like I have three overlapping perspectives and goals re this new stuff:

1. To figure out what we should build I need to have a good understanding of what's possible and useful right now.

2. I talk to our customers a lot. Helping them understand what's possible and useful today (and what that might look like six months or a year from now) is part of my job.

3. I think this is a step-function change in what computers are good at, and that's really exciting and intellectually interesting.

My AI information diet right now is a few podcasts, twitter, and email newsletters. A few links:

  - Latent space podcast and newsletter: https://www.latent.space/podcast
  - Ben's Bites newsletter: https://news.bensbites.com/
  - Ethan Mollick newsletter: https://www.oneusefulthing.org/
  - Zvi Mowshowitz newsletter: https://thezvi.substack.com/
  - Rohan Paul twitter: https://x.com/rohanpaul_ai

You are right! Thank you. I went back and looked at actual benchmark numbers from a couple of years ago and the numbers I got were ~26ms one-way. I rounded up to 30 to be conservative, but then double-counted in the table above. Will fix in the technical write-up. I don't think I can edit the Show HN.