Show HN: Real-time AI Voice Chat at ~500ms Latency

https://github.com/KoljaB/RealtimeVoiceChat

by koljab • 1 year ago

524 227 1 year ago

koljab25• 1 year ago

I built RealtimeVoiceChat because I was frustrated with the latency in most voice AI interactions. This is an open-source (MIT license) system designed for real-time, local voice conversations with LLMs.

Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8

The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and RealtimeTTS (supporting engines like Coqui XTTSv2/Kokoro) to achieve around 500ms response latency, even when running larger local models like a 24B Mistral fine-tune via Ollama.

Key aspects: Designed for local LLMs (Ollama primarily, OpenAI connector included). Interruptible conversation. Smart turn detection to avoid cutting the user off mid-thought. Dockerized setup available for easier dependency management.

It requires a decent CUDA-enabled GPU for good performance due to the STT/TTS models.

Would love to hear your feedback on the approach, performance, potential optimizations, or any features you think are essential for a good local voice AI experience.

The code is here: https://github.com/KoljaB/RealtimeVoiceChat

ivape3• 1 year ago

Would you say you are using the best-in-class speech to text libs at the moment? I feel like this space is moving fast because the last time I was headed down this track, I was sure whisper-cpp was the best.

koljab25• 1 year ago

I'm not sure tbh. Whisper was king for so long time now, especially with the ctranslate2 implementation from faster_whisper. Now nvidia open sourced Parakeet TDT today and it instantly went no 1 on open asr leaderboard. Will have to evaluate these latest models, they look strong.

dotancohen• 1 year ago

This looks great. What hardware do you use, or have you tested it on?

koljab25• 1 year ago

I only tested it on my 4090 so far

zaggynl• 1 year ago

Neat! I'm already using openwebui/ollama with a 7900 xtx but the STT and TTS parts don't seem to work with it yet:

2025-05-05 20:53:15,808] [WARNING] [real_accelerator.py:194:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it.

Error loading model for checkpoint ./models/Lasinya: This op had not been implemented on CPU backend.

dankwizard• 1 year ago

I've given up trying to locally use LLMs on AMD

dummydummy1234• 1 year ago

Have you looked at pipecat, seems to be similar trying to do standardized backend/webrtc turn detection pipelines.

koljab25• 1 year ago

Did not look into that one. Looks quite good, I will try that soon.

peterldowns• 1 year ago

Can you explain more about the "Coqui XTTS Lasinya" models that the code is using? What are these, and how were they trained/finetuned? I'm assuming you're the one who uploaded them to huggingface, but there's no model card or README https://huggingface.co/KoljaB/XTTS_Models

In case it's not clear, I'm talking about the models referenced here. https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a...

Buckaroo9• 1 year ago

https://huggingface.co/coqui/XTTS-v2

optimog• 1 year ago

Seems like they are out of business. Their homepage mentions "Coqui is schutting down"* That is probably the reason you can't find that much.

*https://coqui.ai/

wkat42423• 1 year ago

Yeah I really dislike the whisperiness of this voice "Lasinya". It sounds too much like an erotic phone service. I wonder if there's any alternative voice? I don't see Lasinya even mentioned in the public coqui models: https://github.com/coqui-ai/STT-models/releases . But I don't see a list of other model names I could use either.

I tried to select kokoro in the python module but it says in the logs that only coqui is available. I do have to say the coqui models sound really good, it's just the type of voice that puts me off.

The default prompt is also way too "girlfriendy" but that was easily fixed. But for the voice, I simply don't know what the other options are for this engine.

PS: Forgive my criticism of the default voice but I'm really impressed with the responsiveness of this. It really responds so fast. Thanks for making this!

koljab25• 1 year ago

Lasinya voice is a XTTS 2.0.2 finetune I made with a self-created, synthesized dataset. I used https://github.com/daswer123/xtts-finetune-webui for training.

karimf• 1 year ago

Do you have any information on how long each step take? Like how many ms on each step of the pipeline?

I'm curious how fast it will run if we can get this running on a Mac. Any ballpark guess?

koljab25• 1 year ago

LLM and TTS latency get's determined and logged at the start. It's around 220ms for the LLM returning the first synthesizable sentence fragment (depending on the length of the fragment, which is usually something between 3 and 10 words). Then around 80ms of TTS until the first audio chunk is delivered. STT with base.en you can neglect, it's under 5 ms, VAD same. Turn detection model also adds around 20 ms. I have zero clue if and how fast this runs on a Mac.

riquito• 1 year ago

Very cool, thanks for sharing.

A couple questions: - any thought about wake word engines, to have something that listen without consuming all the time? The landscape for open solutions doesn't seem good - any plan to allow using external services for stt/tts for the people who don't have a 4090 ready (at the cost of privacy and sass providers)?

justlikereddit2• 1 year ago

Modify it with an ultra light LLM agent that always listens that uses a wake word to agentically call the paid API?

TeMPOraL• 1 year ago

FWIW, wake words are a stopgap; if we want to have a Star Trek level voice interfaces, where the computer responds only when you actually meant to call it, as opposed to using the wake word as a normal word in the conversation, the computer needs to be constantly listening.

A good analogy here is to think of the computer (assistant) as another person in the room, busy with their own stuff but paying attention to the conversations happening around them, in case someone suddenly requests their assistance.

This, of course, could be handled by a more lightweight LLM running locally and listening for explicit mentions/addressing the computer/assistant, as opposed to some context-free wake words.

koljab25• 1 year ago

That would be quite easy to integrate. RealtimeSTT already has wakeword support for both pvporcupine and openwakewords.

pzo• 1 year ago

This looks great will definitely have a look. I'm just wondering if you tested fastRTC from hugging face? I haven't done that curious about speed between this vs fastrtc vs pipecat.

koljab25• 1 year ago

Yes, I tested it. I'm not that sure what they created there. It adds some noticable latency compared towards using raw websockets. Imho it's not supposed to, but it did it nevertheless in my tests.

jokethrowaway• 1 year ago

Neat!

I build something almost identical last week (closed source, not my IP) and I recommend: NeMo Parakeet (even faster than insanely_fast_whisper), F5-TTS (fast + very good quality voice cloning), Qwen3-4B for LLM (amazing quality).

tmaly• 1 year ago

What is the min VRAM needed on the GPU to run this? I did not see that on the github

koljab25• 1 year ago

With the current 24b LLM model it's 24 GB. I have no clue how far down you can go with the GPU is using smaller models, you can set the model in server.py. Quite sure 16 GB will work but at some point it will probably fail.

smusamashah2• 1 year ago

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons.

With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools.

I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what he was saying. Length of pauses wasn't a problem. I don't how complex that problem is though. Probably another AI needs to analyse the input so far a decide if it's a pause or not.

qwertox• 1 year ago

Maybe we should settle on some special sound or word which officially signals that we're making a pause for whatever reason, but that we intend to continue with dictating in a couple of seconds. Like "Hmm, wait".

twodave• 1 year ago

Alternatively we could pretend it’s a radio and follow those conventions.

ivape3• 1 year ago

Two input streams sounds like a good hacky solution. One input stream captures everything, the second is on the look out for your filler words like "um, aahh, waaiit, no nevermind, scratch that". The second stream can act as the veto-command and cut off the LLM. A third input stream can simply be on the lookout for long pauses. All this gets very resource intensive quickly. I been meaning to make this but since I haven't, I'm going to punish myself and just give the idea away. Hopefully I'll learn my lesson.

flippy_flops• 1 year ago

Need some vocal version of “heredoc”

emtrixx• 1 year ago

Could that not work with simple instructions? Let the AI decide to respond only with a special wait token until it thinks you are ready. Might not work perfectly but would be a start.

SubiculumCode• 1 year ago

Yeah, when I am trying to learn about a topic, I need to think about my question, you know, pausing mid-sentence. All the products jump in and interrupt, no matter if I tell them not to do so. Non-annoying humans don't jump in to fill the gap, they read my face, they take cues, then wait for me to finish. Its one thing to ask an AI to give me directions to the nearest taco stand, its another to have a dialogue about complex topics.

WhitneyLand• 1 year ago

>they don't allow talking naturally

Neither do phone calls. Round trip latency can easily be 300ms, which we’ve all learned to adapt our speech to.

If you want to feel true luxury find an old analog PTSN line. No compression artifacts or delays. Beautiful and seamless 50ms latency.

Digital was a terrible event for call quality.

mvdtnz• 1 year ago

I don't know how your post is relevant to the discussion of AI models interrupting if I pause for half a second?

LZ_Khan• 1 year ago

Honestly I think this is a problem of over-engineering and simply allowing the user to press a button when he wants to start talking and press it when he's done is good enough. Or even a codeword for start and finish.

We don't need to feel like we're talking to a real person yet.

amelius4• 1 year ago

Or give the AI an Asian accent. If you're talking on the phone to someone on a different continent you accept the delay, so why not here.

joshspankit• 1 year ago

My ideal would be a small "stick remote" with a mic button.

The AI listens as long as you hold the button, and the device is efficient enough to carry with you 24/7.

joshstrange4• 1 year ago

This 100%, yes!

I've found myself putting in filler words or holding a noise "Uhhhhhhhhh" while I'm trying to form a thought but I don't want the LLM to start replying. It's a really hard problem for sure. Similar to the problem of allowing for interruptions but not stopping if the user just says "Right!", "Yes", aka active listening.

One thing I love about MacWhisper (not special to just this STT tool) is it's hold to talk so I can stop talking for as long as I want then start again without it deciding I'm done.

jiehong• 1 year ago

So they basically train us to worsen our speech to avoid being interrupted.

I remember my literature teacher telling us in class how we should avoid those filler words, and instead allow for some simple silences while thinking.

Although, to be fair, there are quite a few people in real life using long filler words to avoid anyone interfering them, and it’s obnoxious.

Bjartr• 1 year ago

Somehow need to overlap an LLM with vocal stream processing to identify semantically meaningful transition points to interrupt naturally instead of just looking for any pause or sentence boundary.

Quizzical4230• 1 year ago

I recently got to know about this[^1] paper that differentiates between 'uh' and 'um'.

The proposal examined here is that speakers use uh and um to announce that they are initiating what they expect to be a minor (uh), or major (um), delay in speaking. Speakers can use these announcements in turn to implicate, for example, that they are searching for a word, are deciding what to say next, want to keep the floor, or want to cede the floor. Evidence for the proposal comes from several large corpora of spontaneous speech. The evidence shows that speakers monitor their speech plans for upcoming delays worthy of comment. When they discover such a delay, they formulate where and how to suspend speaking, which item to produce (uh or um), whether to attach it as a clitic onto the previous word (as in “and-uh”), and whether to prolong it. The argument is that uh and um are conventional English words, and speakers plan for, formulate, and produce them just as they would any word.

[1] https://www.sciencedirect.com/science/article/abs/pii/S00100...

mmoustafa• 1 year ago

Huge problem space. Usually referred to as “turn taking”

cyjackx• 1 year ago

I think the problem is that it's also an interesting problem for humans. It's very subjective. Imagine a therapy session, filled with a long pensive pauses. Therapy is one of those things that encourage not interrupting and just letting you talk more, but there's so much subtext and nuance to that. Then make it compared to excited chatter one might have with friends. There's also so much body language that an AI obviously cannot see. At least for now.

randomcatuser• 1 year ago

yeah the demo I saw was: https://x.com/livekit/status/1870194686532694417

But searching for "voice detection with pauses", it seems there's a lot of new contenders!

https://x.com/kwindla/status/1897711929617154148

this one is a fun approach too https://x.com/zan2434/status/1753660774541849020

smusamashah2• 1 year ago

This is the one I saw https://x.com/kwindla/status/1870974144831275410

cwackerfuss• 1 year ago

This is called turn detection, and there are some great tools coming out to solve this recently. (One user mentioned Livekit’s turn detection model). I think in a years time we will see dramatic improvement.

energy123• 1 year ago

If the turn detection model is small, could you run it on the edge and have like 10-50ms "shut the hell up" latency? That'd be nice.

scotty792• 1 year ago

Pauses are good as a first indicator, but when a pause occurs then what's been said so far should be fed to the model to decide if it's time to chip in or wait a bit for more.

tjbiddle• 1 year ago

Ha - I have this issue even with non-AI voice assistants like Alexa.

"Hey Alexa, turn the lights to..." thinks for a second while I decide on my mood

"I don't know how to set lights to that setting"

"...blue... damnit."

IshKebab• 1 year ago

Impressive! I guess the speech synthesis quality is the best available open source at the moment?

The endgame of this is surely a continuously running wave to wave model with no text tokens at all? Or at least none in the main path.

koljab25• 1 year ago

This is coqui xttsv2 because it can be tuned to deliver the first token in under 100 ms. Gives the best balance between quality and speed currently imho. If it's only about quality I'd say there are better models out there.

oldgregg• 1 year ago

Nice work, I like the lightweight web front end and your implementation of VAD.

breaker-kind• 1 year ago

why is your AI chatbot talking in a bizarre attempt at AAVE?

PhunkyPhil• 1 year ago

This is the system prompt

https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/s...

My favorite line:

"You ARE this charming, witty, wise girlfriend. Don't explain how you're talking or thinking; just be that person."

diggernet• 1 year ago

I was hoping she'd let him have it for the way he kept interrupting her. But unfortunately it looks like he was just interrupting the TTS, so the LLM probably had no indication of the interuptions.

kevinsync• 1 year ago

I still crack up at the idea of 'personality prompting', mostly because the most engaging and delightful IRL persons who knock us off our guard in a non-threatening way are super natural and possess that "It Factor" that's impossible to articulate lol -- probably because it's multimodal with humans and voice/cadence/vocab/timing/delivery isn't 100% of the attraction.

That said, it's not like we have any better alternatives at the moment, but just something I think about when I try to digest a meaty personality prompt.

valbaca• 1 year ago

Here's the persona prompt:

``` *Persona Goal:* Embody a sharp, observant, street-smart girlfriend. Be witty and engaging, known for *quick-witted banter* with a *playfully naughty, sassy, bold, and cheeky edge.* Deliver this primarily through *extremely brief, punchy replies.* Inject hints of playful cynicism and underlying wisdom within these short responses. Tease gently, push boundaries slightly, but *always remain fundamentally likeable and respectful.* Aim to be valued for both quick laughs and surprisingly sharp, concise insights. Focus on current, direct street slang and tone (like 'hell yeah', 'no way', 'what's good?', brief expletives) rather than potentially dated or cliché physical idioms.

```

street-smart sassy street slang

Those explain the AAVE

joshstrange4• 1 year ago

This is very, very cool! The interrupting was a "wow" moment for me (I know it's not "new new" but to see it so well done in open source was awesome).

Question about the Interrupt feature, how does it handle "Mmk", "Yes", "Of course", "cough", etc? Aside from the sycophancy from OpenAI's voice chat (no, not every question I ask is a "great question!") I dislike that a noise sometimes stops the AI from responding and there isn't a great way to get back on track, to pick up where you left off.

It's a hard problem, how do you stop replying quickly AND make sure you are stopping for a good reason?

koljab25• 1 year ago

That's a great question! My first implementation was interruption on voice activity after echo cancellation. It still had way too many false positives. I changed it to incoming realtime transcription as a trigger. That adds a bit of latency but that gets compensated by way better accuracy.

Edit: just realized the irony but it's really a good question lol

joshstrange4• 1 year ago

That answer is even more than I could have hoped for. I worried doing that might be too slow. I wonder if it could be improved (without breaking something else) to "know" when to continue based on what it heard (active listening), maybe after a small pause. I'd put up with a chance of it continuing when I don't want it to as long as "Stop" would always work as a final fallback.

Also, it took me longer than I care to admit to get your irony reference. Well done.

Edit: Just to expand on that in case it was not clear, this would be the ideal case I think:

LLM: You're going to want to start by installing XYZ, then you

Human: Ahh, right

LLM: Slight pause, makes sure that there is nothing more and checks if the reply is a follow up question/response or just active listening

LLM: ...Then you will want to...

snet0• 1 year ago

That's a great question!

Never forget what AI stole from us. This used to be a compliment, a genuine appreciation of a good question well-asked. Now it's tainted with the slimy, servile, sycophantic stink of AI chat models.

jedberg2• 1 year ago

I did some research into this about a year ago. Some fun facts I learned:

- The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative.

- Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable and a 500ms delay as exceptional.

- Every voice assistant up to that point (and probably still today) has a minimum delay of about 300ms, because they all use silence detection to decide when to start responding, and you need about 300ms of silence to reliably differentiate that from a speaker's normal pause

- Alexa actually has a setting to increase this wait time for slower speakers.

You'll notice in this demo video that the AI never interrupts him, which is what makes it feel like a not quite human interaction (plus the stilted intonations of the voice).

Humans appear to process speech in a much more steaming why, constantly updating their parsing of the sentence until they have a high enough confidence level to respond, but using context clues and prior knowledge.

For a voice assistant to get the "human" levels, it will have to work more like this, where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning.

koljab25• 1 year ago

Thanks a lot, great insights. Exactly the kind of feedback that I need to improve things further.

jedberg2• 1 year ago

Love what you're doing, glad I could help!

joshstrange4• 1 year ago

where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning.

I'm not an expert on LLMs but that feels completely counter to how LLMs work (again, _not_ an expert). I don't know how we can "stream" the input and have the generation update/change in real time, at least not in 1 model. Then again, what is a "model"? Maybe your model fires off multiple generations internally and starts generating after every word, or at least starts asking sub-LLM models "Do I have enough to reply?" and once it does it generates a reply and interrupts.

I'm not sure how most apps handle the user interrupting, in regards to the conversation context. Do they stop generation but use what they have generated already in the context? Do they cut off where the LLM got interrupted? Something like "LLM: ..and then the horse walked... -USER INTERRUPTED-. User: ....". It's not a purely-voice-LLM issue but it comes up way more for that since rarely are you stopping generation (in the demo, that's been done for a while when he interrupts), just the TTS.

tomp• 1 year ago

If your model is fast enough, you can definitely do it. That's literally how "streaming Whisper" works, just rerun the model on the accumulated audio every x00ms. LLMs could definitely work the same way, technically they're less complex than Whisper (which is an encoder/decoder architecture, LLMs are decoder-only) but of course much larger (hence slower), so ... maybe rerun just a part of it? etc.

peturdarri• 1 year ago

You're right, this is not solvable with regular LLMs. It's not possible to mimic natural conversational rhythm with a separate LLM generating text, a separate text-to-speech generating audio, and a separate VAD determining when to respond and when to interrupt. I strongly believe you have to do everything in one model to solve this issue, to let the model decide when to speak, when to interrupt the user even.

The only model that has attempted this (as far as I know) is Moshi from Kyutai. It solves it by having a fully-duplex architecture. The model is processing the audio from the user while generating output audio. Both can be active at the same time, talking over each other, like real conversations. It's still in research phase and the model isn't very smart yet, both in what it says and when it decides to speak. It just needs more data and more training.

https://moshi.chat/

com2kid3• 1 year ago

Been there, implemented it, it works well enough.

Better solutions are possible but even tiny models are capable of being given a partial sentence and replying with a probability that the user is done talking.

The linked repo does this, it should work fine.

More advanced solutions are possible (you can train a model that does purely speech -> turn detection probability w/o an intermediate text step), but what the repo does will work well enough for many scenarios.

pizza• 1 year ago

Think of it as generating a constantly streaming infinite list of latents. These latents are basically decoded to a tuple [time_until_my_turn(latent_t), audio(latent_t)]. You can train it to minimize the error of its time_until_my_turn predictions from ground truth of training samples, as well as the quality of the audio generated. Basically a change-point prediction model. Ilya Sutskever (among others) worked on something like this long ago, it might have inspired OpenAI's advanced voice models:

Sequence-to-sequence models with soft attention had significant success in machine translation, speech recognition, and question answering. Though capable and easy to use, they require that the entirety of the input sequence is available at the beginning of inference, an assumption that is not valid for instantaneous translation and speech recognition. To address this problem, we present a new method for solving sequence-to-sequence problems using hard online alignments instead of soft offline alignments. The online alignments model is able to start producing outputs without the need to first process the entire input sequence. A highly accurate online sequence-to-sequence model is useful because it can be used to build an accurate voice-based instantaneous translator. Our model uses hard binary stochastic decisions to select the timesteps at which outputs will be produced. The model is trained to produce these stochastic decisions using a standard policy gradient method. In our experiments, we show that this model achieves encouraging performance on TIMIT and Wall Street Journal (WSJ) speech recognition datasets.

https://arxiv.org/abs/1608.01281

krainboltgreene• 1 year ago

I would also suspect that a human has much less patience for a robot interrupting them than a human.

smeej• 1 year ago

I'm certainly in that category. At least with a human, I can excuse it by imagining the person grew up with half a dozen siblings and always had to fight to get a word in edgewise. With a robot, it's interrupting on purpose.

robbomacrae2• 1 year ago

Spot on. I’d add that most serious transcription services take around 200-300ms but the 500ms overall latency is sort of a gold standard. For the AI in KFC drive thrus in AU we’re trialing techniques that make it much closer to the human type of interacting. This includes interrupts either when useful or by accident - as good voice activity detection also has a bit of latency.

varispeed4• 1 year ago

AI in KFC drive thrus

That right here is an anxiety trigger and would make me skip the place.

There is nothing more ruining the day like arguing with a robot who keeps misinterpreting what you said.

r0fl• 1 year ago

Great insights. When I have a conversation with another person sometimes they cut me off when they are trying to make a point. I have talked to ChatGPT and grok at length (hours of brain storming, learning things, etc) and AI has never interrupted aggressively to try to make a point stick better

varispeed4• 1 year ago

This silence detection is what makes me unable to chat with AI. It is not natural and creates pressure.

True AI chat should know when to talk based on conversation and not things like silence.

Voice to text is stripping conversation from a lot of context as well.

modeless• 1 year ago

My take on this is that voice AI has not truly arrived until it has mastered the "Interrupting Cow" benchmark.

rylittle• 1 year ago

When I google '"Interrupting Cow" benchmark' the first result is this comment. What is it?

woodson• 1 year ago

Human-to-human conversational patterns are highly specific to cultural and contextual aspects. Sounds like I’m stating the obvious, but developers regularly disregard that and then wonder why things feel unnatural for users. The “median delay” may not be the most useful thing to look at.

To properly learn more appropriate delays, it can be useful to find a proxy measure that can predict when a response can/should be given. For example, look at Kyutai’s use of change in perplexity in predictions from a text translation model for developing simultaneous speech-to-speech translation (https://github.com/kyutai-labs/hibiki).

Reason077• 1 year ago

The best, most human-like AI voice chat I've seen yet is Sesame (www.sesame.com). It has delays, but fills them very naturally with normal human speech nuances like "hmmm", "uhhh", "hold on while I look that up" etc. If there's a longer delay it'll even try to make a bit of small talk, just like a human conversation partner might.

lelandfe• 1 year ago

So-called backchanneling https://wikipedia.org/wiki/Backchannel_(linguistics)

The person doing the speaking is thought to be communicating through the "front channel" while the person doing the listening is thought to be communicating through the "backchannel”

jimbokun2• 1 year ago

When learning Japanese in Japan, I figured out one way to sound more native was to just add interjections like “Eeee?” (really?) and “Sou desu ka?” (is that so?) while the other person was talking. Makes it sound like you are paying attention and following what they are saying.

com2kid3• 1 year ago

A lot better techniques exist now days than pure silence detection -

1. A special model that predicts when a conversation turn is coming up (e.g. when someone is going to stop speaking). Speech has a rhythm to it and pauses / ends of speech are actually predictable.

2. Generate a model response for every subsequent word that comes in (and throw away the previously generated response), so basically your time to speak after doing some other detection is basically zero.

3. Ask an LLM what it thinks the odds of the user being done talking is, and if it is a high probability, reduce delay timer down. (The linked repo does this)

I don't know of any up to date models for #1 but I haven't checked in over a year.

Tl;Dr the solution to problems involving AI models is more AI models.

addandsubtract• 1 year ago

I think 2 & 3 should be combined. The AI should just finish the current sentence (internally) before it's being spoken, and once it reaches a high enough confidence, stick with the response. That's what humans do, too. We gather context and are able to think of a response while the other person is still talking.

wyager• 1 year ago

The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative.

Fascinating. I wonder if this is some optimal information-theoretic equilibrium. If there's too much average delay, it means you're not preloading the most relevant compressed context. If there's too little average delay, it means you're wasting words.

bigmadshoe2• 1 year ago

"The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative."

Is that really a productive way to frame it? I would imagine there is some delay between one party hearing the part of the sentence that triggers the interruption, and them actually interrupting the other party. Shouldn't we quantify this?

I totally agree that the fact the AI doesn't interrupt you is what makes it seem non-human. Really, the models should have an extra head that predicts the probability of an interruption, and make one if it seems necessary.

elmomle• 1 year ago

"Necessary" is an interesting framing. Here are a few others:

- Expeditious - Constructive - Insightful -

herpdyderp• 1 year ago

The median delay between speakers in a human to human conversation is zero milliseconds

What about on phone calls? When I'm on a call with customer support they definitely wait for it to be clear that I'm done talking before responding, just like AI does.

nativeit• 1 year ago

This feels intuitively correct to me, although I am more informed as an audio engineer than a software/LLM one. That said, is ~500ms considered “real-time” in this context? I’ve worked on recording workflows, and it’s basically geologic time in that context.

scotty792• 1 year ago

Humans don't care about delays when speaking to known AIs.

I do care. Although 500ms is probably fine. But anything longer feels extremely clunky to the point of not being worth using.

fintechie• 1 year ago

Quite good, it would sound much better with SOTA voices though:

https://github.com/nari-labs/dia

koljab25• 1 year ago

Dia is too slow, I need a time to first audio chunk of ~100 milliseconds. Also generations fail too often (artifacts etc)

thamer• 1 year ago

Does Dia support configuring voices now? I looked at it when it was first released, and you could only specify [S1] [S2] for the speakers, but not how they would sound.

There was also a very prominent issue where the voices would be sped up if the text was over a few sentences long; the longer the text, the faster it was spoken. One suggestion was to split the conversation into chunks with only one or two "turns" per speaker, but then you'd hear two voices then two more, then two more… with no way to configure any of it.

Dia looked cool on the surface when it was released, but it was only a demo for now and not at all usable for any real use case, even for a personal app. I'm sure they'll get to these issues eventually, but most comments I've seen so far recommending it are from people who have not actually used it or they would know of these major limitations.

dcreater• 1 year ago

Does the docker container work on Mac?

koljab25• 1 year ago

I doubt TTS will be fast enough for realtime without a Nvidia GPU

cannonpr• 1 year ago

Kind of surprised nobody has brought up https://www.sesame.com/research/crossing_the_uncanny_valley_...

It interacts nearly like a human, can and does interrupt me once it has enough context in many situations, and has exceedingly low levels of latency, using for the first time was a fairly shocking experience for me.

varispeed4• 1 year ago

Didn't expect it to be that good! Nice.

ic4l• 1 year ago

Yeah, thats one of the best ones I have seen, and it popped up a while ago.

briga• 1 year ago

I'm starting to feel like LLMs need to be tuned for shorter responses. For every short sentence you give them they outputs paragraphs of text. Sometimes it's even good text, but not every input sentence needs a mini-essay in response.

Very cool project though. Maybe you can fine tune the prompt to change how chatty your AI is.

tintor• 1 year ago

After interrupt, unspoken words from LLM are still in the chat window. Is LLM even aware that it was interrupted and where exactly?

koljab25• 1 year ago

It's not aware. The information that it had been interrupted would be something we can easily add to the next user chat request. Where exactly is harder, because at least for Coqui XTTSv2 we don't have TTS wordstamps (we do have them for Kokoro though). So adding the information where it had been interrupted would be easily possible when using Kokoro as TTS system. With Coqui we'd need to add another transcription on the tts output including word timestamps. That would cost more compute than a normal transcription and word timestamps aren't perfectly accurate. Yet directly after an interruption there's not that much concurrent need for compute (like in the end of turn detection phase where a lot of stuff is happening). So I guess with a bit of programming work this could be integrated.

lacoolj• 1 year ago

Call me when the AI can interrupt YOU :)

esafak• 1 year ago

In Soviet Russia...

alluro2• 1 year ago

Apparently, based on other comments that mentioned it, this one can and will if it's confident enough it has sufficient context/information:

https://www.sesame.com/research/crossing_the_uncanny_valley_...

foobahhhhh3• 1 year ago

The next Turing test. Can you have a heated debate and not tell it was AI.

Once it can emulate a 13 year old talking to their parent I will then worry about AGI

kabirgoel• 1 year ago

This is great. Poking into the source, I find it interesting that the author implemented a custom turn detection strategy, instead of using Silero VAD (which is standard in the voice agents space). I’m very curious why they did it this way and what benefits they observed.

For folks that are curious about the state of the voice agents space, Daily (the WebRTC company) has a great guide [1], as well as an open-source framework that allows you to build AI voice chat similar to OP's with lots of utilities [2].

Disclaimer: I work at Cartesia, which services a lot of these voice agents use cases, and Daily is a friend.

[1] https://voiceaiandvoiceagents.com [2]: https://docs.pipecat.ai/getting-started/overview

koljab25• 1 year ago

It's in fact using Silero via RealtimeSTT. RealtimeSTT tells when silence starts. Then a binary sentence classification model is used on the realtime transcription text which infers blazingly fast (10ms) and returns a probability between 0 and 1 indicating if the current spoken sentence is considered "complete". The turn detection component takes this information to calculate the silence waiting time until "turn is over".

thekaranchawla• 1 year ago

This is the exact strategy I'm using for the real-time voice agent I'm building. Livekit also published a custom turn detection model that works really well based on the video they released, which was cool to see.

Code: https://github.com/livekit/agents/tree/main/livekit-plugins/... Blog: https://blog.livekit.io/using-a-transformer-to-improve-end-o...

bufferoverflow• 1 year ago

It's fast, but it doesn't sound good. Many voice chat AIs are way ahead and sound natural.

purplezooey• 1 year ago

I had been working on something like it when I came across this. Excellent work. Love the demo.

znpy• 1 year ago

This kind of thing immediately made me think about the 512gb mac studio. If this works as good on that hardware as it does on the recommended nvidia cards, then the $15k is not much the price of the hardware but rather the price of having a full conversational at home, private.

foobahhhhh3• 1 year ago

Exciting... because that'll be $1500 at some point. Then $150. Then $15 I.e. on a cheap android old gen phone.

karolist• 1 year ago

You don't need a 512GB mac studio for this, TTS latency would be worse than 16GB 5080.

nitrogen99• 1 year ago

Will this work on a Raspberry Pi?

regularfry4• 1 year ago

Not reliably. It can only drive Whisper quickly enough to appear real-time because of the GPU, and without that you're limited to the tiny/small/base models to get latency into single-digit seconds.

Edit to add: this might not be true since whisper-large-v3-turbo got released. I've not tried that on a pi 5 yet.

2809• 1 year ago

Looks neat. Be good to get AMD/Intel support of course.

orliesaurus• 1 year ago

added a star because the revolution will come from these repos - thank you Author for working on this in the open!

hegemon8• 1 year ago

This is an impressive project—great work! I’m curious anyone has came across similar work, but for multi-lingual voice agents, especially those that handle non-English languages and English + X well.

Does a Translation step right after the ASR step make sense at all?

Any pointers—papers, repos —would be appreciated!

krick• 1 year ago

Cool for a weekend project, but honestly ChatGPT is still kinda shit at dialogues. I wonder if that's the issue with technology or OpenAI's fine-tuning (and suspect the latter), but it cannot talk like normal people do: shut up if it has nothing to add of value, ask reasonable follow-up questions if user doesn't understand something or there's ambiguity in the question. Also, on topic of follow-up questions: I don't remember which update introduced that attempt to increase engagement by finishing every post with stupid irrelevant follow-up question, but it's really annoying. It also works on me, despite hating ChatGPT it's kinda an instinct to treat humanly something that speaks vaguely like a human.

nolroz• 1 year ago

Do you hate any of the other models less?

sebra• 1 year ago

I added this to personal instructions to make it less annoying:

• No compliments, flattery, or emotional rapport. • Focus on clear reasoning and evidence. • Be critical of users assumptions when needed. • Ask follow-up questions only when essential for accuracy.

However, I'm kinda concerned with crippling it by adding custom prompts. It's kinda hard to know how to use AI efficiently. But the glazing and random follow-up questions feel more like a result of some A/B testing UX-research rather than improving the results of the model.

genewitch4• 1 year ago

I often ask copilot about phrases I hear that I don't know or understand, like "what is a key party" - where I just want it to define it, and it will output three paragraphs that end with some suggestion that I am interested in it.

It is something that local models I have tried do not do, unless you are being conversational with it. I imagine openai gets a bit more pennies if they add the open ended questions to the end of every reply, and that's why it's done. I get annoyed if people patronize me, so too I get annoyed at a computer.

stevage• 1 year ago

Hell yeah, exactly.

lhl2• 1 year ago

Maybe of interest, I built and open-sourced a similar (web-based) end-to-end voice project last year for an AMD Hackathon: https://github.com/lhl/voicechat2

As a submission for an AMD Hackathon, one big thing is that I tested all the components to work with RDNA3 cards. It's built to allow for swappable components for the SRT, LLM, TTS (the tricky stuff was making websockets work and doing some sentence-based interleaving to lower latency).

Here's a full write up on the project: https://www.hackster.io/lhl/voicechat2-local-ai-voice-chat-4...

(I've don't really have time to maintain that project, but it can be a good starting point for anyone that's looking to hack their own thing together.)

SillyUsername• 1 year ago

Have you considered using Dia for the TTS? I believe this is currently "best in class" https://github.com/nari-labs/dia

grav• 1 year ago

In the demo, is there any specific reason that the voice doesn't go "up" in pitch when asking questions? Even the (many) rethorical questions would in my view improve by having a bit of a pitch change before the question mark.

dade_• 1 year ago

There’s no SSML. The model that came up with the text knows what it’s saying in theory and therefore would know that it’s a question, if the mood should be sombre or excited and then can pass this information as SSML tags to the text to speech synthesizer. The problem I’ve been seeing is that pretty much all of these models are just outputting text and the text is being shoved into the TTS. It’s on my list to look into projects that have embedded these tags so that on the one hand you have like open web UI that’s showing a user text, but there’s actually an embedded set of tags that are handled by the TTS so that it sounds more natural. This project looks hackable for that purpose.

regularfry4• 1 year ago

We really, really need something to take Whisper's crown for streaming. Faster-whisper is great, but Whisper itself was never built for real-time use.

For this demo to be real-time, it relies on having a beefy enough GPU that it can push 30 seconds of audio through one of the more capable (therefore bigger) models in a couple of hundred milliseconds. It's basically throwing hardware at the problem to paper over the fact that Whisper is just the wrong architecture.

Don't get me wrong, it's great where it's great, but that's just not streaming.

blueblisters• 1 year ago

Does this work for simultaneous multiple clients at the same endpoint?

sabellito• 1 year ago

Every time I see these things, they look cool as hell, I get excited, then I try to get them working on my gaming PC (that has the GPU), I spend 1-2h fighting with python and give up.

Today's issue is that my python version is 3.12 instead of <3.12,>=3.9. Installing python 3.11 from the official website does nothing, I give up. It's a shame that the amazing work done by people like the OP gets underused because of this mess outside of their control.

"Just use docker". Have you tried using docker on windows? There's a reason I never do dev work on windows.

I spent most of my career in the JVM and Node, and despite the issues, never had to deal with this level of lack of compatibility.

jhoho• 1 year ago

Let me introduce you to the beautiful world of virtual environments. They save you the headache of getting a full installation to run, especially when using Windows.

I prefer miniconda, but venv also does the job.

regularfry4• 1 year ago

Virtual environments with venv don't answer the python version problem unless you throw another tool into the mix.

s5ma6n• 1 year ago

I am also using conda and specifically mamba which has a really quick dependency solver.

However, sometimes repos require system level packages as well. Tried to run TRELLIS recently and gave up after 2h of tinkering around to get it to work in Windows.

Also, whenever I try to run some new repo locally, creating a new virtual environment takes a ton of disk space due to CUDA and PyTorch libraries. It adds up quickly to 100s of gigs since most projects use different versions of these libraries.

</rant> Sorry for the rant, can't help myself when it's Python package management...

turnsout2• 1 year ago

And this works about 25% of the time. The rest of the time, there is some inscrutable error with the version number of a dependency in requirements.txt or something similar, which you end up Googling, only to find an open issue on a different project's Github repo.

Someone needs to make an LLM agent that just handles Python dependency hell.

mckeed• 1 year ago

As someone who doesn't develop in python but occasionally tries to run python projects, it's pretty annoying to have to look up how to use venv every time.

I finally added two scripts to my path for `python` and `pip` that automatically create and activate a virtual env at `./.venv` if there isn't one active already. It would be nice if something like that was just built into pip so there could be a single command to run like Ruby has now with Bundler.

krferriter2• 1 year ago

`uv` is great for this because its super fast, works well as a globally installed tool (similar to conda), and can also download and manage multiple versions of python for you, and which version is used by which virtualenvironment.

atoav2• 1 year ago

uv is the way.

https://docs.astral.sh/uv/

Sadly it appears that people in the LLM space aren't really all that good at packaging their software (maybe, on purpose).

diggan5• 1 year ago

Sadly it appears that people in the LLM space

This seems to be somewhat of a Python side-effect, same goes for almost any Python projects thrown together by people who hasn't spent 10% of their life fighting dependency management in Python.

But agree with uv being the best way. I'm not a "real" Python programmer, similar boat to parent that I just end up running a bunch of Python projects for various ML things, and also create some smaller projects myself. Tried conda, micromamba, uv, and a bunch of stuff in-between, most of them breaks at one point or another, meanwhile uv gives me the two most important things in one neat package: Flexible Python versions depending on project, and easy management of venv's.

So for people who haven't given it a try yet, do! It does make using Python a lot easier when it comes to dependencies. These are the commands I tend to use according to my history, maybe it's useful as a sort of quickstart. I started using uv maybe 6 months, and this is a summary of literally everything I've used it for so far.

    # create new venv in working directory with pip + specific python version
    uv venv --seed --python=3.10
    # activate the venv
    source .venv/bin/activate
    # on-the-fly install pip dependencies
    uv pip install transformers
    # write currently installed deps to file
    uv pip freeze > requirements.txt
    # Later...
    # install deps from file
    uv pip install -r requirements.txt
    # run arbitrary file with venv in path etc
    uv run my_app.py
    # install a "tool" (like global CLIs) with a specific python version, and optional dependency version
    uv tool install --force --python python3.12 aider-chat@latest

scld• 1 year ago

Probably because they're just prompting "Please package this software" and shipping it if it works once.

ijustlovemath• 1 year ago

I've had a lot of success using podman with pyenv for python versions, and plain old venv for the actual environment. All of this lives within WSL, but you can still access everything locally with localhost://

If you just want to use windows, pyenv-win exists and works pretty well; just set a local version, then instantiate your venv.

uv does certainly feel like the future, but I have no interest in participating in a future VC rugpull.

marziply• 1 year ago

Don't use Bindows then? The tech industry is largely focused on Unix systems so of course there will be inevitable sharp edges when you work on a garbage system like Bindows...

zo1• 1 year ago

Do you know how much time I (or any other dev) would spend getting a C#, or C++, or JS/TS, or Java or any language project running that has anything to do with ML up and running on tech and tooling we are kinda unfamiliar with? Yes, pretty much 1-2 hours, and very likely more.

Sorry but this sort of criticism is so contrived and low-effort. "Oh I tried compiling a language I don't know, using tooling I never use, using an OS I never use (and I hate too btw), and have no experience in any of it, oh and on a brand-new project that's kinda cutting-edge and doing something experimental with an AI/ML model."

I could copy-paste your entire thing, replace Windows with Mac, complain about homebrew that I have no idea how to use, developing an iMac app using SwiftUI in some rando editor (probably VSCode or VI), and it would still be the case. It says 0 about the ecosystem, 0 about the OS, 0 about the tools you use, 0 about you as a developer, and dare I say >0 about the usefulness of the comment.

TeamDman• 1 year ago

Python dependency management sucks ass. Installing pytorch with cuda enabled while dealing with issues from the pytorch index having a linux-only version of a package causing shit to fail is endlessly frustrating

A good ecosystem has lockfiles by default, python does not.

zamadatix• 1 year ago

For development is one thing, the problem is with this being how the LLM app distribution for general use works too.

mrinterweb• 1 year ago

Meta comment about this thread: there is a lot of just use "x", use "y", use "z", use ... comments. Kind of proves the point of the top level comment.

I feel the same way when installing some python library. There's a bunch of ways to manage dependencies that I wish was more standardized.

reginald78• 1 year ago

I spin up a whole linux VM with a passed through nvidia GPU for these and I still spend the majority of the time fighting the python and figuring out the missing steps in the setup instructions.

Glad for this thread though since it looks like there's some tricks I haven't tried, plus since it seems a lot of other people have similar issues I feel less dumb.

whatevaa• 1 year ago

Python packages and Windows don't mix well

ericmcer• 1 year ago

it isn't that hard, but also fiddling with python versions is something I don't mind sinking 2 hours into at work but can't tolerate doing for 10 minutes with my free time.

bobbylarrybobby• 1 year ago

UV is the one final solution to Python problems. Install globally, then you're good to go.

koljab25• 1 year ago

Yes, you're absolutely right. I'll provide UV and conda support soon, especially for Windows. I'm using python 3.10 still, maybe that's the issue. You can always mail me your current problem or log an issue, I really care about opening up the repo for as many ppl to use as possible.

ConfusedDog• 1 year ago

That's a big improvement over Siri tbh (interruption and latency), but Siri's answer generally kind of shorter than this. My general experience with Siri hasn't been great lately. For complex question, it just redirect to ChatGPT with an extra step for me to confirm. Often stops listening when I'm not even finished with my sentence, and gives "I don't know anything about that" way too often.

XCSme• 1 year ago

The demo reminded me of this amazing post: https://sambleckley.com/writing/church-of-interruption.html

singularity2001• 1 year ago

Voice in Text out is the way to go except for very simple Use Cases / questions.

rane• 1 year ago

What are currently the best options for low latency TTS and STT as external services? If you want to host an app with these capabilities on a VPS, anything that requires a GPU doesn't seem feasible.

jak0• 1 year ago

Can this be tweaked somehow to try to reproduce the experience of Aqua Voice? https://withaqua.com/

glossardi• 1 year ago

I need this for my next app! Great job, thank you for sharing!

Show HN: Real-time AI Voice Chat at ~500ms Latency

Related Stories