akadeb

RIP legend

Yeah I sell one here https://www.elatoai.com/products/ai-devkit If you pre-order on the link I can ship you one at a discount

Thanks Boris! The case is made out of Resin but I started with PLA. Will post the open-sourced STEP files there

I built an open-source, screen-free, storytelling toy for my nephew who uses a Yoto toy. My sister told me he talks to the stories sometimes and I thought it could be cool if he could actually talk to those characters in stories with AI models (STT, LLM, TTS) running locally on her Macbook and not send the conversation transcript to cloud models.

This is my voice AI stack:

- ESP32 on Arduino to interface with the Voice AI pipeline

- mlx-audio for STT (whisper) and TTS with streaming (`qwen3-tts` / `chatterbox-turbo`)

- mlx-vlm to use vision language models like Qwen3.5-9B and Mistral

- mlx-lm to use LLMs like Qwen3, Llama3.2, Gemma3

- Secure websockets to interface with a Macbook

This repo supports inference on Apple Silicon chips (M1/2/3/4/5) but I am planning to add Windows soon. Would love to hear your thoughts on the project.

Many parents are concerned about sending their children's chat transcripts to the cloud and privacy is often the first thing that comes up when we talk about AI toys.

So I built OpenToys so anyone with an ESP32 can create their own AI Toys that run inference locally, starting with Apple Silicon chips and keep their data from leaving their home network.

The repo currently supports voice cloning and multilingual conversations in 10 languages locally. The app is a Rust Tauri app with a Python sidecar with the voice pipeline. The stack uses Whisper for STT, any MLX LLMs, Qwen3-TTS and Chatterbox-Turbo for TTS.

You could pipe the output to an audio file with ffmpeg or pyaudio and save it locally

https://www.akadeb.xyz

I would highly recommend gemini 2.5 pro too for their speech quality. It's priced lower and the quality is top notch on their API. I made an implementation here in case you're interested https://www.github.com/akdeb/ElatoAI but its on hardware so maybe not totally relevant

I like the sound of that! I think youre gonna like what we are building here https://github.com/akdeb/ElatoAI

Its as if the rubber duck was actually on the desk while youre programming and if we have an MCP that can get live access to code it could give you realtime advice.

I understand, is it the realtime conversational aspect or just in general you wouldn't want a child to play with a TTS-like service?

Murphy's law

thanks for checking it out Bert

Hi Mr. teddy bear!

Hey there buddy! Have you tried brushing with Sensodyne now available at your nearest CVS only for $9.99!

Thank you! It's been super fun to work on. The challenges were more on the ESP32 side. Like getting audio to work smoothly with Opus and the audio timing challenges. This is one of the reasons I open-sourced.

It seems pointless to think that everyone should cross that C++/Audio barrier to make something cool. Using this cuts a lot of dev time and brings products out to market wayy quicker. The repo basically helps launch your AI toy brand

The emojis are all AI. The content is a mix of me n cursor and I added the mermaid chart to make it easier to visualize the system diagram.

The circuit diagram in on figma

And demo video edited on capcut

thank you stavros!

Pipecat is awesome! is it similar to what livekit provides?

I think Realtime API adoption would be higher if it is offered on Arduino rather than ESP-IDF as the latter is not very beginner friendly. That was one of the main reasons I built this repo using edge functions instead of a direct WebRTC connection.

with limited exposure to the creative power of children.

This is true, I am not a parent. But I have some domain expertise in building a conversational toy... talking to many parents and having been a child myself for several years has helped

The Elato toy is currently not aimed at children. The current version has adult characters that are entertaining and fun to engage with like the Chad Brew Barkley character in the videos. I put up more such funny videos on my tiktok tiktok.com/@elatoai

However, while testing it with a friend who has a 5-year old daughter, I added a `Story mode` feature to create dynamic stories for her which she enjoys.

I think what would be even cooler is if each character in a story has unique voices (like voice of an ogre, voice of an elf etc.) which is currently unsupported in the single websocket connnection.

thanks David, let me know if you get a chance to try it out!

I agree, it's still pricy. The cost works out better with `gpt-4o-mini-realtime-preview-2024-12-17`.

Yep its constrained to the system prompt but I pass in conversation history with each new session to keep it relevant. It also supports tool calling which is clutch.

Have you tried Hume AI? They've got a neat suite of APIs that give you more control on each session.

The willow team has iterated fast. I think ESP-IDF is more advanced and using Arduino makes it easier for people to jump on and tinker with Speech-to-Speech AI which is why i created this repo

Currently our device is a toy accessory. And for children we are strictly focusing on `Story mode`. Where adventure stories / fairy tales feel more engaging. I think there's value in getting the AI to create epic stories consistently

Thanks for the feedback. I have attached the raw unedited video here: https://drive.google.com/file/d/1kEmbVInvUrYFwjddyGL8Rz03c0N... (sorry the video is a bit long ~5min with some intro about my company :-)

Yeah the way I am handling this is turn detection which feels unnatural. I like how Livekit handles turn detection with a small model[0][1] [0]https://www.youtube.com/watch?v=EYDrSSEP0h0 [1]https://docs.livekit.io/agents/build/turns/turn-detector/

``` turn_detection: { type: "server_vad", threshold: 0.4, prefix_padding_ms: 400, silence_duration_ms: 1000, }, ```

The long connections are ultimately handled by Deno Edge so the site isn't used there. The NextJS frontend (which also could be an iOS/Android app) helps provide an interface to select character, create AI characters, set ESP32 volume, and view conversation history.

For parents we added a `Story mode` option (similar to Yoto toy / Toniebox). The idea is: the AI crafts a story and invites the child to craft the story together in a more engaging way. The story prompt keeps the story focused and in scope.

Currently we connect to a Wifi network to reach the Deno edge server. Some popular toys doing it: Yoto, Toniebox

thank you! The nextjs frontend is to set things like device volume, selecting which character you are interacting with, viewing conversation history etc. I just tried it and for a 15 minute chat, it's roughly 20c. Roughly 570 input tokens

Saving this for my next trip