Build an AI telephony agent for inbound and outbound calls

https://github.com/videosdk-community/ai-telephony-demo

by sagarkava • 12 months ago

66 48 12 months ago

sagarkava14• 12 months ago

Hi I am Sagar, We just open-sourced a complete framework to build an AI-powered telephony agent that can handle both inbound and outbound calls—using Python, SIP, and cloud LLMs like OpenAI or Gemini.

You can use it to create smart appointment bots, voice feedback collectors, or even enterprise IVR systems. It’s modular (plug in your SIP provider or AI model), production-ready, and extensible for real-time workflows.

Features include:

SIP & VoIP call handling (Twilio, Plivo, etc)

LLM-integrated AI agent (customizable prompt & tools)

FastAPI-based server for routing and control

Plugins for STT, TTS, sentiment analysis

Support for Agent2Agent and MCP protocols

GitHub Repo:https://github.com/videosdk-live/agents Full Blog: https://www.videosdk.live/blog/ai-telephony-agent-inbound-ou...

Would love feedback from anyone working with telephony, LLMs, or real-time automation!

gerdesj2• 12 months ago

I'm still getting to grips with all this AI stuff but I do know SIP n RTP rather well. Your project looks ideal for me to play with.

My initial project will be to replace my aging parent's and my home FreePBX IVR with something a bit more useful. At the moment it requests you press 1 to make the phones ring and that drops pretty much all unwanted calls but that will soon be automatically defeated.

I do also have some rather more commercial ideas but baby steps. Mind you a RPi based drop in box with a decent auto attendant for say £100 hardware, open source software and some fettling time seems a pretty good opportunity. You could go for somewhat cheaper hardware. The UK is being migrated away from copper A and B wires to what is called SOGEA and/or FTTP. Basically its going to be all VoIP for telephony from pretty much now on. The good old days of an analogue handset that is powered by the line will mostly go away within a few years. On the bright side our signalling was bloody weird and SIP is nearly eternal!

sagarkava14• 12 months ago

That is a fantastic idea. We’ve tested it on low-resource hardware. It’s SIP-agnostic and modular, making it ideal for home or SMB setups. We would love to highlight your RPi implementation if you publish it. GitHub: https://github.com/videosdk-live/agents

methuselah_in2• 12 months ago

No offense, but how many people will be jobless because of this?

sagarkava14• 12 months ago

Totally valid concern. The goal isn’t to replace people, but to offload repetitive tasks so humans can focus on higher-value work.

We’ve also built in Human-in-the-Loop support so a person can step in anytime the AI falls short. More on that here: https://docs.videosdk.live/ai_agents/human-in-the-loop

It’s about shifting roles, not eliminating them and doing it responsibly.

methuselah_in2• 12 months ago

What you say is totally going to do opposite! In developing countries like India here people don't have too many skills are going to face the heat anyways

rwyinuse• 12 months ago

I'm not convinced that most companies are going to choose shifting all those people to "higher value work", when the alternative is firing at least some of them them to improve short-term shareholder profits.

herval• 12 months ago

it turns out many professions are essentially a long loop of repetitive tasks. Think telemarketing, or phone support, for instance. What kind of "higher-value work" would a phone support agent do?

48terry2• 12 months ago

The goal isn’t to replace people [...] It’s about shifting roles, not eliminating them and doing it responsibly.

With what knowledge you have of the entire history of capitalism ever, do you, genuinely and earnestly, believe this is what's going to happen, and there isn't, perhaps, a different outcome that is more likely?

tantalor• 12 months ago

The question is totally fair, but it's unreasonable (imho) to expect owners of this project to have to answer. We are looking at < 500 lines of python, mostly just gluing together SIP and agents.

My reaction was slightly different: how many companies selling this (meager) service at a high premium will go out of business now that it's free/open?

sagarkava14• 12 months ago

That is one of our goals. If a solution is under 500 lines of Python, you should not need to pay $499 per month for it. We want to lower the barrier for developers and businesses to build their own voice agents.

GitHub: https://github.com/videosdk-live/agents

https://docs.videosdk.live/ai_agents/voice-agent-quick-start

https://docs.videosdk.live/ai_agents/sip

logical_proof• 12 months ago

We just hook up generators to bikes so that the former phone workers can now power the AI thats replaced them. This will eventually be a cheaper alternative to the current power grid as ai electricity consumption increases.

weikju• 12 months ago

Ah, so THIS is the background story of that Black Mirror episode!

matt-p3• 12 months ago

A huge number of these jobs will go yes. A call centre supervisor may be safe taking escalation calls that need a human and so on but the masses will be replaced by this kind of tech.

However what will actually happen is society will use these people to brick lay for houses, care for the elderly or something else. That's honestly a good thing for society as we have massive shortages there, and not a bad thing for the individuals as a whole.

losteric• 12 months ago

However what will actually happen is society will use these people to brick lay for houses, care for the elderly or something else. That's honestly a good thing for society as we have massive shortages there, and not a bad thing for the individuals as a whole.

Labor "shortages" for those jobs exist because they are not financially attractive. Why is it a "good thing" to eliminate more attractive roles? How does this materially reduce the cost of living, or increase for the roles you point to?

Buttons8402• 12 months ago

One of the pillars of our society is that real-estate value must never go down, and so I'm skeptical that a lot more people are going to start building real-estate.

sagarkava14• 12 months ago

The same technology can also enable businesses that never had live phone support to offer it affordably. The goal is augmentation and access, not mass replacement.

mixdup• 12 months ago

I'll take an alternative view of someone else's comment. No offense, but how much shittier will my customer service experience with my power company or ISP be because of this?

tguvot2• 12 months ago

as somebody who works in this area: depends on how much effort they will be willing to invest. it might be better than current service or it might be worse.

big question will be availability of humans as fall back that actually have some skills

morkalork• 12 months ago

I still can't wrap my head around how making functional online portals and well written documentation for customers is too hard, but laying off entire call centers and replacing them with a chat bot that relies on checks notes a functional set of APIs for tool use and well written documentation for a KB, is easier. It looks like all the same effort plus more!

gerdesj2• 12 months ago

No offense, but how about evaluating the project on its own merits?

In my experience AI based CS agents are not deployed to actually provide customer service. That does not differ too much from the "old school" phone call centre with scripts etc.

Anyway, you can run a phone exchange on a RPi. My favorite anti marketing thing is a simple IVR: "If you think we would like to speak to press 1, if you are making an unsolicited sales call, then hang up". However, that will soon be defeated (but not yet, I'm happy to report). Eventually, I will need the machine to take calls for me. For that I will probably need a system like OPs.

When it comes to telephony, the biggest issue is who pays for the call. For me it is the caller. For some (US int al) it might be the recipient. The only "cost" to me is my time to take a call.

Now, back to call centres. You want problem resolution and they want call stats. There is a bit of a disconnect. Power companies and other utilities all claim to be competitive but nearly everywhere that is bollocks.

In the UK we have privatised water companies, covering regions but how on earth can I, within the purvey of Wessex water, get my water from say Scotland. Ironically enough SSSE (one of the S is Scottish) is the local area electrical provider. I can choose my electic and gas supplier but not my water supplier. How on earth is that a free market? Well it might be but not for me, the consumer. I understand that ISP provision in the US is pretty similar to our water company situation: you have a choice of one.

So, I think, that the problem here is not the AI or the medium or whatever but something far deeper and far more entrenched.

Your shitty customer experience is because the status quo is say 50-100 years old and nothing to do with some nerdy new technology.

cootsnuck2• 12 months ago

I'm someone who builds these types of voice agent systems for businesses. And the short answer is, I think especially for the bigger corporations, customer service probably will get shittier.

It doesn't have to be inevitable...but the track record of big corps and their view of customer-facing departments as being loss-centers instead of being directly and indirectly connected to retaining and bringing in revenue doesn't leave me optimistic.

For small-to-medium businesses...it just depends. SMBs are typically more sensitive to the opinions and sentiments of their customers (and also bad press) and also typically have workers with relatively more institutional and tacit knowledge compared to big corporations. So I would like to believe more of them will be much more methodical and intentional about their approaches (especially with cautionary tales like Klarna and DuoLingo). There's no reason for any business to ruin their customer service with AI other than impulsive and/or poorly executed decisions being made out of some mix of fear, hype, greed, or willful ignorance. Entirely avoidable.

sagarkava14• 12 months ago

That risk is real. That’s why we made this open-source to empower smaller businesses to build responsible systems with their own logic, prompts, and escalation paths. GitHub: https://github.com/videosdk-live/agents

esseph2• 12 months ago

Possibly better, possibly worse.

The bot will have more info and doesn't degrade every few years (as nobody is there to be replaced and lose the tribal knowledge that makes a business run).

It also.. is just a dumb bot and _only_ knows what it is told.

nextworddev• 12 months ago

Just use pipecat

sagarkava14• 12 months ago

Thanks for the mention. Curious—what challenges are you finding with Pipecat that you're hoping something else (like https://github.com/videosdk-live/agents) might fix?

Always looking to improve based on real gaps devs are facing.

Macha• 12 months ago

Personally, it's hard to see outbound calls generated by AI as anything other than robocall spam.

TZubiri2• 12 months ago

I did some freelancing around LLMs for startups and I can't tell you the amount of times I've had to reject "great ideas" that were just about spamming people. Almost had a big red sign "no outbound!"

sagarkava14• 12 months ago

Completely agree. Our goal is to enable smart automation for legitimate, helpful use cases, not intrusive marketing. We also built in Human-in-the-Loop (HITL) support so humans can intervene any time. Docs on HITL: https://docs.videosdk.live/ai_agents/human-in-the-loop GitHub: https://github.com/videosdk-live/agents

Abishek_Muthian2• 12 months ago

I would like to use AI to talk to those spammers/scammers. When the adversary is using AI why shouldn't we to defend from it.

sagarkava14• 12 months ago

You absolutely can. Our framework can answer calls, run speech-to-text, analyze intent, and respond with LLMs, making it a great defensive tool against spam or scam calls.

GitHub: https://github.com/videosdk-live/agents

victorbjorklund• 12 months ago

Boardy ai is pretty nice. Not sure it counts as outbound since you request it to call you.

sagarkava14• 12 months ago

Valid concern. That’s why we focused on responsible use cases such as appointment bots, IVRs, or call reminders, not spam. Our project is open-source, modular, and designed for ethical, contextual automation.

GitHub: https://github.com/videosdk-live/agents

mmasu2• 12 months ago

we tried to build something similar lately for outbound calls (for simple reminders to partners) and faced massive issues using gpt-4o-realtime-audio. Noise detection, turn detection, random telephony issues (we were using Twilio too), prompt not holding together, and more.

We dropped the project because it would have resulted in a terrible experience for the person on the other side of the phone. Building these things is non trivial.

The plan would have been to A/B test and see what the response would have been (watching NPS and business metrics uplift). Human handoff was always the plan in case things got too tricky for the LLM to handle.

I see some hostility here towards this project and while I share many concerns, it is very naive to think that these services won’t be massively leveraged going forward. An AI agent can handle things as well as humans (not in our case but there are good services out there, i.e. Parloa) and the key elements are the same as all the other agentic based workflows:

- narrow use cases

- human in the loop ready to pick up/steer/correct

we will see a lot more of this and as LLM capabilities improve, it will only get better - it is inevitable at this point and might (_might_) result in a better experience for customers in some cases.

Nevertheless I also see the possibility that we will go full circle and we will always reach for a human, maybe showing up in person in a physical office to make sure cases or requests are handled well… or not :-)

48terry2• 12 months ago

I've read this comment twice and I genuinely can't understand it.

Uh, so your own attempt at a similar project didn't work and was a terrible experience and the fundamentals of the system are specific and still require babysitting. But it's inevitable (???) that it'll get better... and this improvement only MIGHT make things better for people, only some of the time?

I'm not alone in being unimpressed by this, right? Nothing about what was written here sounds... good? Even the most optimistic part is "well, maybe it might be good, sometimes". Like, this sucks. This is a bad system that doesn't work and makes things worse.

mmasu2• 12 months ago

what I mean is: building these systems is nontrivial, but if done well it can help. Imagine non being in an endless queue on a phone call when having to do a simple task through a customer center call, or having a phone reminder with more information and less noise than from a written notification. The fact that I failed at it (for lack of experience and resources) does not mean it should just be shrugged off as useless or impractical. Some companies offer this service and it works just fine for narrow use cases.

sagarkava14• 12 months ago

Fair point. But when implemented properly, these agents can reliably handle narrow, production-grade tasks like appointment reminders or smart call routing.

sagarkava14• 12 months ago

We experienced similar challenges. That’s why we made audio handling, turn detection, and LLM retries modular. You can swap models or providers as needed. GitHub: https://github.com/videosdk-live/agents Blog: https://www.videosdk.live/blog/ai-telephony-agent-inbound-ou...

Build an AI telephony agent for inbound and outbound calls

Related Stories