Building the Future of AI Interactions

https://blog.diarupt.ai/building-the-future-of-ai-interactions/

by marvinified • 3 years ago

19 4 3 years ago

blog.diarupt.ai

ilaksh2• 3 years ago

Anyone have any idea how this works? Is it really as fluid as shown in the video, or are there pauses while the text and audio/video are generated?

Do they wrap something like gpt-3.5 and HeyGen or D-ID APIs? Did they build their own system?

Is there a paper or even a GitHub repo with these models out there somewhere to build off of by any chance?

marvinified2• 3 years ago

Hi ilaksh,

Is it really as fluid as shown in the video, or are there pauses while the text and audio/video are generated?

Yes the video is as smooth as shown in the video, and there are no pauses while video is generating.

There is alot of moving parts to get it be that fluid but for the video generation we built a NeRF-based framework which allows for realtime synthesis.

Do they wrap something like gpt-3.5 and HeyGen or D-ID APIs? Did they build their own system?

We didn't use HeyGen or D-ID, we tried these out initially but when we plugged in all the moving parts they were too slow to allow for realtime AI conversations, which drove us to build out our own system. On the LLM side we use GPT-3.5/4 as the brain which is currently probably the most efficient (cost/output wise) at the moment but we also have Llama 2 integration in the works.

Is there a paper or even a GitHub repo with these models out there somewhere to build off of by any chance?

We're building this out as an API, you can integrate into your own product, to build more complex flows.

It's still in private alpha and we are still polishing the edges but you can join the waitlist (https://diarupt.ai/) to try it out once we start rolling out.

ilaksh2• 3 years ago

Congratulations. Its an amazing technical achievement. I joined the wait-list as "runvnc".

Building the Future of AI Interactions

Related Stories