Orpheus-3B – Emotive TTS by Canopy Labs

by Zetaphor • 1 year ago

186 39 1 year ago

hadlock2• 1 year ago

I'm looking forward to having an end-to-end "docker compose up" solution for self hosted chatgpt conversational voice mode. This is probably possible today, with enough glue code, but I haven't seen a neatly wrapped solution yet on par with ollama's.

nickthegreek• 1 year ago

You can glue it with home assistant right now, but it’s not a simple docker compose. Piper TTS and Kokoro were the main 2 voice engines people are using.

Orpheus would be great to get wired up. I’m wondering how well their smallest model will run and if it will be fast enough for realtime

Zetaphor4• 1 year ago

With some tweaking I was able to get the current 3B's "realtime" streaming demo running on my 12GB 4070 Super with about a second of latency running at BF16

tough• 1 year ago

harbor is a great docker bedrock for llm tools, has some tts stuff havent tried them https://github.com/av/harbor/wiki/1.1-Harbor-App#installatio...

thot_experiment4• 1 year ago

Open WebUI has this already. Works okay, there's definitely a lot a fair bit of latency tho.

michaelgiba• 1 year ago

Nice, I’m particularly excited for the tiny models.

deet• 1 year ago

Impressive for a small model.

Two questions / thoughts:

1. I stumbled for a while looking for the license on your website before finding the Apache 2.0 mark on the Hugging Face model. That's big! Advertising that on your website and the Github repo would be nice. Though what's the business model?

2. Given the LLama 3 backbone, what's the lift to make this runnable in other languages and inference frameworks? (Specifically asking about MLX but Llama.cpp, Ollama, etc)

mmoskal• 1 year ago

I wonder how can it be Apache if it's based on Llama?

Philpax• 1 year ago

That's a good question - I was initially thinking that it was pretrained from scratch using the Llama arch, but https://github.com/canopyai/Orpheus-TTS/blob/main/pretrain/c... implies the use of 3.2 3B as a base.

nico• 1 year ago

even on an A100 40GB for the 3 billion parameter model

Would any of the models run on something like a raspberry pi?

How about a smartphone?

Zetaphor4• 1 year ago

They're going to be releasing a few more smaller models, as small as 150M

That said if you want something to use today on a Pi you should check out Kokoro

hadlock2• 1 year ago

What kind of binary do you run Kokoro with for audio output

Metricon2• 1 year ago

GGUF version created by "isaiahbjork" which is compatible with LM Studio and llama.cpp server at: https://github.com/isaiahbjork/orpheus-tts-local/

To run llama.cpp server: llama-server -m C:\orpheus-3b-0.1-ft-q4_k_m.gguf -c 8192 -ngl 28 --host 0.0.0.0 --port 1234 --cache-type-k q8_0 --cache-type-v q8_0 -fa --mlock

Zetaphor4• 1 year ago

I've been testing this out, it's quite good and especially fast. Crazy that this is working so well at Q4

thot_experiment4• 1 year ago

Wait, how do you get audio out of llama-server?

hexaga• 1 year ago

Orpheus is a llama model trained to understand/emit audio tokens (from snac). Those tokens are just added to its tokenizer as extra tokens.

Like most other tokens, they have text reprs: '<custom_token_28631>' etc. You sample 7 of them (1 frame), parse out the ids, pass through snac decoder, and you now have a frame of audio from a 'text' pipeline.

The neat thing about this design is you can throw the model into any existing text-text pipeline and it just works.

gianpaj• 1 year ago

You need to decode the tokens into audio. See `convert_to_audio` method in `decoder.py`

You can run `python gguf_orpheus.py --text "Hello, this is a test" --voice tara` and connect to the llama-server

See https://github.com/isaiahbjork/orpheus-tts-local

See my GH issue example output https://github.com/isaiahbjork/orpheus-tts-local/issues/15

Imustaskforhelp2• 1 year ago

Can somebody please create a gradio client for this as well. I really want to try this out but the complexity messes me up.

NetOpWibby• 1 year ago

Having a NetNavi is gonna be possible at some point. This is nuts.

rcarmo• 1 year ago

Slightly less enthusiastic Californian - good - but the “British” voice feels cringe.

ben_w• 1 year ago

Aye. As a native Brit myself, I'm not entirely sure which region that accent is supposed to be from.

It's the vocal equivalent of a triple-jointed arm, or a horizon that's different on the left and right side of a portrait.

huijzer• 1 year ago

I always am a bit skeptical of these demos, and indeed I think they didn't put much effort into getting the most out of ElevenLabs. In the demo, they used the Brian voice. For the first example, I can get this in ElevenLabs [1]. Stability was set to 20 here and all the other settings were at their default. Having stability at the default of 50 sounds more like what is in the demo on the site [2].

Having said that, I'm fully in favor of open source and am a big proponent of open source models like this. ElevenLabs in particular has the highest quality (I tested a lot of models for a tool I'm building [3]), but the pricing is also 400 times more expensive than the rest. You easily pay multiple dollars per minute of text-to-speech generation. For people interested, the best audio quality I could get so far is [4]. Someone told me he wouldn't be able to tell that the voice was not real.

[1] https://elevenlabs.io/app/share/3NyQKlL6EeOHpIDtL5pA

[2] https://elevenlabs.io/app/share/TUx4yluXtV3pFTHr7Cl7

[3] https://github.com/transformrs/trv

[4] https://youtu.be/Ni-dKlCpnb4

Imustaskforhelp2• 1 year ago

the [4] is such that since you've told me that its AI , my brain can say that of course its AI , but if you hadn't told me that , I might have thought that maybe this guy speaks like this or reading it in monotonous-ish way (like reading from a script?) and wants to sound professional.

Crazy.

Though I still wish open source to better than elevenlabs. but its all just a dream.

thorum• 1 year ago

It’s kind of like ChatGPT writing, where it can easily fool people who see it for the first time, but after a while you start to recognize the common patterns.

sebastiennight• 1 year ago

Great demo on #4 ; and content-wise, a good lesson I needed to be reminded of.

I was such a fan of CoquiTTS and so happy when they launched a commercially licensed offering. I didn't mind taking a small hit on quality if it enabled us to support them.

And then, the quality of the API outputs were lower than what the self-hosted open source Coqui model provided... I'm thinking this was one of the reasons usage was not at the level they hoped for, and they ended up folding.

The saddest part is they still didn't assign commercial rights to the open-source model, so I think Coqui is in a dead-end now.

evrimoztamur• 1 year ago

Impressive for a small model, and I think it could be improved by fixing individual phrases sounding like they were recorded separately. Subtle differences in sound quality, and no natural transitions between individual words, it fails to sound realistic. I think these should be fixable as we figure out how to fine tune on (and thus normalizing) recording characteristics.

admiralrohan• 1 year ago

What is the difference between small and large models in case of TTS?

For language models I understand the thinking quality is different. But for TTS? Do anyone used small models in production use case?

8organicbits• 1 year ago

A couple things I noticed:

- in the prompt "SO serious" it pronounces each letter as "ess oh" instead of emphasizing the word "so"

- there's no breathing sounds or natural breathing based pauses

Choosing which words in a sentence to emphasize can completely change the meaning of a sentence. This doesn't appear to be able to do that.

Still, huge progress over where we were just a couple years ago.

ForTheKidz• 1 year ago

It sounds like reading from a script, or like an influencer. In that sense it's quite good: i could buy this is human.

However it's not a very good reading of the script, in human terms. It feels even more forced and phony than aforementioned influencers.

Orpheus-3B – Emotive TTS by Canopy Labs

Related Stories