Show HN: We built a voice to music AI app in 54 hours at a hackathon

http://blog.beatmaticapp.com/

by seertaak • 14 years ago

130 58 14 years ago

blog.beatmaticapp.com

daralthus• 14 years ago

Cool stuff I hope you go on with it! Also worth to check out "The Mouth" from Native Instruments: http://www.youtube.com/watch?v=zzyr66QhrOw

seertaak6• 14 years ago

First off, thanks!

I've been thinking about playing around with that plugin for some time, I just haven't got around to it yet. My understanding is that there is no voice to music feature -- it's more about vocoder and harmonizer effects. It sounds like a genuinely innovative and cool product, though, so I have nothing bad to say about it.

The difference between the two products is that the emphasis of "The Mouth" is on vocal performance, whereas the emphasis in Beatmatic is on composition. The compositional aspect probably doesn't come across that clearly in the videos because, well, all we have implemented is beat detection. But suppose the next step in the app was to voice a bassline, or a melody instrument, or have some cool pre-built synth loops to add to your beat, then we're talking about a music app that can, in theory, offer the ability to write a song to anyone who can sing "Happy Birthday" roughly on key and in time. As a musician who has experienced the joy of writing songs a few times (my band: babylonbaby.de), I'm excited about the idea of sharing that awesome experience with a much larger slice of the consumer market.

rootedbox2• 14 years ago

What is AI about it? There have already been a few audio to midi plugins.. How is this different? Can it detect more than bass, and snare?

seertaak6• 14 years ago

It's different in that almost all audio to midi plugins only consist of onset and tempo detection. Whereas Beatmatic also features classification of the sounds, so it lays them onto tracks for you.

And yes, Beatmatic can detect any two (and in principle, 3+) "distinct" sounds. We use unsupervised learning techniques (clustering), so your vocal sounds don't need to sound like those used to optimize the software.

As for whether it's AI, the term is a bit subjective. I think it's not unfair to use it as there's a fair amount of statistical techniques and tricks to make it work, and the result is an app doing things you normally wouldn't expect a computer to do.

rootedbox2• 14 years ago

Actually a lot of audio to midi are doing pitch on top of onset, and tempo.. some are also polyphonic; some are available as real time VSTs (are you on any of the music dsp mailing lists? you really should be.. the prob you are doing has been handled)...

The 2 minute hack to do what you guys are doing but being able to detect way more that 3 distinct sounds would be to use ableton with the following audio chain..

channel1(audio) -> looper -> vst audio to midi vst -> channel 2 (midi)

set your looper to trigger the on / off recording on channel 2.

sp332• 14 years ago

It's AI because it pulls semantic meaning out of a raw, fuzzy, human-sourced audio feed. It figures out what rhythm you wanted, and which notes correspond to which instruments. But it also knows that you want a regular, repeating pattern that rounds to the nearest 8th note. And I think I saw him push a button to add a high-hat automatically, which is cool.

gilini• 14 years ago

Great video, I'm already sold. Software + music is one of my passions, I bet it's yours too, so I'm hoping you guys get traction with this app :)

thomasf19• 14 years ago

Thanks! What direction would you take it first?... We´ve jotted down a few "product" ideas in the blog...

famoreira• 14 years ago

Cool stuff guys! Can you share more details on the serve side implementation?

seertaak6• 14 years ago

Sure, on the server side we have jetty running an HttpServlet, which currently just applies the computations straight on the request thread. Yes, you read that right! It's designed to be run on remote servers, but we haven't gotten quite that far yet and were more focused on making a demoable app. But it's basically Java 1.7/Maven and your typical libs like Guava, Commons {Lang, Math,...}, and a few other bits and bobs.

Basically a file (currently .wav, but it ought to be .mp3) is uploaded to a the server, which processes it directly in thread and returns the results back in JSON. Really it should be fanned of to a worker pool and update the client with ajax but again, we're not quite there yet.

Revex• 14 years ago

Sweet guys. I'm looking forward to a nice write up.

thomasf19• 14 years ago

We´d love to hear your thoughts on where to take this...

Either here or on Quora :) http://www.quora.com/Startup-Advice-and-Strategy/We%C2%B4ve-...

_girishrao• 14 years ago

Awesome stuff guys. Hackathon projects are fun. Also curious about server side implementation!

tsycho• 14 years ago

Awesome stuff! If you make an iPhone/Android app for this, it might even sell well for amateur impromptu musicians. Alternately, once your backend is more stable, you could make an API, and others can build mobile apps on top. Contact me if you want to collaborate on the iPhone end.

thomasf19• 14 years ago

Thanks for your offer, sure we´d like to hear from you. It is already a iPhone app, although limited by time constraints one build in PhoneGap rather than native. We probably will move the app into a native app, or extend PhoneGap via plugins for more advanced sound functionalities like filters.

Our emails by the way are thomas / martin @ the app domain...

jurre• 14 years ago

Awesome stuff! No need to be so apologetic about some stuff not working optimal!

experiment02• 14 years ago

Hi Guys, Did you do this in the Startup Weekend event at Campus Party Berlin? I was doing it too! Well done for your app it looked awesome!

thomasf19• 14 years ago

Absolutely right... :) We really liked the challenge and not knowing if we could pull it off... It was good fun...

experiment02• 14 years ago

Awesome, we did Reco, the picture deals one. Was a lot of fun and met some awesome people!

portlander52232• 14 years ago

Wow... I've been working for a month on an app that does exactly this. Great job, guys! I'd love to know about your algorithm.

seertaak6• 14 years ago

Sorry for treading on your toes then! :) Feel free to email us at dev@beatmaticapp.com, and we can have a chat about implementation. There's a lot of rough edges and it would be great to talk with someone who's dealing with the same issues!

radarsat1• 14 years ago

Damn i was working on something like this too :-P

Congrats, it sounds cool.

thomasf19• 14 years ago

Would be great to hear from you and work together.. drop us a email!

khiner• 14 years ago

I've been working on this app, too! For Android. We should all have pow-wow about our approaches. karl.hiner@gmail.com

aculver2• 14 years ago

I recommend getting this in the App Store so musicians can start using this. Even for a very simple version, charge at least $1.99.[1] Don't over engineer it in terms of features within the app, but make sure the core functionality works as advertised for a broad range of folks. There is a massive novelty factor here which I think will create a viral growth of sales. This will allow people who can't beatbox to experience the joy of beatboxing. :-)

If it continues to works as advertised as it evolves, I think you could quite easily find yourself either in a licensing agreement (or small acquisition) with Apple. Their Garageband app is packed full of "smart instruments" that are in a very similar vein as this. Most of them are quite awesome. My son is obsessed with playing them. They allow even kids to have fun making music and experimenting with sound. Something like this would be a real improvement from what they currently do for "smart" drums.

So, I think there is an initial awesomeness which is that you could have a successful app, which in itself would be a real accomplishment these days. This concept definitely has the potential. However, I really, really hope to see this picked up and incorporated into a larger tool for creating music in a fun way.

Best of luck, guys. Great job.

[1] Might be a bit specific in terms of advice, but not knowing your background/experience building and marketing apps, I thought I'd throw it out there. Coming in with a price tag that's at least $1.99 gives you flexibility with promotions, etc. down the road. Furthermore, something like this definitely provides at least that much value/entertainment.

seertaak6• 14 years ago

Thanks for your advice and comments!

I've always thought the app has the potential to go viral but it's really nice to hear it from someone else. Thanks also for the advice on not over-engineering and delivering a functioning if minimal product, I'm sure we'd be wise to do that, even though my natural impulse is to add extra functionality! As for the price, I think you're right, in fact Thomas had already suggested it in previous conversations :)

thomasf19• 14 years ago

Domo arrigato gozaimas... you seem to have some experience in marketing apps... would be great to talk a bit about how you would approach it and if what part (if any) you still would work on... we´re at dev@beatmaticapp.com

aculver2• 14 years ago

I'll definitely reach out. Looking forward to talking.

pstuart• 14 years ago

I've wanted to make this type of app for some time (voice input is the future, after all).

A voice driven smart portastudio would be sublime.

supertrill• 14 years ago

agreed. I'd definitely pay 1.99 for the app and maybe even spring for in-app purchase / upgrades from time to time.

great job and looking forward to see this on the app store!

_cbb1• 14 years ago

real cool app!

emigal• 14 years ago

Well done gents!

nollidge• 14 years ago

This is cool. I'm a beatboxer and think of rhythm in that "language" (the way I always see musical notes on a piano, because that was my first instrument). At this stage (at least based on the videos), it doesn't really offer any efficiency benefit over just tapping out a rhythm or putting dots on a grid, but it'll be cool to see where this goes...

davj• 14 years ago

Kickstarter

SoftwareMaven• 14 years ago

My son is a Jazz nut, often playing for hours just for fun. The rhythm section is always the most challenging for him. He usually plays to a CD that has a bunch of different rhythms. I can see him and his friends having a blast with this. Let me know when I can buy it!

thomasf19• 14 years ago

Thanks... that really motivates :)... Let´s do some market research here if you don´t mind: how much would you pay and what price do you think is fair?

mkhalil2• 14 years ago

What OS/Programs where you running on those dell monitors in that video? Looks like it's doing some sampling? Pretty curious.

thomasf19• 14 years ago

It´s Eclipse // Java FX and Mathematica on Linux... The Macbook runs OSX with XCode and some more stuff...

mkhalil2• 14 years ago

Cool thanks!

Bad ass application btw. Looking forward to it.

skant• 14 years ago

This is just awesome guys.. Wish you all the best :)

joaoff• 14 years ago

Excellent work, guys! As an amateur sax player, I can see myself using your app to produce some funky beats on-the-fly. I wish you all the best and I hope I can buy your app in the near future. :)

fudgy73• 14 years ago

Thought this was pretty neat. I'd pay a buck for it and I don't really even listen to music.

You should get to a level to where an average beatboxer can input and this thing spits out something sweet. Game-ify it by having people try to match something onscreen with their mouths or do something a la "draw something" except I input the beat and you guess what song it is or vice versa. Fun!

sic1• 14 years ago

I really dig this work. Been wanting something just like this! I find myself beatboxing here and there randomly, and sometimes i really brew up something nice. Would be awesome to have a great way to catalog these "beat-deas". I can envision this app as an easy means for me to pop out of my pocket real quick when i have a good one. Would be nice to have the original beatbox sample saved as well for a reference.

From there - getting this (beat pattern at the least) into something like Ableton (or some importable format beat format (does that even exist? (Inception!))) would be so awesome.

The biggest thing for me would be for it to be able to handle a pretty complicated rhythm. When i get goin, its a lot faster than your examples (and i'd imagine 2 notes hitting in the same place create a problem as well). I'd imagine it gets pretty complicated pretty quick.

Maybe even a way to record a beat, have it do the conversion, then record over it to make more complicated rhythms.

As you can see i'm excited about your product, and am probably getting way ahead of myself on some of the comments. But either way, keep it up, and let me know when i can try it myself. Thanks!

seertaak6• 14 years ago

Thanks, the ability to export to Ableton would be very cool. It's actually not that hard; Ableton can import midi, so as long as you set up the actual sounds yourself it's as simple as storing a .mdi file on Dropbox or Google Drive. That's something we could feature. And exporting entire loops should be easy as well.

As far as complicated rhythms go, I'd love to get some audio samples of you beatboxing! One of the things we have planned is a petition for beatbox samples -- we'll write a web app or rich client that makes it as easy and quick as possible. That'll allow us to get some better data than the little corpus I've been building up!

If you want to help out, you can either sign up for updates at http://beatmaticapp.com or email us at dev at beatmaticapp com. It would be much appreciated :)

thoughtpeddler3• 14 years ago

Heh, they weren't kidding when they said Berlin was leading a new tech renaissance in Europe (I can see the Fernsehturm in your landing page photo). One day hopefully Beatmatic can join SoundCloud and the rest!

Anyway, this is 100% Kickstarter material.

I would easily pay $5 just to show off this functionality to friends when we're hanging out (but I agree that it should be kept around the $1.99 price point). The novelty here is huge.

A lot of music apps are intimidating to most people but this is the most intuitive way to take human musical creativity and make a structured, manipulable track out of it. And trust me, most human beings can hum a tune/beat out. It's instinctive so the market for this is huge.

I recommend Kickstarter rather than just shooting for the App Store because it's going to be tough to go viral and get easily discovered on the App Store. Kickstarter, in addition to the money, will get you a solid fan base before you get on the App Store which is really important these days. It's just too easy to get lost among all the other apps, no matter how novel your entry might be.

So get on Kickstarter, raise a decent amount of money to get a very polished app ready, but more importantly, use the time to get your concept viral BEFORE debuting on the App Store. You won't believe how much publicity is generated by the weekly updates that Kickstarter projects provide. Each week you give a little nudge to all the supporters through your product development updates and each week they're reminded to tell friends and family about their backing. The press will of course catch on and you'll get that steam too.

Good luck to you guys. This is definitely at the intersection of liberal arts and technology that Steve Jobs loved touting.

If you guys want to talk more about strategy from here on out, check my user profile to get in touch. I'm definitely supporting this project.

joshschreuder2• 14 years ago

I'm not sure Apple's distribution really supports Kickstarter though. You wouldn't be able to give away the app in that way.

thomasf19• 14 years ago

Kickstarter would be great for this, especially building a community.

Maybe a way would be to do a 5$ Kickstarter for people to get early access to the app, even before their friends can buy it from the App Store. This could work through the Apple Enterprise Program, but it might not be allowed in the small print...

That could generate some hype, and backers would have something exclusive for a while till it´s available in the app store. It also would build a community that surely gives great feedback and the first few iterations wouln´t need to go through the app store approval process (which would mean faster iterations / feedback).

thoughtpeddler3• 14 years ago

There's promo codes that allow you to download an app for free. I would assume the Kickstarter backers would get those promo codes.

soulboy01• 14 years ago

is the basic concept of this any different from the Vocal Beater app that's currently available for iphone/ipad by Migamo? (mike gao mobile) perhaps you should link up with him, and see if you guys could combine your research or exchange ideas. i'm sure you guys could provide each other with some mutually beneficial insight with what you guys are trying to accomplish. either way, I look forward to seeing something like this available for android, since that vocal beater app is only available for ios devices.

logn• 14 years ago

I think $1.99 could serve the mass market. But by and large musicians are used to shelling out some serious cash for audio apps that have a lot of features. A desktop or more full-featured version could easily sell around $30-50 I think, especially if you find ways to integrate it into other tools, e.g., export as Apple Logic project (if that's even possible).

gtardini• 14 years ago

Congrats! i was in berlin and watched your presentation at campus party. Good job!

domfilosa• 14 years ago

I agree with many of the comments: Get this in the app store quick!! Then, release alternative versions later.

I'd buy it just to show my friends and maybe jam a little like you did. So rad. Triple "like". I hope you make a million dollars.

deadlysyntax• 14 years ago

This simplifies the process even further than typical midi editors do - lowering the barrier to entry for creating music.

But as a professional drummer and tutor, a tool like this could seriously help me, if you decide to expand the feature set beyond a novelty. For example, this could be a way for me to take notes of drum beats that come to me when I'm not at the kit, if it supported tuplets.

hospital• 14 years ago

really cool stuff.

dpakrk• 14 years ago

great hack ! cheers guys

apercossi• 14 years ago

This has got good commercial potential, not just as an app but also as a Guitar Hero type game played by groups. Keep it simple and ensure it can generate "immediate gratification" and you're onto something big.

Show HN: We built a voice to music AI app in 54 hours at a hackathon

Related Stories