A.I. Duet: A piano that responds to you

https://aiexperiments.withgoogle.com/ai-duet

by dsr12 • 9 years ago

262 66 9 years ago

aiexperiments.withgoogle.com

krosaen• 9 years ago

I like the concept and the framing of the demo is really cool, but the actual music generated in response to me doing basic things like playing a blues riff or a pentatonic scale was usually just a repeat of some fragment of what I was playing. It felt like something that could be built using a basic heuristic...

standardly• 9 years ago

Agreed, it seems to either play a note you've played or pick a random interval. The interval and timing perhaps aren't truly random but it makes no difference to the listener's ear. Doesn't appear to have any concept of, or the ability to learn, scales, rhythm, or melody (or chords). Neat idea, but akin to playing piano with a 5-year-old (perhaps not as fun).

GrinningFool• 9 years ago

Of course that may change as people could be expanding its training through their interactions with it.

raverbashing• 9 years ago

Yeah, had the same impression

It seems it almost gets the concept of tone, but barely

And for some reason it worked on Firefox, but not Chrome (it doesn't even load for me)

hellofunk4• 9 years ago

Could just be a limitation of the training resources available.

6stringmerc4• 9 years ago

I don't think there's a practical limitation on training resources with respect to Music. Maybe the authors didn't get into any of the rubric, but it's certainly out there. Gobs of it.

zodPod• 9 years ago

It almost felt random at times.

svantana• 9 years ago

Yeah either the examples from the video are cherry-picked, or something's wrong with the deployment, because what I get is way worse than what's in the video.

I really don't like this new trend of throwing Deep Learning at anything without really checking whether a simple model (e.g. markov chains in this case) would give equal results. Because using a simple model wouldn't be "cool" or newsworthy...

ocdtrekkie• 9 years ago

When you've described the entire new vision of your company as being "doing things with deep learning", you aren't going to bother seeing if the simpler solution works. Just isn't good for business.

sb8244• 9 years ago

Here's a relevant excerpt from research I did on college into this. FWIW, my research was specifically into genetic algorithms and generating similar musical styles as the input music. It didn't perform particularly well but had some surprises.

From my research, a Markov model could potentially be viable for this demo in particular. However, it wouldn't be viable for longer generation of music where you're trying to produce an actual song. The biggest problem being that it doesn't have the concept of melody in it.

--- research after here ---

A Markov model utilized by Herfort and Rehberger [17] uses the immediate previous note in order to determine the next note against a set of probabilities. This is a fairly simple algorithm as the set of rules is not large, but it is extremely short-sighted by nature. This “first order” Markov chain is not capable of determining a probability based on a large sequence of pitches. To counteract this, “N-order” Markov chain models can be determined to look more deeply into the past, but the complexity of the rule set increases exponentially with added depth. Further, every aspect of music that is to be modeled would require its own Markov model. For instance, there would need to be a model for pitch, duration, etc. This leads to a potentially huge Markov model rule set where all chains are independent of each other. Thus, Markov chains are good at composing musical melodies on the micro level (such as a measure) but fail to produce quality music at the macro level.

[17] Rehberger, Leopold. “Composing music using AI algorithms.” GRIN Publishing GmbH. 2004. <http://www.grin.com/en/e-book/108991/composing-music-using-a....

Source: https://github.com/sb8244/GeneticMusicComposition

LoSboccacc• 9 years ago

even going a simple jingle bells loop (I know, I haven't studied much) get's a pretty much nonsensical repeat response.

on a side note I'd like to have an option to put ai on mute and have the duet run after a session, since it's extremely disorienting for a noob to play with random noise on the feedback loop

dominotw• 9 years ago

just tried this on my midi keyboard( web midi is awesome!) but the "AI music" response was very underwhelming and sounded bad.

I would be interested if "AI" can help me with different timbres based on the physical environment that music is playing using various sensor like air pressure, humidity, room resonance, maybe even heartbeat, perspiration ect. Make that a new dimension of information coded into the song along with pitch, timing, amp and timbre.

rrherr• 9 years ago

Style transfer and generative adversarial networks have worked well with images. How can these be applied to music?

increment_i• 9 years ago

It does seem like an interesting edge to explore doesn't it? ML techniques combined with spectral analysis could yield some interesting outcomes. The datasets wouldn't be too hard to find either - most people have pretty extensive digital music libraries.

iamwil• 9 years ago

Is that how they did it? Or is it just training some RNNs to repeat the same style back?

iverjo• 9 years ago

Sony has been working on style (genre) transfer in music lately: https://youtu.be/I9M8l2guPSo?t=21m30s

6stringmerc4• 9 years ago

Definitely a work in progress. Was like having a goddamn cat following my fingers around the piano. No sense of rhythm or tonality whatsoever. Like, it objectively made whatever I was playing sound worse.

It may "respond" but it's a lot more like throwing electricity through a dead body than working with a competent duet partner.

Ajedi32• 9 years ago

FWIW, it's not really meant to be playing alongside you. The idea is for you to play a short tune, then wait for the AI to respond with a tune of its own based off of yours.

6stringmerc4• 9 years ago

Well, FWIW as well I tried exactly that as well and it was like it could barely follow what I did (rhythm was off) and then it went in other directions that were not pleasing to the ear. Nor made much of any sense musically. So, again, it's like a cat or child was mimicking my motions just enough to make me dislike the result.

vegabook• 9 years ago

Haha...love the cat and dead body idioms. Captures the fail perfectly.

ovulator• 9 years ago

I'm not sure if it is a reflection of my own lack of understanding of music, but it sounded like chaos to me. Garbage in garbage out though.

catnaroek4• 9 years ago

To whoever implemented this AI: Can you actually play music yourself? Your AI can't. I tried to play “Satori Maiden” (link: https://www.youtube.com/watch?v=0c9bea_thaM ), and your AI only replied with unmusical nonsense.

hellofunk4• 9 years ago

Go easy man. It's an "experiment" for a reason.

catnaroek4• 9 years ago

I'm opposed to this trend of trying to replace actual understanding of the problem at hand with massive amounts of computation. If you want to write a program that writes music, then learn how music is actually written first.

---

@hellofunk: If you want the ML algorithm to actually learn how to write music, at the very least, the programmer must be able to express the goal of producing musical sequences of notes.

jff2• 9 years ago

The problem is that this experiment might as well just replay a random collection of the N most recent notes played, with durations shuffled around, for all the musicality it demonstrates. He wired a neural network to some midi samples, and it didn't generate anything of use. Yeah, it was an experiment, and there's value in failed experiments, but you don't typically make a big website to present your failed experiment.

Maybe if you could set it to "Blues" or something like that, switching training sets, it would do better, but as is it's a nonsense generator.

wufufufu• 9 years ago

I played the A-section for Heart and Soul (the song from the movie "Big") and its response was enough to tell me it was not going to be playing actual duets.

I'd be interested in a version that has learned accompanying portions to many popular melodies. If I could play the right hand part of a piece and it played the left hand I'd be very impressed -- also this would actually be useful software for aspiring pianists.

jgalloway___4• 9 years ago

As an experiment I think this is well done. It's fun to mash the keys and see what happens. If there was a feedback loop to register if the user was happy or displeased with the results via facial recognition or other input with expanded training data it could become quite interesting I think.

Thanks for this!

ssharp• 9 years ago

My immediate reaction was to try and confuse it, so I played "Roll Call" from the Thomas show (kids are obsessed !) It's a bit chromatic so I thought it would have trouble, which it did.

Next, I tried putzing around on the white keys but made sure to never hit a F or F#. I wanted to see if it would try and guess which key I was playing in, C or G major. It seemed to guess C most of the time, though one time I tried hinting at G by starting out on G, it got confused and starting playing both F and F#.

I was just clicking with a mouse, so rhythm was hard to judge but overall this thing did a really bad job "complimenting" my playing!

jff2• 9 years ago

You can use your keyboard, the middle row maps to the white keys and the upper row maps to black keys. Works better than a mouse for tempo, IMO.

coroxout• 9 years ago

Are these standard keys for some other piece of music software?

I keep getting confused because I'm so used to the old-style tracker keys (I keep hitting J for Bb and getting B, which isn't helping my C minor improv). Also one octave isn't great, lag is killing the rhythm, and it keeps not waiting for me to finish.

Neat idea though.

(I'm curious to know whether the training set for the version shown in the demo is the same as for the live web version, as the demo sounded way better than anything I'm getting out of it - though obviously they would have cherrypicked the results for the video.)

jgalloway___4• 9 years ago

For those looking for harmony try letting the machine lead .. if you mimic it's style and complement with a hint of your own rhythm it can be satisfying.

amelius• 9 years ago

I think Google can learn something from Microsoft here: [1]

[1] https://www.youtube.com/watch?v=3oGFogwcx-E

_7fvc• 9 years ago

I think it'd sound more interesting if the computer compliments what I play. For example, if I play in C key, the computer tries to guess the chord progression and play the rhythm that matches what I'm playing.

stevehiehn3• 9 years ago

This is cool! I've been trying to use ML to automate the entire production pipeline. Choose chords, synths, loops, mix and add FX. http://signalsandsorcery.com/

Im finding extracting audio features from loops the hardest part. The selection of notes is actually not as difficult as choosing a good timbre for the notes.

jrowley• 9 years ago

I want to hear one trained on Monk, Bill Evans, etc.

zhemao• 9 years ago

I get the feeling this one was trained on Cecil Taylor.

stevehiehn3• 9 years ago

I've had the most success in my own experiments using many dedicated Models. The first and most important model is a NN that generates a chord progression. After that is much easier to select good notes! http://signalsandsorcery.com/

tyingq• 9 years ago

Markov chains might do better for progressions :)

stevehiehn3• 9 years ago

I have tried just using N-grams which is pretty good but it doesn't have any structure. My next experiment will be to use the 'word2vec' algorithm the same way it it used for NLP. Meaning first create relationships between chords then create sequences with RNN.

cocktailpeanuts• 9 years ago

I know this is meant to be a "cool experiment", but from trying this out for a while my impression is:

I could just build a random sequence generator and tell the world "I made this with tensorflow" and people would believe. I mean, none of the response sounded like music. Which is fine, but then again, what is the machine learning for, if it just sounds like random keystrokes?

sordina• 9 years ago

I think that this response is confused. What is your point? That people can't differentiate random notes from a duet? That people would believe that anything associated with tensorflow was imbued with sophisticated behavior?

How does that relate to this experiment? It's clearly not just playing random notes in the example video...

y04nn• 9 years ago

Make me think about the movie "Electric Dreams"

https://youtu.be/aVE8taDWmMc

cyberferret• 9 years ago

I was excited to play with this, but ultimately left disappointed. I was hoping that my initial melodies would be expanded upon and that I could build a series of riffs that would become increasingly complex but based on the same starting melody, but in the end, the responses from the AI seemed to be fairly random, and didn't seem to be built upon the previous sequences at all. The AI also changed keys a couple of times, which was disconcerting.

camtarn• 9 years ago

Me: plays a couple of minor chords

AI: long pause; holds down one random key for several seconds; boops another random key "Nailed it."

After spending a few minutes playing, I did see evidence of it occasionally tracking my chords (as long as they were very common - e.g. c major, a minor) and tempo (but only matching tempo, as in vaguely matching how fast I was playing, not actually playing in time with me). I played an extended chord sequence and it made one or two nice contributions, but mostly it was just random noise.

I wonder if this is due to lack of training data? Maybe it needs an actual data set of people specifically trading musical phrases with each other, rather than just a set of general music.

It's an amazing idea, and I can actually see it being quite interesting for things like musical phrase creation - supply the first half of a phrase and see what the machine comes up with for the second half, and tweak as appropriate. But currently it's a bit of a case of over promising and under delivering.

return0• 9 years ago

I believe ML can do better than this.

jpalla• 9 years ago

If you are interested in real-time generative music, check out IRCAM's project SoMax/OMax: http://repmus.ircam.fr/omax/home

Sample (software is piano trained with Bill Evans corpus): http://www.dailymotion.com/video/x2j39hf_somax-with-remi-fox...

biot• 9 years ago

This is fun. It actually feels like I'm taking turns with someone else on the other end of the screen who takes what I play and sends back a riff on the notes. The trick is (as the little intro shows) to only play for a few seconds, don't play simultaneous notes, and then stop and let the AI respond. Once the AI has finished, you can try again. If the AI starts playing before you're done, you've screwed up and should make your little sequence of notes shorter. Basically, the title of this is misleading... if you keep playing expecting the AI to join you in a simultaneous duet, it'll sound like cats fighting inside a piano.

Try playing the "Dukes of Hazard" horn: on your keyboard, press (without overlapping the keypresses) "LJGGGHJKLLLJ" and let the AI respond. More often than not, it'll come back with a vaguely pleasing riff on that sequence using mostly the same notes and in the same tempo.

shams93• 9 years ago

I'm getting some good results but I'm also a clasically trained musician so if you haven't studied music it may not give you a pleasing result but if you are a trained musician you may get more mileage.

algomusic2• 9 years ago

Even more sophisticated duets with computationally creative computer: CIM sessions 2017 https://www.youtube.com/user/QCGInteractiveMusic/videos?sort...

algomusic2• 9 years ago

Even more sophisticated duets with a computationally creative computer: CIM Sessions 2017. https://www.youtube.com/user/QCGInteractiveMusic/videos?sort...

erinjerri1678• 9 years ago

I tried this at Google Tensorflow Summit and it was fun!

A.I. Duet: A piano that responds to you

Related Stories