MusicLM: Generating Music from Text

https://google-research.github.io/seanet/musiclm/examples/

by _4bjt • 3 years ago

220 41 3 years ago

google-research.github.io

andreyk• 3 years ago

Link to paper: https://arxiv.org/abs/2301.11325 (the one on the site seems broken)

Even with the knowledge that something like this is inevitabely coming, it's pretty crazy to see the examples. The fact it can create fitting music from painting descriptions (including ones such as or Guernica) is unexpected. And it seems to generate almost naturalistic vocals with reasonable-sounding lyrics, with no embedded structure for lyrics or anything.

The paper itself is pretty cool. While GPT-3 was just "GPT-2, but bigger", this seems to have required some clever combination of models. Wonder if similar ideas will be needed for longer-form text or video generation.

robobro• 3 years ago

I can't seem to load any of these WAV files; any mirrors?

thekyle• 3 years ago

I found that they worked for me in Chrome, but not in Safari

vlaaad• 3 years ago

Ohh I want a product where I can play and experiment with it!

bsaul• 3 years ago

yeah, same here. I'm also curious to see what the average quality is like, compared to those hand-picked ones.

nprateem• 3 years ago

Wow that is awesome. Hopefully the first feature they'll add will be to spit out midi rather than wavs (or ML that can decompose wavs to midi)

notahacker2• 3 years ago

I'm surprised that ML-based MIDI generation hasn't been done quite a bit earlier and pretty well tbh. Sound is incredibly complex, but sequences of chords and notes aligned with music theory and genre conventions have well defined and easily imitated patterns. I guess part of the reason MIDI generation hasn't been a major research focus is that toy scripts get you a lot of the way.

These are pretty good, even the vocals are OK, although I wonder how much if parts of the prompt like "it may be used during a festival during two songs for a buildup" are actually adding much to the mix and the music's association with the painting descriptions is as loose as I'd expect.

zozbot234• 3 years ago

The old Biaxial-RNN by Daniel D. Johnson generates very good output for MIDI music, albeit limited to a single keyboard-like instrument. It's available at https://github.com/danieldjohnson/biaxial-rnn-music-composit... and AIUI there's a GitHub fork that forward-ports it to up-to-date versions of Python (3.x series) and Theano.

Transformer models are quite a bit more computationally intensive than the LSTM this used, and GPT adds attention mechanisms; but the basic approach is loosely comparable and the LSTM model can be easily trained on a single machine.

pclmulqdq• 3 years ago

That's a very hard thing for them to add given the way the model is trained. It looks more like a 1-D version of an image generator than past attempts at music AI, which generated MIDI.

mk_stjames• 3 years ago

This is literally trained on and is processing at the level of the audio bitstream. So, getting it to 'spit out midi' would be no different than the current task of taking a full, mixed audio track and generating MIDI from it (which isn't easy). This is using a transformer architecture directly on tokenization of .wav audio. There is no underlaying stage of 'instrumentation' like a MIDI track is before it gets synthesized into an audio stream.

anigbrowl2• 3 years ago

Which is a Bad Thing, because it limits the ability to use it as an educational tool by anyone who wants to learn or modify the music.

survirtual4• 3 years ago

This is unbelievable. I would have never expected AI to take the jobs of creatives first, but here we are. Artists and musicians can create much better work than AI, but the AI creates consistent works usable for people who aren’t artists, and need assets for their projects — whether video games, prototyping, storyboarding, etc.

It is going to be interesting to see artists revert to being a no-revenue passion again instead of being well compensated. That “well compensated” did not last long for them and I feel for the struggle. At the same time, a part of me feels art has no place being motivated by money anyway. Perhaps this change will restore the balance. Artists will need to get real jobs again like the rest of us and fund their art as a side project.

I hope an open model for music generation will be soon to follow, it will really be a big deal.

azaras• 3 years ago

It is going to be interesting to see artists revert to being a no-revenue passion again instead of being well compensated.

Art should be a communication way, a hobby... All the people should have time to do art, do love or whatever they want... Also, all the people, who are able, should do productivity work, to maintain the physical bodies.

notahacker2• 3 years ago

It is going to be interesting to see artists revert to being a no-revenue passion again instead of being well compensated.

Huh? Music already is a near-zero revenue passion for the vast majority of artists, has been since mainstream access to recorded music if not even earlier, and the well compensated minority's superstar status, royalties for hit songs and sellout tours aren't really threatened by DIY tools that generate nice-sounding, genre-appropriate noise a bit more efficiently than a DAW and sample pack.

Juliate• 3 years ago

the AI creates consistent works usable for people who aren’t artists, and need assets for their projects

Right but those will become more and more available, dull, and not original (because feeding from the same « repository »).

So to differentiate and attract/inform/delight/surprise/tell better, they might still have to go to specialized people. Or find a way for those models to, like, hallucinate in different ways.

Virtually anyone today can make their own jewels or even setup a shop on Etsy. At low cost, low price.

Still, there’s a huge market for high quality and original/custom-made jewels. Especially when you want something with a story within it. And kind of a human relationship to the provider too.

flashgordon• 3 years ago

Yeah this is pretty sad. I play an instrument (used to hate music as a kid but somehow paths crossed). I always felt life is best to be about multiple passions instead of all in one thing (whether work or art or sport etc). This could be a great thing for "indie" hobbyists? Sure use AI for stock things but May be we will value the "local" artist more? Or chamber concerts? Or the good old fashioned private dinner with that author? Or a soccer class with Messi's old teacher/coach who lives down the road.

The teacher in me is also optimistic. May be all this AI generated will be great for teaching and democratizing art for students who cannot afford to learn or have easy access to learn!

sharperguy• 3 years ago

I like to think about the things which this kind of tech makes possible that wouldn't have been possible at all before. E.g:

Music generated on the fly for any given scenario, that is unique every time.
Generating custom royalty-free samples which can be used by actual artists as part of a greater work.
A way to learn about musical styles, motifs and timbres by creating many custom examples
A way to experiment by generated many hundreds of examples, clashing together many different types of sounds together to find inspiration

survirtual4• 3 years ago

There are so many, but music on the fly is the top one for me. It would be game changing for all forms of media, able to create a new musical experience with every use of whatever media it is.

smat• 3 years ago

I would argue that the people who use AI generated assets and combine them for a bigger thing such as a video game are still artists. They are simply realizing their vision on a higher level.

survirtual4• 3 years ago

We agree, I am just speaking to what we currently view as artists.

There is a big backlash against AI by current artists and while I understand their pain, I wish we were able to all just stand in awe at the possibilities we’ll soon be having.

Unlimited dynamic catered experiences, accessible by all regardless of income. I just cannot imagine standing in the way of that future. I can’t imagine standing in the way of raw unbridled creation in the hands of everyone. It is just incredibly beautiful.

belter• 3 years ago

"...At the same time, a part of me feels art has no place being motivated by money anyway. Perhaps this change will restore the balance. Artists will need to get real jobs again like the rest of us and fund their art as a side project..."

I seems to be missed on your comment, the AI will take on the real job first. The real job you were planning to reserve as survival means for the artist... are we not all now, creatives out of a job and without a backup plan?

Tv9m2• 3 years ago

"It's a cook book!" hasn't yet entered the Overton window.

parpar• 3 years ago

At the same time, a part of me feels art has no place being motivated by money anyway. Perhaps this change will restore the balance. Artists will need to get real jobs again like the rest of us and fund their art as a side project.

As someone who has experienced both the career paths of a full-time musical artist and technical employee in the tech ecosystem, I find this comment saddening, because it revolves around a highly prevalent way of thinking in our society that needs to die.

Why shouldn't artists have their own career? Why wouldn't you consider this a "real job"?

My experience as an individual artist is that it feels very much like being the founder of a company. You’re dealing with technical, administrative, logistical, marketing issues every day, and you’re required to wear many hats until you generate enough revenue to use external actors/hire a few people.

With all due respect: best-case scenario, these lines might be fuelled by a form of untold jealousy; at worst, they reveal how deeply pessimistic and capitalist our worldview has become. Because such situations have long been observable in modern society, we listeners confuse this longitudinal data with empirical truth, when in reality it’s very much preserved by corporate interests and power dynamics within the music industry.

Artists have long been the last agent of the music business to be paid, and usually get the smallest slice of the cake. This is mostly preserved by a handful of actors who benefit most from this situation. Enter: 1. Labels (including, but not limited to, majors) exploiting the complexity of music rights (master, publishing royalties, ancillary rights, etc.) and imposing harsh contractual conditions on artists. 2. Digital streaming platforms (DSPs) built on vastly unprofitable, VC-money-ridden business models and making up for it by squeezing every possible bit of revenue they can.

The ever-persistent myth of the romantic, tortured artist who needs to live on skid row and experience dire living conditions to create the purest piece of art, devoid of any mercantilism is another problematic piece of the puzzle.

Empirically, extreme precariousness throughout the art world leads to mental health issues I’ve experienced first-hand and observed with many colleagues. When you don’t have the physical or mental resources to produce art, because you’re living from hand to mouth and focused on making a buck on the side, it directly and negatively impacts your practice. A steady, albeit small, income goes a long way.

But because gig-economy mechanisms and atomization of the artistic practice vastly profit the aforementioned actors (you could also throw in social networks, with their black-box, engagement-driven algorithmic updates), it stays the same.

Bottom line, it doesn’t have to be this way. That the ad-tech bubble is about to burst, or that some startup relied way too much on free VC money to prolong its ill-thought-out business model should not have such a drastic impact on the vast majority of artists. And more importantly, citizens like you and I should have no part in perpetuating these toxic mentalities.

Van Gogh was commercially unsuccessful during his lifetime and suffered from severe depression when he cut his left ear off; and here we are arguing this social model is the right one.

My 2 cents.

survirtual4• 3 years ago

If you look at my comment history you will see I am pretty outspoken in being against the system. But I am also fundamentally a technologist and believe in constant forward progress. AI is going to automate “commercial” artists. This is a fact. That was all I was speaking to. For better or for worse, this will happen, it needs to happen, it should not be resisted, and we all need to adapt to this new reality.

It is fundamentally another piece in the collapse of materialism and capital. Soon, all work will be machine-renderable, and what then? We will all be out of “jobs” and will need to rediscover what it means to be humans. A pursuit of experience, meditation, exploration, and significantly less tedious creation — it will be a radically different experience, and far more fundamental to our true nature, than what we experience now.

Me personally? I love to paint, write, and make video games. I love nature and would rather be hiking in remote terrain than pretty much anything else. But there is a lot of work to be done to make what I described in the previous paragraph a reality, and I have skills that can contribute to it, so I do that instead of what personally want. None of us are entitled to simply do what we want while most of us suffer and struggle. Some artists are skilled and lucky enough where they can make money. That bar is going to go a lot higher now with AI.

Artists should be compensated, but for whatever reason, fate would have it that work of the soul might involve acceptance of poverty. It isn’t about capitalism, it is about purity. I think we need universal basic income so that people can pursue their passions without concern of the finances. I also think that passionate people don’t care much about riches; and if they do, that occupies space in a mind that could otherwise be used for their art, and so someone else who properly uses that space will be more capable than them. This is why parasites can take so much from the top artists; they just don’t have the capacity left for that kind of bullshit.

The term “get a real job” might be harsh, but it simply means “get a job capital is interested in paying for”. This criteria constantly changes and AI is pushing it radically. Most people trudge through the shit show of that reality, just because someone calls themself an artist doesn’t make them exempt. I don’t agree with this reality so I would like to see it changed, but we cannot be offended by speaking about the way something is. This is the way it is. If we don’t like it, let’s change it.

Tv9m2• 3 years ago

Artists will need to get real jobs again like the rest of us and fund their art as a side project.

You said you were surprised that the creative jobs are being automated first. Are you expecting the "real jobs" to also be automated, then?

hoosieree• 3 years ago

I listened to some of the longer "story mode" samples on that page and I'm impressed by how bad it is. I feel like I am owed some monetary compensation for my trouble.

If aliens came here and someone told them this was our music, they'd be justified in destroying the planet to make way for an interstellar bypass.

Diapason• 3 years ago

Artists will need to get real jobs again like the rest of us. How disrespectful is this for professional artists?
instead of being well compensated. Hello? What % of pro artists are "well compensated"?

bulbosaur123• 3 years ago

I'm really excited to hear this has been released so soon after Riffusion.

Perhaps both ycombinator threads on MusicLM can be merged together.

If not, I will just post my comments here. I've always seen music as the "final frontier". With images and text you can afford tiny errors and noise that isn't that noticeable by the human eye, but with music everything has to be impeccable and if a note is slightly off, you can instantly tell. Machine learning models that will be able to create outstanding music, imo, will mark the end-game of creative AI endeavors.

Anyone want to share their experiments with MusicLM, feel free to join community-fan discord and subreddit:

https://reddit.com/r/MusicLM/

https://discord.gg/pjVcsyfCJR

BTW, I really hope this gets integrated in software like Ableton similarly how image generators get slowly integrated into Adobe Creative Suite.

rany_• 3 years ago

It's just disappointing that Google has a ton of cool AI projects that have nice announcement pages and seem promising but are never available to the general public.

college_physics• 3 years ago

Art in all its forms had always serious problems getting funded but things only seem to get worse at each iteration. In older times the problem was that most people had other worries, so artist numbers were small and they seeked patronage from the elites that could afford it.

Later we had mass commercialization which "democratized" art but artists got squeezed by the intermediaries / distributors that enabled the mass market. As digital mass reproduction became a thing, this ushered zero marginal cost dynamics which is essentially still an unresolved stalemate.

The "AI" generation of all sorts of combinatoric works without attribution or payment comes to compound the challenges. You now have arbitrary amounts of derivative art that can be reproduced at zero cost.

Never a dull moment in homo sapiens land.

BeFlatXIII• 3 years ago

Perhaps the lesson is that art ought to be done because it is inherently fulfilling and not a as a profession.

necrotic_comp• 3 years ago

One could say the same thing about programming. If I'm playing or writing music for people, and they are enjoying it, I want to get paid.

anigbrowl2• 3 years ago

Great, go tell landlords, grocery stores etc.

It takes non-trivial investments of time to develop artistic capability, often with the aid of tuition or materials/equipment which also cost money. This is common knowledge and I frankly don't understand the point of ignoring it.

btown• 3 years ago

The combination of the physical/audible depiction of art + the human artist's biographical experience, together, makes a form of art that cannot be replicated. (That is, until we have sentient AI - and at that point, is its lived experience any less valid?)

There's debate here, of course, about the "death of the author" [0] and whether work should exist independent of its author's intentions and intentionality - but perhaps our social understanding of that question will evolve, at least for works created after the AI explosion.

As a crude example that buoys me somewhat, there are numerous virtual youtubers who one might discount as avatars of a corporate music product - but their fans love them not only for their musical works in isolation, but for the personality and personal journey that the character (and, by extension, their voice actor, as the lines fade quite quickly) streams to the world, baring a pseudonymized version of their lives for all to see. There's real human intent and struggle and perseverance there, and it's appreciated, and that's really cool. Even if the human is hidden, the humanity shines through.

[0] https://en.wikipedia.org/wiki/The_Death_of_the_Author

Aissen• 3 years ago

A bit sad to see that the audio in the (great!) dataset consists of links to YouTube videos with timecodes, knowing the war on youtube-dl that happened just last year (long live yt-dlp !). Not really future-proof either.

gigel82• 3 years ago

This reminded me of DirectMusic; I remember the demos back in early 2000s (I think it was Age of Empires) where the music was dynamically generated based on what you were doing in the game (smooth transitions and everything).

I wonder why that never went anywhere.

legendofbrando• 3 years ago

I appreciate Google’s research, but it does feel like they’re behind the bar on demos/products consistently. A landing page and a research paper aren’t the best way to demonstrate software.

rhacker• 3 years ago

Everyone better get into construction soon. AI will replace drafters/architects/engineers but the walls going up is going to be a huge payout as Google ML engineers needing a 3rd or 4th house for their second family.

72deluxe• 3 years ago

I think we need not worry. Under "Musician Experience Level" with "professional guitar player", it still sounds like a cat being strangled.

MusicLM: Generating Music from Text

Related Stories