Friends don’t let friends train small diffusion models

https://nonint.com/2022/05/04/friends-dont-let-friends-train-small-diffusion-models/

by webmaven • 4 years ago

92 25 4 years ago

In my experience tinkering with plain vanilla neural networks, training quality is quite sensitive to the width of the first couple of base layers. In particular, using a rhombus / "diamond" shape works best.

Here's my intuition. The output of the first layer is mostly a linear matrix transform. These linearities form the basis for the more complex features. If the network doesn't have enough of these base linear features, it can't compose the higher-level (and lower-dimensional) features later down the line.

duped2• 4 years ago

You might want to just make the first layer a DCT of the input vector which will split them into decorrelated components.

wardedVibe2• 4 years ago

I see we are re-deriving conv nets

ausbah2• 4 years ago

by rhombus or diamond shaped do you mean the first layer being wider than the input layer - with each following hidden layer being smaller than the previous hidden layer?

Scene_Cast22• 4 years ago

By rhombus, I meant something like input -> 20 -> 50 -> 200 -> 50 -> 20 -> output.

i_like_apis• 4 years ago

Hang on, is this thing actually generating halfway intelligible rap?

Utterly amazing.

svantana• 4 years ago

As I read it, this is just a spectrogram-->pcm converter (in speech known as a vocoder), which you can do decently without a model, just pure optimization. The hard part is to generate the spectrogram, which I assume they will tackle next.

woodson• 4 years ago

Well, there’s Griffin-Lim, but there is a reason most recent work uses a trained neural vocoder.

jjoonathan• 4 years ago

Is it halfway intelligible rap or fully intelligible ghost rap?

moralestapia• 4 years ago

I also have this question and it's a bit more complicated for me as I'm not a native speaker + the accent is heavily skewed because of the nusic style.

So, are there some real words in there? Do the lyrics make sense? Or is it just "random" vocalizations that go along with the song?

gwern• 4 years ago

No, he's still just experimenting with the audio representation. However, you certainly can generate halfway intelligible rap, and much more, if you check out https://openai.com/blog/jukebox/ (with GPT-3 lyrics) or consider other kinds of audio synthesis like https://speechbot.github.io/dgslm/ (fully end to end).

durnygbur• 4 years ago

lol, thisrapdoesntexist.com

zone411• 4 years ago

If you connect various steps of what can already be done well, such as melody generation (my project), lyrics generation, singing voice synthesis, chord generation, and various semi-automated VST plugins with generation features, you can create some good stuff already. Automating them completely is not even a huge leap forward but you'll still need to do some cherry picking of the results.

songeater• 4 years ago

shameless plug ging down this route: OpenAI Jukebox creating my singer/songwriter project... https://soundcloud.com/songshtr/sets/machinecroon

lostmsu• 4 years ago

What's the size of the final model? What was the dataset? Training source code?

WhyOhWhyQ• 4 years ago

Can it be proved that any work generated with one of these models (i.e., music generation, visual art generation, writing generation) isn't blatantly stealing from something in the dataset? If the dataset is large enough, how can you know that you're not going to be sued at a later date for publishing work generated with one of these models?

astrange• 4 years ago

You can't, which is why using general web scrapes are problematic. You're fine if you track the copyright status of your training set.

leereeves• 4 years ago

The models do sometimes generate verbatim copies of their training data. Here's an example, Copilot regurgitating Quake code, including sweary comments:

https://news.ycombinator.com/item?id=27710287

smegsicle• 4 years ago

of course you can't, see: the episode of seinfeld where elaine accidentally rips off a ziggy

bitforger• 4 years ago

Wow! Super cool results.

This is just begging to be text-guided with the lyrics. Then you could put in the lyrics and have the model generate a clip it thinks would go with the lyrics. :)

wardedVibe2• 4 years ago

Any chance you'll post the code? Neat project

forgingahead• 4 years ago

This blog is by the author of Tortoise-TTS (stylised as TorToiSe): https://github.com/neonbjb/tortoise-tts

Very cool work being done in ML these days, both inside large corps, and also by independent researchers/"hobbyists" (quotation marks because there is some really expert work being produced by them).

And from yesterday, an open-source version of Google's Imagen by lucidrains: https://github.com/lucidrains/imagen-pytorch

ausbah2• 4 years ago

I got to thinking, though: what if there was some minimum bound to the number of channels for the base layer of a diffusion network?
To test out this theory, I needed a way to compress the output of the diffusion model. One idea that came to mind is using the concepts behind PixelShuffle. So I compressed the waveform by 16x using a 1D form of PixelShuffle, increased the base dimensionality to 256, and started another training run.

I'm not familiar with the author's previous projects or the terminology here. Do they just mean he increased the size of the first hidden layer and (base dimensionality to 256) and compressed the output layer by 16?

Friends don’t let friends train small diffusion models

Related Stories