How to Train a Gen AI Kick Drum Model on Your Old Linux Desktop with 6GB VRAM

https://www.zhinit.dev/blog/training-a-kick-drum-diffusion-model

by zhinit • 6 days ago

165 86 6 days ago

zhinit.dev

pringk02• 6 days ago

I just wish it had samples! I want to hear it

zhinit22• 6 days ago

For sure! I just added a couple

motoxpro• 6 days ago

They sound cool! Add a few more! :P

andai• 6 days ago

Did you save any of the "failed" results? I'd love to hear what kind of weird sounds it makes out of distribution (e.g. on the keywords it didn't have much data for).

scoot• 6 days ago

Not sure if there's something wrong with the player, or if it's just me, but they both sound like noise. I guess the first sounds vaguely kick drum-like (but distorted), the second is just noise.

Chrome 149.0.7827.200 (Official Build) (arm64), macOS Tahoe 26.0.1

kleiba2• 6 days ago

I have to admit I don't understand what exactly the problem is we're trying to solve with ML here...?

tgv2• 6 days ago

Decomposing sounds from (fully produced?) tracks into underlying components, and then giving the user the option to synthesize them with different parameter settings. I think.

pennomi• 6 days ago

I was trying (and failing) to do this the other day. It’s a really interesting problem space and I love to see someone with a more solid foundation give it a try.

zhinit22• 6 days ago

I wouldn't exactly say it's trying to solve a problem. It's to explore and see what happens which is what music is all about. It's also a unique niche model I haven't seen before.

mock-possum• 6 days ago

This is a really really fun sounding project - ironically, because there are no audio samples provided at all. I would have thought a music producer creating samples for music would naturally let you listen to what they were making.

zhinit22• 6 days ago

Good point! I just added a couple at the end of the intro

lardosaurusrex• 6 days ago

I always roll my eyes when I see LLM weirdos talk about getting models to run on "old" hardware and finding out it's hardware that's still better than what most people have access to.

It doesn't make it any less impressive to those who know what hardware requirements for LLMs usually is/are but for those with no idea it usually ends up reinforcing bitterness towards it as they feel annoyed that their own hardware is somehow worse and yet are unable to upgrade because of said LLMs stealing all the hardware in the world all while RAM/memory/storage manufacturers manipulate the market(s) against them.

zhinit22• 6 days ago

For sure.

If you are curious I used a NVIDIA GeForce GTX 1660 SUPER So to be exact, it came out 7 years ago (I upgraded at some point on this desktop a long time ago and didn't remember the exact year) (I updated the article to reflect this now)

This cost $230 new and you can get one now for $100 which I don't think is too out of reach.

jdboyd2• 6 days ago

Saying 10yo 6GB will make most people think you are talking about a Geforce GTX 1060.

jdboyd2• 6 days ago

The Geforce GTX 1060 launched 10 years ago with a MSRP of $249. It spend 5 years and 4 months as the #1 card according to the Steam Hardware Survey. That makes it hard to feel that it is fair to accuse it of still being better than what most people have access to, unless you are asserting that most people have access to no GPU at all, which is likely accurate, but not likely to be accurate here, nor in any sort of enthusiast circumstance. If you lump the Intel Xe built in graphics (started with the 11th gen Core Is) and the Intel UHD (launched with 8th gen Core Is) together, the combined group would come in 6th place, with the 5 places above that in commonness for people who are actively playing steam games all being considerably faster than the Geforce GTX 1060 or Geforce GTX 1660 cards.

Interestingly, now the #1 GPU is the GeForce RTX 4060 Mobile version, which I believe is the first time the top has been a laptop chip instead of desktop chip.Items #2 and #3 on the list are the 2 generation old RTX 3060, followed by the 1 generation newer RTX 4060. 4th and 5th are RTX 5070 and RTX 3050.

bigyabai• 6 days ago

but for those with no idea it usually ends up reinforcing bitterness towards it as they feel annoyed that their own hardware is somehow worse

I don't think "those with no idea" spend much time thinking about their hardware at all. They respond to marketing and peer-pressure influences, but most of them are not upgrading phones or laptops because they can't run AI on it.

Most people I know have been wanting upgrade cycles to slow down for quite some time, now. I think that those people will survive deferred retail therapy for a few years.

kube-system• 6 days ago

English adjectives are highly contextual. In this context, the author's clear meaning is in context to the current generative AI boom.

juancn2• 6 days ago

Excellent article! I think it has the right level of detail, one question though: why the shape of the tensor? 4x8x11.

That I didn't get from the text.

zhinit22• 6 days ago

the spectrograms are 128x173 (128 mel frequency bins by 173 time frames) the encoder is downsampling 4 stages of stride 2 convolutions so it halves dimensions 4 times

0: 128 x 173

1: 64 x 87

2: 32 x 44

3: 16 x 22

4: 8 x 11

Then i used 4 separate channels.

This was somewhat arbitrary due to the local training constraint. This would be a hyper parameter worth tuning if I had time to dig into this more.

I trained this a few month ago and don't remember exactly what I tried before I arrived here, but I only ran the whole process 2 or 3 times because of how long it took to train. Hope this answers your question!

juancn2• 6 days ago

Yeah, thanks!!

johndear223• 6 days ago

Articles like this are why I come back to HN. Interesting technically, kinda novel and fun. Got me thinking about datasets that may be sitting on old HDD, got TBs of old video and audio from projects of past. Blogs like this help point the way.. Now if only I had the time..

alex7o• 6 days ago

If you know what you want to achieve you can asks claude/codex/glm whatever, to do the proof of concept first and Dave some time like that

zhinit22• 6 days ago

Thanks!

dj_axl• 6 days ago

Modeled reverb yet no modeled compressor, hrmm, is compression not used on kick drums (or not a big part of the sound)?

zhinit22• 6 days ago

The compression is the OTT which stands for Over The Top compression. It was originally a multiband compressor preset in ableton and is now used widely throughout dance music.

larme2• 6 days ago

People who are interested in this application should check synplant[0]. It has a ML technology called "Genopatch" which gives you 2 functionality:

1. you can try to describe a sound with some tags and it will try to generate a sound to capture the feeling of these tags

2.you can feed it with a sound sample and it will try to re-synthesize the sound with its synth engine. Though the end result will usually be just a "re-imagined" version of your input sample.

My guess is the underlying model is not a "deep" model. The main benefit is that the end result is not a wave file, but a list of generated parameters that can be synthesized by the synthplant engine. And now it comes the interesting part: you can tweak these parameters to finetune the generated sound. These parameters have actual meanings (FM ratio, reverb etc.)

[0] https://soniccharge.com/synplant

aw123• 6 days ago

How far are we from getting a general model that can resynthesize any instrumental audio sound without fiddling with any knobs, so that we can recreate instruments we hear from any song? Seems like it should exist by now?

zhinit22• 6 days ago

SUNO is pretty close. It still has some weird things going on with high frequency artifacts and phase between left and right channels but if you aren't listening on a good system (like a phone) most people probably wont notice.

larme2• 6 days ago

For me creating the exact sound is not very interesting from sound designing perspective. You can always sample the real instrument.

Like physical modeling synthesis, the interesting part is to compress the sound to some parameters that you can tweak and generate new sounds

Another approach is VAE, which also you give your some latent embedding, you can tweak the embedding to generate new sound. However the meaning of this embedding is not explicit.

tashian• 6 days ago

Fiddling with the knobs is the fun part. https://www.youtube.com/watch?v=la2u4VlGwbQ

aa-jv• 6 days ago

In the early days of FM synthesis it was not uncommon to hear the refrain "FM synthesis can reproduce any sound you can hear on the radio" from some of the wilder-haired synth nerds of the time.

Aphex Twins' MIDI Mutant came pretty close (quite a few years ago now) to delivering on that promise:

https://magazine.raspberrypi.com/articles/aphex-twin-midimut...

I'm pretty sure another pass at this problem would prove quite fruitful - as others have noted elsewhere in this thread, there are tools like SynPlant which promise this kind of functionality, although - for my needs - I much prefer AT's approach, having it all in a single box.

Seems to me that the Zynthian/Monome[1] folks might have something like this in their toolkits, somewhere. Might be time to catch up with those projects...

[1] = https://zynthian.com / https://monome.org

zhinit22• 6 days ago

synplant is a great synth!

trencedamp• 6 days ago

For a moment I thought Gen AI meant the current generation of kids. It's a fitting moniker

cocodill• 6 days ago

someone needs to take care of the snares

zhinit22• 6 days ago

If you are committed the model should work about the same on any type of one shot sample. The code is public and documented so if you have the snare collection and a macbook you could probably point claude/chatgpt at it and it would be able to train on your laptop.

tunk2• 6 days ago

This has been done years ago. See https://audialab.com/products/emergent-drums-2/ for instance.

zhinit22• 6 days ago

Interesting! I had not seen this. On their website they mention diffusion but not the other models so it might not be identical but its definitely similar.

tunk2• 6 days ago

That's fair :) I also could've phrased my comment a bit more politely.

robotswantdata3• 6 days ago

Confused. Why not just make the kick drum from a sine? Seconds

zhinit22• 6 days ago

I sound design a lot of stuff (in fact I made some of the default kicks in the app), but this is just a different tool, and I wanted some practice training and deploying a generative AI model.

lilbigdoot• 6 days ago

A sine is only a part of most kicks

robotswantdata3• 6 days ago

Everything is a sine

psd1• 5 days ago

Thank you, thank you, we're the Fast Fouriers, good night!

Cloudef• 6 days ago

Its possible and boy do i have a youtube channel for you https://youtu.be/ndG-6-vONNc https://youtu.be/8dfgum9XlJc

robotswantdata3• 6 days ago

Good modelling

thangalin2• 6 days ago

Slightly off-topic. Now that 1920s jazz music is falling into public domain, has anyone tried to reinvigorate the music using AI and generative adversarial approaches? Pre-1940s music didn't have high-fidelity sound, so the strong bass lines weren't captured. In theory, we could "downgrade" modern recordings to sound like 1920s recordings, then use adversarial techniques to train the machine on how to restore the antique recordings. Anyone know of any work being done in this area?

gregdaniels421• 6 days ago

So the idea would be to reconstruct the low frequency components from whatever upper harmonics are left in the recording? If you know the instruments and positioning of the recording device and something of its(the instruments, recorder, environment, etc.) characteristics, it might be possible to solve that using classic methods. There would be huge numbers of parameters, it is an interesting thought. Is there a large easily/freely available corpus of those recordings?

zhinit22• 6 days ago

To do this, I think you are right that you would need to 'downgrade' modern recordings to sound old so that you have both sides of the training data covered.

This would be a cool project to work on. Ideally you would buy some vintage gear and then run the audio through both, but that would be very expensive. You could may be find some vst emulations though and get decent results.

wildzzz• 6 days ago

It might be easier than that. Are the bass lines totally missing or are they just very weak? If you can capture a recording using vintage equipment and the placing of it, you can get the system response. Run the original recordings through an inversion of the response and you should get really close. Another possible method is to find the transform between an identical modern recording of the song and use the difference between the two recordings to make your transform.

embedding-shape• 6 days ago

Are the bass lines totally missing or are they just very weak?

I think it might be that it's missing a large part of the lower frequencies, not that entire bass sound is missing. And I guess it'd be hard to faith-fully regenerate those, if we simply don't have a lot of samples.

seanhunter• 6 days ago

Neither. You can hear them if you listen carefully, just the recording tech plus lack of amplification makes it harder than modern music.[1]

Source: have degree and postgrad in jazz and used to be a bass player. Have made transcriptions of early bass players from original recordings. (by ear without any kind of fancy tech)

[1] and the playing technique for various reasons.

codedokode2• 6 days ago

The problem might be more complicated. In those times, they might not use a separate mic for every instrument (and the mics probably were not great), they might might not do the mastering properly, the amps could distort the sound, and instruments could overlap each other. And if you try to simply amplify lower frequencies, you might end up getting too much noise.

thangalin2• 5 days ago

Some examples:

The Mooche

* https://www.youtube.com/watch?v=BPJ5vNmVL3I (The Duke, 1928)

* https://www.youtube.com/watch?v=tLdCq2PKM4o (John Barry, 1984)

Sugar Foot Stomp

* https://www.youtube.com/watch?v=qEdIWVsfPXs (Henderson, 1925)

* https://www.youtube.com/watch?v=35foefTrdLo (Nighthawks, 1993)

Doctor Jazz

* https://www.youtube.com/watch?v=HTYAaX7lqjQ (Morton, 1926)

* https://www.shazam.com/song/1687138286/doctor-jazz (Asaro & Fat Babies, 2016)

I was wondering if anyone was working on this (i.e., using pure software). Period equipment, real musicians, and such would be time consuming, arduous, and cost prohibitive.

nxpnsv• 6 days ago

There are plenty of current human bands that play this music really well though...

BrandNewRetro2• 6 days ago

Just giggling silently as I imagine the stereotypical trad jazz purist's response to OP. Man, those guys are... Uhhh... Loyal to that vision of music.

neuralhawke• 6 days ago

The problem is AI "improvisation".

I could take my sequencer and crank the tempo up to a level on a Chopin etude that would smoke Yuja Wang too.

Who cares? The performance that is interesting is a human performance under these artistic constraints.

We didn't need transformers for algorithmic jazz or algorithmic composition in general.

It is also the bullshit of algorithmic Bach. Bach produced 1,100 works and most people haven't listened to even 1% of arguably the greatest artist who ever lived. What is the point of generating more?

fabiofzero• 6 days ago

You could save so much time and processing power by just learning how to sidechain.

bgenchel• 6 days ago

nice work!

zhinit22• 5 days ago

Thanks!

BrandNewRetro2• 6 days ago

I have three (and pray I do not come up with any more) needs for AI audio apps.

One, a ubiquitous restoration model. Find degraded copies of music in the wild, old YouTube's, transcodes, vinyl rips, bad masters, half destroyed tapes... Pair them with modern pristine lossless encodes of the same music, train. Then use that model on music we don't have pristine copies of.

The second is similar but more specific. There are so many stems floating around from popular music. My idea is to compare individual stems against the results from MVS/Spleeter(same song, same instrument). This would surely stand a chance of pushing that tech forward, so we can treat the FFT artefact heavy sound of new efforts.

Thirdly, from a creative point of view, I wanna do the equivalent of image to image on my tracks... But I actually want it to hallucinate in the manner of the early deep dream images, I want to be able to play with that space..

I can knock out musak to spek in minutes already, gen music is just reducing low effort to nearly no effort, preventing people with needs from networking with creators.... Uhh.. but I think that's a very general issue with Gen AI away from the corporate/entrepreneurial dev space

zhinit22• 5 days ago

For 1) another commenter had a simmilar idea. I think a GAN would be able to do this. It's just a matter of collecting the data

For 2) SUNO's stem separation is pretty good. The open sources ones (like spleeter) are also not bad, but they are pretty hit or miss.

For 3) This is a really cool idea. The deep dream images produced the fur and eyeball textures because they used imagenet data which had a bunch of dogs and other animals in it. The trick would be finding a good data set to use for this. May be there is a public source of animal sounds floating around you could try this with.

And yeah think people 100% generating songs is pretty wack, but I could see why it would be fun to someone who is not a musician.

Tade0• 6 days ago

A latent space is just a compressed representation of something.

I love these one-liner explanations - the absolute minimum information one needs to move forward. Great writing.

farlow• 5 days ago

Great writing and cool project! I really enjoyed your intuitive explanations.

zhinit22• 5 days ago

Thanks!

mk_stjames• 5 days ago

This is cool, I had some questions because I was trying to understand this more - not the diffusion model training part, because that I've actually seen before- but the actual way you bundled this into a web application and the choices made...

Did you publish the actual trained model anywhere? I see how in the code there is python for how individual samples can be generated, but the model checkpoint pulldown comes from a directory that... I don't see.

I then went through the code of how this runs on the web and- I'm not a web dev guy- so I'm pretty confused at all the bits bolted together to make this into a web app. It seems like there is a WASM bit that is compiled from a typical C++ audio plugin that is doing the stuff like conv reverb and limiting and distortion in the web app - all that is oldschool, non-generative AI, DSP being applied to the samples. Then the samples are just... a few default generated samples, to start- where are they pulled from, physically? And you have a login requirement to spool up the actual generative AI part to generate new samples to run into the DSP (because that needs a GPU on the backend to do, so, a login to help rate limit this)

How big is the actual generative model? Did you ever think about building the generation engine into the WASM bundle, using maybe WebGPU in the WASM to accelerate in a platform agnostic way, so that the entire app would run offline in someone's browser window?

I'm having fun just playing with the kick program without a login, which, again, am I right in saying in that mode there is no gen AI of sampled happening server side, it is just playing starting with some pre-made samples?

zhinit22• 5 days ago

The weights are ~300MB and are on hugging face here. https://huggingface.co/zhinit/kick-gen-v1

It was a lot of work to get a good DSP to work on the web hahah. Yes, I am writing the DSP in C++ and compiling to WASM. Im using multithreading so the audio work is done in the AudioWorklet while the UI is run in the main thread. I was thinking of writing another article on this because it's pretty interesting and a bit complex.

I sound designed some of the kicks myself and some of them are from sample packs. I just renamed them all to have German names.

If I wanted the model to run completely in the frontend I would need the user to download 300MB of weights and it would probably be tricky making sure it works on everyone's hardware. So I though about this but it didn't seem like the best option. I'm pretty sure it is possible though.

And yes I put rate limiting so no one goes crazy on my credits.

Im glad you're enjoying it!

codedokode2• 5 days ago

What about copyright? Can one take 13,000 copyrighted works, train the model and release it for free so that nobody ever buys any of those 13,000 samples? Can one (legally) convert all commercial sample libraries into a free neural network?

zhinit22• 5 days ago

Im not selling a product here so I'm not going to worry about it for my purposes.

This is the hot topic in AI ethics.

Is it just learning the same way that a human learns from hearing songs on the radio and playing them, or is it just a compression algorithm? There is not a clear answer here but it seems like the legal system so far is letting things slide and agreeing with the former argument.

(This can also be applied to other areas like open AI reading the NYT)

nativeit• 5 days ago

I remember making kick drum models with kilobytes of RAM.

thenthenthen2• 5 days ago

Couldnt get sound to work on iOS but back at my Desktop now and wow these examples/presets sound really cool, great job! (The examples in the write up sound very weird btw...). Try out the 'app' here > https://kick-with-reverb.vercel.app/

zhinit22• 4 days ago

Yeah, for web apps you have to make sure your phone is not on silent and the ringer volume is up. This is why most audio apps for phones will force you to use their native app.

I was looking into a way around this and I don't think it exists without building out an entire phone app.

seth17• 4 days ago

Have you tried setting `navigator.audioSession.type = playback`?

thenthenthen2• 4 days ago

Im building some web audio things and iOS is really a pain. Sometimes the audio plays, sometimes not, sometimes it get stuck playing in the bg. The silent mode button is also something i forget often haha thanks for reminding me again (my phone is always on silent because i get about 15 spam calls a day…)

How to Train a Gen AI Kick Drum Model on Your Old Linux Desktop with 6GB VRAM

Related Stories