zhinit

Thank you! This one liner worked. I searched for a solution for this when I was building this app and am not sure why I couldn't find this.

Looks like this has been possible since iOS 17 but is not well documented.

Yeah, for web apps you have to make sure your phone is not on silent and the ringer volume is up. This is why most audio apps for phones will force you to use their native app.

I was looking into a way around this and I don't think it exists without building out an entire phone app.

Thanks!

Thanks!

For 1) another commenter had a simmilar idea. I think a GAN would be able to do this. It's just a matter of collecting the data

For 2) SUNO's stem separation is pretty good. The open sources ones (like spleeter) are also not bad, but they are pretty hit or miss.

For 3) This is a really cool idea. The deep dream images produced the fur and eyeball textures because they used imagenet data which had a bunch of dogs and other animals in it. The trick would be finding a good data set to use for this. May be there is a public source of animal sounds floating around you could try this with.

And yeah think people 100% generating songs is pretty wack, but I could see why it would be fun to someone who is not a musician.

The weights are ~300MB and are on hugging face here. https://huggingface.co/zhinit/kick-gen-v1

It was a lot of work to get a good DSP to work on the web hahah. Yes, I am writing the DSP in C++ and compiling to WASM. Im using multithreading so the audio work is done in the AudioWorklet while the UI is run in the main thread. I was thinking of writing another article on this because it's pretty interesting and a bit complex.

I sound designed some of the kicks myself and some of them are from sample packs. I just renamed them all to have German names.

If I wanted the model to run completely in the frontend I would need the user to download 300MB of weights and it would probably be tricky making sure it works on everyone's hardware. So I though about this but it didn't seem like the best option. I'm pretty sure it is possible though.

And yes I put rate limiting so no one goes crazy on my credits.

Im glad you're enjoying it!

Im not selling a product here so I'm not going to worry about it for my purposes.

This is the hot topic in AI ethics.

Is it just learning the same way that a human learns from hearing songs on the radio and playing them, or is it just a compression algorithm? There is not a clear answer here but it seems like the legal system so far is letting things slide and agreeing with the former argument.

(This can also be applied to other areas like open AI reading the NYT)

I just added one with the "techno" keyword in the keywords section. Its pretty weird lol

To do this, I think you are right that you would need to 'downgrade' modern recordings to sound old so that you have both sides of the training data covered.

This would be a cool project to work on. Ideally you would buy some vintage gear and then run the audio through both, but that would be very expensive. You could may be find some vst emulations though and get decent results.

Interesting! I had not seen this. On their website they mention diffusion but not the other models so it might not be identical but its definitely similar.

I sound design a lot of stuff (in fact I made some of the default kicks in the app), but this is just a different tool, and I wanted some practice training and deploying a generative AI model.

Seems like if you build some more scaffolding around it, it wouldn't be bad. I think AI video isn't quite there yet so you probably would want to lean into that. For example you could ask for an animated or cartoon music video so the real shots don't look weird. Also if you gave it some guidance on what a good music video is like it would probably help as well. But yeah idk may be that's not the goal here.

If you are committed the model should work about the same on any type of one shot sample. The code is public and documented so if you have the snare collection and a macbook you could probably point claude/chatgpt at it and it would be able to train on your laptop.

Thanks!

SUNO is pretty close. It still has some weird things going on with high frequency artifacts and phase between left and right channels but if you aren't listening on a good system (like a phone) most people probably wont notice.

synplant is a great synth!

The compression is the OTT which stands for Over The Top compression. It was originally a multiband compressor preset in ableton and is now used widely throughout dance music.

the spectrograms are 128x173 (128 mel frequency bins by 173 time frames) the encoder is downsampling 4 stages of stride 2 convolutions so it halves dimensions 4 times

0: 128 x 173

1: 64 x 87

2: 32 x 44

3: 16 x 22

4: 8 x 11

Then i used 4 separate channels.

This was somewhat arbitrary due to the local training constraint. This would be a hyper parameter worth tuning if I had time to dig into this more.

I trained this a few month ago and don't remember exactly what I tried before I arrived here, but I only ran the whole process 2 or 3 times because of how long it took to train. Hope this answers your question!

That was an oversight. The tower is more than 10 years old. I updated the article to be exact now

For sure.

If you are curious I used a NVIDIA GeForce GTX 1660 SUPER So to be exact, it came out 7 years ago (I upgraded at some point on this desktop a long time ago and didn't remember the exact year) (I updated the article to reflect this now)

This cost $230 new and you can get one now for $100 which I don't think is too out of reach.

For sure! I just added a couple

Good point! I just added a couple at the end of the intro

I wouldn't exactly say it's trying to solve a problem. It's to explore and see what happens which is what music is all about. It's also a unique niche model I haven't seen before.