Eager

Open weights is one thing, but we don't even have that with OpenAI at least.

Even then, open weights is like me checking in a .exe and acting surprised if people look at me funny.

I'm definitely in the camp where all the artefacts are provided along with fully reproducible build and test environment for anyone who wants to retrace the steps.

Whatever 'open' means, I don't think it is eight shell companies, not even weights provided and closely guarded secrets around how RLHF, alignment and safety testing is carried out.

In fact, you would think that being 'open' about at least alignment and safety testing procedures would be the least one could expect.

I do understand that revealing these things may disclose zero day exploits for bad actors, but on the other hand, being open for inspection is how things get fixed, and I've never been a fan of security through obscurity.

I can't tell if you are joking.

It seems like you are implying the reason we do stuff is to make money, or at least the main motivating factor for you, or you believe it is for most people.

I started working in a factory when I was about 13 casually, and by 14 I was putting in enough hours for my technical apprenticeship.

During those years I did get paid, and I was thankful for it, but I definitely wasn't doing it for the money, and I was still going to school and then to college in parallel.

I'm sure things are different these days. It just makes me sad to think personal progress is somehow conflated with earning money.

With regard to the article. I think given what we know about social networks and young minds, it's already been shown to be a toxic mix. Getting kids on the capitalistic treadmill so young seems extremely cynical and it really does concern me.

A few months ago I made a neat little linux utility.

It was a drop in replacement shim for an arbitrary executable that would pretend to be the original when invoked, fork off the original and hook up to its stdout and stderr.

The error output was then fed to a custom GPT assistant that knew what program the errors came from. That assistant was tasked with turning the original errors into friendly human readable form. The output from the assistant was then sent out of the shim stderr.

It worked very well, but then I got really sick and wasn't able to work on it anymore.

I was using it for GCC / Clang errors because I had become tired of staring at heavily nested compiler dumps for concept/template issues, but you could use it for anything of course.

It would be a nice project for someone to build again, do it properly and generalize it since it doesn't look like I am going to be bouncing around again for a while.

This is rather excellent!

It would be great to have a VST or CLAP (preferably) plugin that hosts strudel and can be hosted in-DAW

It might also be nice to have a VSCode extension that let's you experiment in live mode like the guys at cmajor.dev have.

Finally, I could see this working really well with say cmajor in general or for embedded synthesizer development on say the Daisy Seed if there were a native version.

In any case, it's all great as it is. I will give it a try integrating with Bitwig this afternoon.

I don't know the pharmacology, but it is well known that Saffron makes you happy and a bit giggly.

The grocery store relies on the mystique of the thing. The people making those expensive jars probably also seed articles online about it being extremely expensive. Then when the average person goes to the store they look at it and aren't shocked by the ridiculous price because somewhere at the back of their mind they will have read it is worth more than gold. lol. Meanwhile, anyone in the know is buying it by the wheel for quite reasonable amounts, and likely better purity.

Grinding with sugar is definitely the way to go. I generally buy a big wheel of Iranian Saffron on eBay or a Persian shop, then grind it all up in a mortar and pestle with some sugar and store it in a spice jar in a dark cupboard. Then when you need it you don't need to be faffing about. If you want to be fancy, save a few whole strands that you can add to rice for decoration, but using whole stamens is really not necessary.

Oh that's a nice one. I think you are on to something that if you want to go beyond push button output you really need to think holistically about what you want to achieve.

My experiments taught me that it is an instrument that is easy to approach but hard to master right now at least.

It's kind of weird because you have to play it through writing text, which is super strange.

Everything I have done previously has been very in the moment with keyboards or guitars or whatever. With this I had to put a lot more thought in ahead of time, and try to put myself in the position of the generator and how it might take my prompts and convert them.

There is certainly quite some thinking that has to go on. How to describe the sound you are looking for is a little bit unusual.

I am finding it quite fascinating anyway, and I really believe it can be tasteful if done right and as the tools improve.

As far as covering specific artists, it isn't something I am attempting, it's more about the feel for me. I imagine people who do that have their own tricks and techniques.

As far as I know, Suno blocks any mention of specific artists. Probably you can jailbreak it, but I don't know about that.

I agree in especially the first one. It was quite a challenge from the original content as the bitrate is quite low still, and in that particular case it was quite severely compressed. I had to dig quite deep to get the headroom back.

The last one was my first attempt.

I think the middle one is okay considering. By the time I got to that I had figured out how to get Suno to create multiple takes with much more open mixes, which left me a lot more latitude in the studio.

I expect I will get better at it, and I don't doubt the compression artifacts and the rest will improve.

To me at least, it is quite impressive where we are at. A ways to go, but very promising.

I have been playing with it for the last couple of weeks.

I do a lot of traditional music production for fun and was wondering how I could use Suno together with Leonardo for video and then bring it all together in my existing tools.

Here are some examples. I wrote the lyrics by hand and the music has been reinforced with my existing studio equipment.

For me, that is where the gold is. Not replacing myself, but extending what I can do.

https://youtu.be/Qip6eUbD8zs

https://youtu.be/mfFV3Cm_Kow

https://youtu.be/DZSpi6ySe-g

You try your best, and if you provide enough examples, it will undoubtedly get figured out.

I have tried feeding some of the foundation models obfuscated code from some of the competitions.

People might think that the answers would be in the training data already, but I didn't find that to be the case. At least in my small experiments.

The model's did try to guess what the code does. They would say things like, "It seems to be trying to print some message to the console". I wasn't able to get full solutions.

It's definitely worth more research, not just as a curiosity, but these kinds of problems are good proxies for other tasks and also excellent benchmarks for LLMs particularly.

This is why round-tripping the code is important.

If you decompile the binary to source, then compile the source back to binary you should get the original binary.

You just need to do this enough times until the loss drops to some acceptable amount.

It's a great task for reinforcement learning, which is known to be unreasonably effective for these types of problems.

extremely underrated comment. nice one!

I don't know what they did specifically with GPT-2 as far as tokenization. It is probably hand massaged BPE.

GPT-2 was a pretty small model and it only had to do enough to be impressive at the time in showing some magic.

It doesn't matter of course that the token identifiers fall in any kind of continuous range since the model never sees them, since they are replaced with the embeddings. That said, sometimes I have wondered if it would have been better to force some natural ordering for these things.

The token layout for LLaMA and LLaMA 2 also leaves much to be desired, even GPT-3 and GPT-4 are all over the shop. Gemini is pretty neat from what I have seen of it. As far as I know it mostly deals with digits. Also, the way spaces are dealt with is different. You don't get a bunch of words with hard coded spaces on the front like you do with LLaMA.

It's interesting because there seems to be a trend with these models to head toward almost byte oriented approaches in certain areas of the dictionary.

That would kind of make sense as the models get larger, the tokens don't have to do as much.

I must say that in my own experiments on very small models, you do get an itch to use pretty fat tokens. It looks good when you demo it, that is for sure. But, I think it is much harder for the model because as you noted, it has to create a lot of rules and exceptions internally that more simplistic token schemes would avoid.

Interesting stuff.

What a great project! One of my favorite things to see is when people implement as much as possible themselves. It really makes a big difference to have control over what is going on. And it is great for educational purposes of course.

I'm definitely going to be looking at this. Hope you had fun making it.

I used to be so optimistic and less cynical.

Now I look at this and all that comes to mind is that actions like this would provide a basis for tapping people closer to source.

All the p2p and e2e encryption in the world won't help if someone is reading every key you type, or in the near future, every thought that crosses your mind.

Still, I applaud your spirit.

I'm not pushing what to talk about so much as pushing the point not to talk about stuff that is obsolete and starting to smell.

It's that hype-churn marketing that is a motivating factor for the groups to innovate, much like Formula 1. It might be distasteful, but that doesn't mean it isn't working.

Give Claude 3 Opus a shot maybe.

One of the reasons I have stayed well clear of the IDE tools is they force me to use their own model.

While they might be convenient it means I can't switch to whatever the SOTA model of the day is at the drop of a hat.

Opus is awesome and well worth a shot.

You might be on to something here. It definitely seems to be the case because I'm using multiple different models as part of my everyday process and getting excellent results as a very experienced low level C++ systems engineer.

What is worse is that seems to be leading to a self-amplifying feedback loop, where people not up to speed enough with the models try to use them, fail and give up making them fall even further behind.

It's incredible how far behind HN of all places is w.r.t. what the current best tech is.

So many people talking about GPT-4 here, or even 3.5 when the SOTA has moved way along.

Gemini Advanced is also a great model, but for other reasons. That thing really knows a boat load of low level optimization tricks.

We already have Claude 3 Opus and it is clear for anyone who has used it that it is way better than GPT-4, especially for coding.

The model names, version numbers or who makes them are irrelevant.

What an amazing magazine. I used to order copies from the library of congress every month via my local town library here in the UK. They would always show up eventually although a month or two out of date. I would have been about ten at the time in 1978. My dad got me into them. I have no idea where he heard about it. Maybe via amateur radio probably. As a kid I can't describe the sense of wonder in the air back then. Something was happening for sure. It's so nice when posts about these old publications occasionally show up on here.

Someone should have an LLM start generating random valid CUDA programs.

Compile each one to get a binary.

Train a language model with the source and output binary.

Hey presto, clean room compiler.

Edit: Oh wait.. duh.. just train it on the equivalent target source.

Presumably you can do this for other targets as well.

Oh give me a break

7.7 billion people in this planet are not part of ISIS.

Nice business model though.

Even if WhatsApp or Telegram or Signal are not compromised, you realky should assume the kernel or baseband are.

When I was a kid, I did an experiment cracking Apple] [ software.

Turns out forget the disk encryption, just hook up the NMI interrupt and you are golden... snapshot whenever you want.

Seriously, security is a joke. Nothing is safe. Get over it.