evilmathkid

This is a small transformer trained from scratch in 1.5hrs on a 5090 that beats many LLMs. Code is open source.

I want to solve sample efficiency and this work is an attempt to find the limits of transformers and today's methods while keeping costs low so I can iterate fast.

Blog: https://mvakde.github.io/blog/44-on-arc-1/ X thread: https://x.com/evilmathkid/status/2029519274835148829

(This is the public eval. Also it scores 7% on ARC-2)

(1) Use ublock origin to block reels, comment section and other features

(2) Wipe history and only watch productive topics on your account. Watch all other videos on incognito - I set up violentmonkey scripts to help. Here's a cool secret: your feed will ONLY be related to the videos in your history. My feed is now ONLY ML and music

(3) Delete the app on the phone. Only use youtube through firefox (if you have android) which allows the ublock extension

(4) Alternatives like newpipe helped before I stumbled onto this solution

You still need a little willpower. Find the strength to not use youtube for 3 weeks. Then it becomes much easier

Edit: clarity

You also need to make the CNN recurrent, allow it to unfold over many steps, ensure input and output grid are same size and avoid non-local stuff like global pooling, certain norms, etc.

Either way, parent comment is correct. An arbit NN is better than a CA at learning non-local rules unless the global rule can be easily described as a composition of local rules. (They still can learn any global rule though, its just harder and you run into vanishing gradient problems for very distant rules)

They are pretty cool with emergent behaviors and sometimes they generalise very well

This is why OS is so important for LLMs and the AI ecosystem in general.

Its also why we should not trust large AI corporations that appoint themselves as stewards of "AI safety". If a company that once had the slogan "don't be evil" can do this, so can all the frontier labs