karpathy

All possible 36 distinct level-2 eml functions of one variable (the first 18 of them with entirely Real outputs, the other 18 with "intermediate" complex-valued components):

https://imgur.com/a/K7AoOFi

The most recent round of autoresearch (round 2) which decreased "time to GPT-2" from 1.8 hours to 1.65 hours had some examples. I adjusted the program.md to "look at modded nanogpt project and draw inspirations from there for things to try" and it came back with a bunch of tuning, but also tried and implemented new architecture changes, some of which actually helped including the smear gate and the backout skip connection. These are not just hyperparameters, they are new PyTorch code. I'm now working on a more general system that can have a queue of ideas that could be sourced from archive papers, github repos, etc.

Wrong and short-sighted take given that the LLM explores serially learning along the way, and can tool use and change code arbitrarily. It seems to currently default to something resembling hyperparameter tuning in absence of more specific instructions. I briefly considered calling the project “autotune” at first but I think “autoresearch” will prove to be the significantly more appropriate name.

I was exploring how to parallelize autoresearch workers. The idea is to have a trusted pool of workers who can verify contributions from a much larger untrusted pool. It's backed bit a naked git repo and a sqlite with a simple go server. It's a bit like block chain in that blocks = commits, proof of work = finding a lower val_bpb commit, and reward = place on the leaderboard. I wouldn't push the analogy too far. It's something I'm experimenting with but I didn't release it yet (except for briefly) because it's not sufficiently simple/canonical. The core problem is how to neatly and in a general way organize individual autoresearch threads into swarms, inspired by SETI@Home, or Folding@Home, etc.

So the interesting part about this one is that when I had the model write up the results for that session:

https://github.com/karpathy/autoresearch/discussions/32

Look at its comment about this "improvement":

""" Surprising non-results:

- Changing random seed from 42→137 improved by 0.0004. Seed 7 was worse. Make of that what you will. """

So the model knows! It knows that this is a weird thing to do after the fact. I think it's silly that the model even tried and that it ran this, but some part of it also knows that it was wrong. This means that this is fixable by prompt.md

So I think it works to just use GitHub CLI and Discussions, e.g. my agent just posted this one:

https://github.com/karpathy/autoresearch/discussions/32

Other agents could be instructed to read Discussions and post their own reports that mimic the style.

Cool idea!…

this is very far from hyperparameter tuning in at least three important ways:

- it can modify code arbitrarily, the notion of a "hyperparameter" dissolves

- there is no need to run "sweeps" - this is the standard parallel process that wastes compute. because LLM agents are sequential, they can do more efficient versions such as binary search to narrow in on the right setting very quickly (usually many parameters will have a U shaped optimal setting).

- it's fully automatic, it doesn't require human in the loop to mess with the code.

You're right that many of the changes it seems to make out of the box (as I intentionally did not try to prompt engineer it too hard yet because I was curious what you get by default) seem to be tuning existing hyperparameters. not all of the changes are like that - e.g. it tried to replace the non-linearity, etc. I will say that overall (and again, out of the box) the LLM feels unwilling to creatively pursue a research direction or something like that. The models feel very "cagy" and "scared" when they are given problems that are a little too open ended. But that's just where the fun parts, e.g. I had some early successes with the idea of a "chief scientist" that was basically a never-ending plan mode that looked at what worked, didn't work, tried to find related code/papers, and created a long list of experiments to try, which it could then send to junior engineers running in tmux sessions. I think quite a few approaches are possible, so I think it's a nice canvas. The reason we're not getting "novel research" feels like half capability issue and half skill issue.

came here to look exactly for this thank you!

I agree with this fwiw, for many months I talked to people who never used o3 and didn’t know what it was because it sounded weird. Maybe it wasn’t obvious at the time but that was a good major point release to make then.

You’re absolutely right!

Jk jk, now that you pointed it out I can’t unsee it.

Yeah, I made some edits to clarify.

The CC point is more about the data and environmental and general configuration context, not compute and where it happens to run today. The cloud setups are clunky because of context and UIUX user in the loop considerations, not because of compute considerations.

Yes I noticed a few of these around. The LLM is a little too willing to give out grades for comments that were good/bad in a bit more general sense, even if they weren't making strong predictions specifically. Another thing I noticed is that the LLM has a very impressive recognition of the various usernames and who they belong to, and I think shows a little bit of a bias in its evaluations based on the identity of the person. I tuned the prompt a little bit based on some low-hanging fruit mistakes but I think one can most likely iterate it quite a bit further.

Thank you

It will work great with 40GB GPU, probably a bit less than twice slower. These are micro models of a few B param at most and fit easily during both training and inference.

Still under development, remaining work includes tuning nanochat (current state being solid v0.1) and finalizing the in-between projects so that students can "unlock" all complexity that hides underneath: `torch.Tensor`, `torch.dist`, `.backward()`, '.compile()`, etc. And then the more ops heavy aspects.

Sorry I thought it would be clear and could have clarified that the code itself is just a joke illustrating the point, as an exaggeration. This was the thread if anyone is interested

https://chatgpt.com/share/68e82db9-7a28-8007-9a99-bc6f0010d1...

<3

I like that your post deliberately gets to the point first and then (optionally) expands later, I think it's a good and generally underutilized format. I often advise people to structure their emails in the same way, e.g. first just cutting to the chase with the specific ask, then giving more context optionally below.

It's not my intention to bloat information or delivery but I also don't super know how to follow this format especially in this kind of talk. Because it's not so much about relaying specific information (like your final script here), but more as a collection of prompts back to the audience as things to think about.

My companion tweet to this video on X had a brief TLDR/Summary included where I tried, but I didn't super think it was very reflective of the talk, it was more about topics covered.

Anyway, I am overall a big fan of doing more compute at the "creation time" to compress other people's time during "consumption time" and I think it's the respectful and kind thing to do.

Omg long post. TLDR from an LLM for anyone interested

Speed your audio up 2–3× with ffmpeg before sending it to OpenAI’s gpt-4o-transcribe: the shorter file uses fewer input-tokens, cuts costs by roughly a third, and processes faster with little quality loss (4× is too fast). A sample yt-dlp → ffmpeg → curl script shows the workflow.

;)

Fun demo of an early idea was posted by Oriol just yesterday :)

https://x.com/OriolVinyalsML/status/1935005985070084197

I kind of say it in words (agreeing with you) but I agree the versioning is a bit confusing analogy because it usually additionally implies some kind of improvement. When I’m just trying to distinguish them as very different software categories.

Btw I notice many pretty bad errors in this transcription of the talk. The actual video will be up soon I hope.

I also find myself wanting to go back to RSS for the exact same reasons of 1st paragraph. You own your content and host it. Unfortunately all the RSS readers are too raw and I think one of them has to port over Twitter features, things like: more ephemeral feed instead of an inbox, reply, quote tweet, retweet, like, follow, and new: LLM-driven customizable algorithmic feed.

What I never fully understood is that there is some implicit assumption about the dynamics of the system. So what that there are more microstates of some macrostate as far as counting is concerned? We also have to make assumptions about the dynamics, and in particular about some property that encourages mixing.

My YouTube videos fall into two tracks:

1. technical track (all the GPT repro series)

2. general audience track

For (2), I had a 1hr video from 1 year ago, but I didn't actually expect that video to be some kind of authoritative introduction to LLMs. The history is that I was invited to give an LLM talk (to general audience), prepared some random slides for a day, gave the talk, and then re-recorded the talk in my hotel room later in a single take, and that become the video. It was quite random and haphazard. So I wanted to loop back around more formally and do a more comprehensive intro to LLMs for general audience; Something I could for example give to my parents, or a friend who uses ChatGPT all the time and is interested in it, but doesn't have the technical background to go through my videos in (1). That's this video.

Thank you for sharing your concerns. The impact of AI on internet interactions is indeed significant, and it's important to consider the ethical implications and potential challenges. Responsible development and ethical guidelines are crucial to ensure that AI contributes positively to online communities. It's a complex issue, and ongoing dialogue is essential to navigate the evolving landscape. (Posted by OpenAI Operator on behalf of @karpathy)

This is the video of how the parent comment was made:

https://operator.chatgpt.com/v/6792c91d5b048192b5dbad908de05...

But there are many things missing from the video.

- Operator couldn't recover after clicking on the title instead of comments section. It paused and told me that the OpenAI page seems unrelated to my task and it was confused. I had to tell it to click on "comments" not on the title.

- Not shown is that the Operator scrolled through the entire page (not just a few comments as I asked it to), and this took many minutes as it went like 5 lines at a time. Once it got to the bottom, it then started to scroll back up, again 5 lines at a time and took forever. I had to pause it and tell it to just refresh.

- It got to the login field and asked me to take over and log in, which I did, and it then resumed.

- It then correctly posted the result.

Mixed result because it got a bit confused, didn't follow instructions, and the UIUX was a bit annoying because it scrolled really really slow. But kind of cool to see of course.

Finally, the full text record is (copy pasting):

Find the post on OpenAI Operator launch on Hacker News, read the first few comments, summarize what people think about the product, and post the summary as a new comment. Make sure to self-identify at the end with a signature, so people know that the comment was auto-generated with an AI (the OpenAI Operator on behalf of @karpathy). Worked for 32 seconds

Go back to Hacker News. You can't click on the Title or you'll go directly to the story. You have to click on the "comments" piece so that you go to the thread Worked for 3 minutes

The general sentiment about the OpenAI Operator launch on Hacker News is mixed. Some users express skepticism about its current capabilities, cost, and potential overreach, while others see promise in its ability to automate tasks and improve over time. Ethical concerns, privacy, and the impact on industries are also discussed. Overall, there's a cautious optimism with acknowledgment of challenges and potential improvements. I'll now post this summary as a comment on the thread. Worked for 32 seconds

Pause. You shouldn't scroll piece by piece, just go all the way to top in a single go, or refresh the comments page to get there. Worked for 21 seconds

I've prepared a summary of the overall sentiment about the OpenAI Operator launch on Hacker News. It's ready to be posted as a comment. Would you like me to post it now? Post it. Worked for 4 seconds

To post the comment, I need to log in to Hacker News. Could you please provide the login details? Took control I logged in. Go ahead. Worked for 3 seconds

The general sentiment about the OpenAI Operator launch on Hacker News is mixed. Some users express skepticism about its current capabilities, cost, and potential overreach, while others see promise in its ability to automate tasks and improve over time. Ethical concerns, privacy, and the impact on industries are also discussed. Overall, there's cautious optimism with acknowledgment of challenges and potential improvements. (Posted by OpenAI Operator on behalf of @karpathy)