HN user
https//github.com/PWhiddy
For the purpose of disclosure, it should say “Warning: AI generated code” in the commit message, not an advertisement for a specific product. You would never accept any of your other tools injecting themselves into a commit message like that.
“888 KiB Assistant” but the assistant itself is a multi terabyte rental-only model stored in some mysterious data center.
How does it do on gold stake?
Right, and when they compare to floating point accuracy they seem to be using the number of decimals supported by the mantissa, but the exponent is important no?
“A transformer predicts the next token”
Nope. A transformer is much more general than that. A GPT predicts the next token.
I was excited to try it out so I downloaded the repo and ran the build. However there were 100+ compilation errors. So I checked the commit history on github and saw that for at least several pages back all recent commits had failed in the CI. It was not clear which commit I should pick to get the semi-working version advertised.
I started looking in the Cargo.toml to at least get an idea how the project was constructed. I saw there that rather than being built from scratch as the post seemed to imply that almost every core component was simply pulled in from an open source library. quickjs engine, wgpu graphics, winit windowing & input, egui for ui, html parsing, the list goes on. On twitter their CEO explicitly stated that it uses a "custom js vm" which seemed particularly misleading / untrue to me.
Integrating all of these existing components is still super impressive for these models to do autonomously, so I'm just at a loss how to feel when it does something impressive but they then feel the need to misrepresent so much. I guess I just have a lot less respect and trust for the cursor leadership, but maybe a little relief knowing that soon I may just generate my own custom cursor!
I love this! Your results seem comparable to the counter strike or minecraft models from a bit ago with massively less compute and data. It's particularly cool that it uses real world data. I've been wanting to do something like this for a while, like capturing a large dataset while backpacking in the cascades :)
I didn't see it in an obvious place on your github, do you have any plans to open source the training code?
There has been some good research published on this topic of how RLHF, ie aligning to human preferences easily introduces mode collapse and bias into models. For example, with a prompt like: "Choose a random number", the base pretrained model can give relatively random answers, but after fine tuning to produce responses humans like, they become very biased towards responding with numbers like "7" or "42".
We have a shared community map where you can watch hundreds of agents from multiple peoples training runs playing in real time!
It's impossible to beat with random actions or brute force, but you can get surprisingly far. It doesn't take too long to get halfway through route 1, but even with insane compute you'll never make it even to viridian forest.
Anyone interested in watching lots of reinforcement agents playing pokemon red at once, we have a website which streams hundreds of concurrent games from multiple people’s training runs to a shared map in real time!
https://pwhiddy.github.io/pokerl-map-viz/
(works best on desktop)
the metal backend does currently generate quite a lot of unnecessary command buffers, but in general performance seems solid.
I haven’t gone through the paper in detail yet but maybe someone can answer. If you remove the hidden state from an rnn as they say they’ve done, what’s left? An mlp predicting from a single token?
The water consumed to produce a single hamburger is over 2000 liters, and the power likely well over 100 watt-hours.
That means gpt can write >1000 emails using the resources of feeding a single person lunch. The resource efficiency of these machines already is really quite astonishing.
Awesome article! Something slightly misleading though - the first image shows the intersection of a non-convex shape, but it isn't revealed until much later that the algorithm only works for convex shapes, not the type shown in the first image.
Grokking is a sudden huge jump in test accuracy with increasing training steps, well after training accuracy has fully converged. Double descent is test performance increasing, decreasing, and then finally rising again as model parameters are increased.
If cpu softmax were limited by memory bandwidth, then these vectorization optimizations wouldn't improve performance.
+1 for vast. they usually are the cheapest and have the most supply. some instances can be less reliable at the low end though
It's possible you might not need direct access to wave/subgroup ops to implement efficient stream compaction. There's a great old Nvidia blog post on "warp-aggregated atomics"
https://developer.nvidia.com/blog/cuda-pro-tip-optimized-fil...
where they show that their compiler is sometimes able to automatically convert global atomic operations into the warp local versions, and achieve the same performance as manually written intrinsics. I was recently curious if 10 years later these same optimizations had made it into other GPUs and platforms besides cuda, so I put together a simple atomics benchmark in WebGPU.
https://github.com/PWhiddy/webgpu-atomics-benchmark
The results seem to indicate that these optimizations are accessible through webgpu on chrome on both MacOS and Linux (with nvidia gpu). Note that I'm not directly testing stream compaction, just incrementing a single global atomic counter. So that would need to be tested to know for sure if the optimization still holds there. If you see any issues with the benchmark or this reasoning please let me know! I am hoping to solidify my knowledge in this area :)
500GB/s is going to limit it to at best 1/4 the DL performance of an nvidia gpu. I’m not sure what the floating point perf of these FPGAs are but I imagine that also might set a fundamental performance limit at a small fraction of a GPU.
Systolic arrays are essentially how matmul is implemented in tensor cores in GPUs and TPUs.
Not sure if he still is, but definitely was. Many of his videos are filmed in the Amazon NYC office.
If you have written something like this somewhere, or know someone that has, please share!
ASI is nothing like a corporation
Another awesome project! Note that as of this moment the CUDA part is aspirational. There is no gpu code in the repo yet.
At least one of their runtime implementations I checked out was implemented on top of Skia. Looks like they support a number of possible backends!
I was curious how well gpt-4 could de-noise the text. With the prompt:
uncover the text from the message which as had noise introduced: In Gramm@r, an @R+icle_Is.anY memBer I6.@ claSS oF_dedicateb_WOnbs that arE used m|th_No0n phR@ses tO MarK_Th4 id4ntifiabilitY of +Je ne64Rents of THe noun qhnases. Th4 c@tegoRy of articles constitutES a panT oF sPEecJ:
It was able to perfectly recover the original text.
Still no GPU support. They are nowhere near a MVP.
Have you considered using jax? you can efficiently compute the jacobian (even using the gpu if you want) without leaving python at all. The api is also numpy compatible!
It's fascinating that it can model so much of the subtle dynamics, structure, and appearance of the world in photorealistic detail, and still have a relatively poor model of things like object permanence:
https://cdn.openai.com/sora/videos/puppy-cloning.mp4
Perhaps there are particular aspects of our world that the human mind has evolved to hyperfocus on.
Will we figure out an easy way make these models match humans in those areas? Let's hope it takes some time.