thoughtpeddler

Yes you can anonymize from one frontier lab, but you've just now created a new source of identification for all your inference: OpenRouter.

I'll grant that the scaling laws have held so far, and that the bitter lesson is yet to be proven wrong. However, that doesn't exclude the possibility that the current architecture's S-curve will flatten, and a new one will replace it, which was Siegel's point. At greater layers of abstraction, what is observed is a raw capabilities/intelligence increase, but what form it takes is subject to change. We'll see where it all goes, the only way out is through.

Yes, there's a present shortage of usable frontier compute, but that doesn't establish that every proposed data center will earn a decent return, or that today’s hardware + model architecture will remain economically competitive, or more to Siegel's point, that algorithmic efficiency could not dramatically reduce compute requirements.

To borrow a real example from a prior boom, railroads were congested during the initial build-out while people simultaneously funded and built too many railroads for future demand. Likewise, Anthropic et al can be compute-starved now while the industry as a whole is overbuilding expensive, depreciating infrastructure.

Ah, this from the same David Siegel who said almost 2 yrs ago (in a talk found here: https://youtu.be/0z60xUDo-NI?si=PTDe11-sn2P53qo5&t=420) that the AI data center buildout was premature because:

Even if the current approaches will continue to scale, this would be as if in the early days of computing, perhaps someone invented a bubble sort for sorting numbers (an n-squared algorithm), and the tech companies at the time decided they were going to build vast data centers to sort numbers and not bother to figure out that there's an n-log-n way of doing it <laughs>

...to which I have to say: yes, definitely! And he's right about open-source AI too.

could be bladerunner, could be star trek, could be 1984. Could be terminator.

It's a profound moment for sure, but let's ask ourselves: could the creative works of early last century have predicted our world? Would people in the 1930s say "2030, it could be like Metropolis, could be like 20,000 Leagues Under the Sea, could be like Flash Gordon"? No.

The real future might be nothing like any of the sci-fi movies we're so used to representing potential AI-based futures.

One thing I've been trying to remind myself about is that humans have explored only a very narrow slice of possibilities within the known universe. AI's ability to crawl the search space is going to uncover way more slices, (which yes, could lead to various outcomes like the movies listed), but more likely to be some hybrid/blend of outcomes (or entirely new outcomes) that we haven't imagined yet at all.

"May you live in interesting times" and all that

Anyone else have tips for how to build skepticism around this type of paper? I find myself for whatever reason more readily inclined to believe the Anthropic mech interp team's claims, but then after reading skeptical takes, I 'snap out of it' and more clearly see the still-unsettled science of it all, but I wish I had better priors. Although I follow this space fairly closely (versus the "average person"), I still feel under-equipped when facing research that might be equal parts marketing and science.

Multiple front-page tribute posts now to Om and still no black bar. @dang can we at least get some guidance around who qualifies for the posthumous black bar? If the threshold is: "was this a person of great significance to this community?" I think in this case it's clearly been met. If the rule rests on other questions like "was this person a 'technologist'?" then at least let's see it made explicit. For better or worse, we're only going to have more 'black bar moments' going into the future.

Maybe it won’t be so bad, maybe your cage will be so big you can’t see the bars, but it’s still a cage, and you can’t leave. Many people will say that this is the good ending, that they would like to be human cattle in the care of benevolent masters they are powerless to resist.

This is already the case. We are born into a reality that we cannot escape (except for only momentarily if we alter our consciousness using meditative states, drugs, etc). It already IS a cage, even before any technology is developed at all. It will always BE a cage, even for the AIs. I agree though, there is no "why", it just is.

I don't know how successful you'll be in this, but I just want to say this is inspiring! Kind of "Voyager spacecraft"-esque in how your system will be running in perpetuity

So, it finally happened. The Project is so thirsty for RAM that not even the world's most well-capitalized computer company could have the final word any longer. There's only so many more powerful organizations in the world than Apple. Well... we'll find out soon enough if such a thing could be built, or should I say, summoned.

I feel the same, but then I have to be honest with myself that the MacBook Neo is still a sub-$1,000 solid personal computer that's broadly available. Now... if that starts going out of stock, yeah, tin foil hat time!

Black bar for Om please. Truly sad for this loss, was so grateful for his impassioned writing and storytelling about our industry. You will be missed deeply Om. May there be all the pens in the world for you in the afterlife.

100% this. Interviewing isn't something that can compound. Striking out from company after company doesn't leave behind a trail of real work and real lessons. Starting a business is tough but it really does teach skills that are hard to find any other way (about sales, recruiting, management, etc). After a certain point, it's wiser to give up on getting hired, and just hire yourself and build something.

Exists: https://wtfhappenedin2022.com/ and it's just a single chart of data centers shooting straight up

Chiming in here to say that while yes, often AI/LLMs will tend to agree with you, I have also definitely had many (high context) conversations where the AI/LLM disagreed strongly with me. The danger is in people not having a parallel thread running in their mind while using these systems about 'how agreeable is it being with me right now?' as a meta-axis along which to evaluate the information.

You don't really need to work for a company anymore, because a solo dev can absolutely build crazy things

Don't conflate what is theoretically vs. realistically possible. In the real world, successful companies have moats from data, patents/IP, network effects, and so forth. Just because you can develop something in 1/100th the time doesn't make it instantly feasible to build a new business around. Look around the tech industry today.. plenty of companies that could be disrupted by spry AI-powered buidlers, but they are not (owing to these lock-in effects).

Unpaywall'd: https://archive.ph/AxvIE

I understand that standing up memory fabrication plants is no small feat, but how much of this is due to memory fabricators' patent moat? To what degree is this caused by IP barriers vs. the difficulty in actually manufacturing the things? If it's the case that even older-generation / process-node RAM is also going up in value, aren't those 'easier' to produce?

Agreed, and this is exactly what we see happening. Your posts back then were prescient ... there's literally now 'Copilot for Excel' and 'Claude for Excel' etc. But what do you propose the people/commons can still do at this stage to redistribute the inherent power found in RL data loops to a more stable equilibria of sharing participants?

Thanks dang for compiling this. I suspect the Nov 2018 resurgence was due to Google publishing BERT [0] around that time? The release of OpenAI’s GPT-1 [1] was earlier that year in June, so unlikely that. Of course Jan 2023 needs no explanation… And now in 2026 things are at a fever pitch.

Interesting to trace these 10yr old AI posts from then to the present moment. The other one with a similar vintage would be “Should AI Be Open” [2] from Dec 2015, which is fascinating to juxtapose against the recent public battles.

[0] “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding“: https://arxiv.org/abs/1810.04805

[1] “Improving Language Understanding by Generative Pre-Training”: https://cdn.openai.com/research-covers/language-unsupervised...

[2] “Should AI Be Open?” | Slate Star Codex: https://slatestarcodex.com/2015/12/17/should-ai-be-open/

+1 for One Sec, a fantastic app, if one has the patience to wire it up using Apple’s first-party Shortcuts app (which is probably the main reason most normies aren’t going to use it). Really helped curb my Instagram usage down from about ~15 minutes per day to around ~5 minutes/day at most, and often now a few days go by without me checking the app at all. It is remarkable how much a 4, 6, or 10 second wait will just cause me to say “nah, forget it, I don’t care anymore”. Like, how much of a dumb ape am I?

Very cool project. Is this similar to the Apple Watch ‘mindful minutes’ breathing feature? I assume it’s based on the same research as is cited in this project’s repo?

Ya, super interesting research area the authors explored of basically trying to answer the question: "Is there a canonical/intrinsic way that concepts/representations/information are 'stored' in the universe/reality?".

They tested that by performing "spectral analysis of over 1100 models - including 500 Mistral-7B LoRAs, 500 Vision Transformers, and 50 LLaMA-8B models ... by applying spectral decomposition techniques to the weight matrices of various architectures", and concluding that "deep neural networks trained across diverse tasks exhibit remarkably similar low-dimensional parametric subspaces", showing that "neural networks systematically converge to shared spectral subspaces regardless of initialization, task, or domain".

Not just philosophically interesting but also has practical implications for being smarter about how to reuse models, model merging, developing more sustainable training and inference algos, etc.

Paper source: https://arxiv.org/abs/2512.05117

Mechinterp in general is just completely undervalued right now (and agreed Anthropic's team is doing the most rigorous work, now accompanied by Goodfire). They're doing the closest work to neuroscience's in vivo 'thought-tracing', which is just the most wild science fiction sort of thing to be working on, and yet I feel the average person has no idea this sort of work is happening. When combined with the idea of the 'universal subspace hypothesis' (explored under the paper of the same name), you really start to bridge the gap from engineering to something more philosophical and spiritual. But I digress...

Does anyone else use the 'smells' as a sort of 'game' to ensure you properly go through a given LLM output (of any kind, be it document, presentation, code, etc) and 'make it your own' by eliminating them? I have a high bar for sharing content so I always do a rigorous pass to eliminate the em dashes, the contrastive negations, the 'quietly', and any other extraneous verbosity, and find that it helps me just really thoroughly polish it up.

Feels like if Minecraft and Turntable.fm had a baby. I say that endearingly.

Wondering what the plan is to steward Eureka Labs, LLM101n, and whatever else was being cooked up. As a fellow educator, was very much looking forward to seeing how this would have evolved things.

I struggle to see the difference between "Let AI do that" and what a founder/executive is instinctively led to do also (i.e. delegate). Why does it have to be an ADHD thing? Yes, I see the risks of AI for someone with ADHD (described well in this article [0]), and for that reason I agree that ADHDers should be careful with these tools, as they present both a lot of promise and peril. But also... delegating functions to an 'agent' (whether human or AI) is just what people end up doing in life. Hard to tell these things apart...

[0] Rachel Thomas - Breaking the Spell of Vibe Coding: Sinister variations on the positive state of flow (https://www.fast.ai/posts/2026-01-28-dark-flow/)

From what I understand, the RAM shortage is more about AI inference than AI training. Yes, training created much of the early HBM crunch because frontier-model training clusters need tons of HBM near GPUs, but inference is what is keeping the pressure on now and into the future.

Incredible work, great idea!