solarwindy

Heh, one vestigial bit of code, and they all are. Mind you, it's quite a creaky codebase, so it's forgivable to keep finding these appendices and calling them out as such. Useful, even.

The possibility to continue to sound like yourself after permanently losing your voice (e.g. from motor neurone syndrome) is one. Perhaps almost the only one.

Quoting Troy from a thread beneath the article:

The easiest approach in that case is to take out the subscription, then immediately cancel it. It'll still last the full month, more here: https://support.haveibeenpwned.com/hc/en-au/articles/7707041...

Uh, no, most crypto wallet addresses have either a checksum or some other means of typo detection / prevention.

100 billion a quarter is Alphabet, right? Given how much click fraud there is, and that every org and business under the sun is held to ransom to feature on the SERP for their own name even — it’s tempting to say Google’s become a private tax on everything.

Outstanding video, thank you. No wonder this took months’ worth of research and animation to make.

Well, yes. Rather than that being a takedown, isn’t this just a part of maturing collectively in our use of this technology? Learning what it is and is not good at, and adapting as such. Seems perfectly reasonable to reinforce that legal and scientific queries should defer to search, and summarize known findings.

FWIW, Claude Sonnet 4.5 and ChatGPT 5 Instant both search the web when asked about this case, and both tell the cautionary tale.

Of course, that does not contradict a finding that the base models believe the case to be real (I can’t currently evaluate that).

I’m finding that whether this process works well is a measure (and a function) of how well-factored and disciplined a codebase is in the first place. Funnily enough, LLMs do seem to have a better time extending systems that are well-engineered for extensibility.

That’s the part which gives me optimism, and even more enjoyment of the craft — that quality pays back so immediately, makes it that much easier to justify the extra effort, and having these tools at our disposal reduces the ‘activation energy’ for necessary re-work that may before have just seemed too monumental.

If a codebase is in a good shape for people to produce high-quality work, then so too can the machines. Clear, up-to-date, close-to-the-code, low redundancy documentation; self-documenting code and tests, that prioritizes expression of intent over cleverness; consistent patterns of abstraction that don’t necessitate jarring context switches from one area to the next; etc.

All this stuff is so much easier to lay down with an agent loaded up on the relevant context too.

Edit: oh, I see you said as much in the article :)

How about a narwhal spacewalking from the ISS, with Earth visible below (specifically the Niger delta)?

https://claude.ai/public/artifacts/f3860a8a-2c7d-404f-978b-e...

Requesting an ‘extravagantly detailed’ version is quite impressive in the effort, if not quite the execution:

https://claude.ai/public/artifacts/f969805a-2635-4e30-8278-4...

VLLMs are incredibly good at decoding math from screenshots, if you’re working from a PDF textbook. ChatGPT especially, and since it’s conversant in LaTeX, it can respond directly in the notation you don’t recognize to break it down for you. It even manages with photos of my handwritten scrawl (mostly).

Isn’t GP’s point, that it’s already enough for those two to have solved it? Not every country with a civil nuclear program needs its own waste containment, it’s just such a small absolute quantity.

Also Antartica → Antarctica

Relative to humans, these models sure have ungodly amounts of knowledge, but they also kinda have a lobotomy, in never having moved through the world. It’s remarkable they work as well as they do trained chiefly on text, but being so untethered from the only reality we know intelligence to have emerged from... frankly, what do we expect?

What is a visualisation?

Our rod and cone cells could just as well be wired up in any other configuration you care to imagine. And yet, an organisation or mapping that preserves spatial relationships has been strongly preferred over billions of years of evolution, allowing us most easily to make sense of the world. Put another way, spatial feature detectors have emerged as an incredible versatile substrate for ‘live-action’ generation of world models.

What do we do when we visualise, then? We take abstract relationships (in data, in a conceptual framework, whatever) and map them in a structure-preserving way to an embodiment (ink on paper, pixels on screen) that can wind its way through our perceptual machinery that evolved to detect spatial relationships. That is, we leverage our highly developed capability for pattern matching in the visual domain to detect patterns that are not necessarily visual at all, but which nevertheless have some inherent structure that is readily revealed that way.

What does any of this entail for machine intelligence?

On the one hand, if a problem has an inherent spatial logic to it, then it ought to have good learning gradients in the direction of a spatial organisation of the raw input. So, if specifically training for such a problem, the serialisation probably doesn’t much matter.

On the other hand: expecting a language model to generalise to inherently spatial reasoning? I’m totally with you. Why should we expect good performance?

No clue how the unification might be achieved, but I’d wager that language + action-prediction models will be far more capable than models grounded in language alone. After all, what does ‘cat’ mean to a language model that’s never seen one pounce and purr and so on? (Pictures don’t really count.)

I think the version hosted on the book's website would work fine on smaller screens (and also seems to have been updated more recently):

https://www.sscardapane.it/assets/alice/Alice_book_volume_1....

I remembered once, in Japan, having been to see the Gold Pavilion Temple in Kyoto and being mildly surprised at quite how well it had weathered the passage of time since it was first built in the fourteenth century. I was told it hadn’t weathered well at all, and had in fact been burnt to the ground twice in this century.

“So it isn’t the original building?” I had asked my Japanese guide.

“But yes, of course it is,” he insisted, rather surprised at my question.

“But it’s burnt down?”

“Yes.”

“Twice.”

“Many times.”

“And rebuilt.”

“Of course. It is an important and historic building.”

“With completely new materials.”

“But of course. It was burnt down.”

“So how can it be the same building?”

“It is always the same building.”

I had to admit to myself that this was in fact a perfectly rational point of view, it merely started from an unexpected premise. The idea of the building, the intention of it, its design, are all immutable and are the essence of the building. The intention of the original builders is what survives. The wood of which the design is constructed decays and is replaced when necessary. To be overly concerned with the original materials, which are merely sentimental souvenirs of the past, is to fail to see the living building itself.

— Douglas Adams, Last Chance to See

When framed like this, it's quite unsurprising that LLMs struggle to emulate reasoning through programming problems: there's just not that much signal out there. We tend to commit what already works, without showing much (if any) of the working.

A test for generality of intelligence, then: being able to apply abstract reasoning processes from a domain rich in signal to a novel domain.

Your observation also points to screen recordings as being incredibly high value data. Good luck persuading anyone already concerned for their job security to go along with that.

Because 'solar' seems to many like a generic descriptor, not specifically related to our star, Sol.

The relevant research field is known as mechanistic interpretability. See:

https://arxiv.org/abs/2404.14082

https://www.anthropic.com/research/mapping-mind-language-mod...

https://archive.ph/4MEnQ

For anyone wondering: https://www.mun.ca/biology/scarr/MGA2_02-07.html

Not a bad idea. For an effective ruse, there ought to be real company formation records, website, job listings, press mentions, and so on.

Stepping back for a second though, doesn’t this all underline the safety researchers’ fears that we don’t really know how to control these systems? Perhaps the brake on the wider deployment of these models as agents will be that they’re just too unwieldy.

It’s role play until it’s not.

The authors acknowledge the difficulty of assessing whether the model believes it’s under evaluation or in a real deployment—and yes, belief is an anthropomorphising shorthand here. What else to call it, though? They’re making a good faith assessment of concordance between the model’s stated rationale for its actions, and the actions that it actually takes. Yes, in a simulation.

At some point, it will no longer be a simulation. It’s not merely hypothetical that these models will be hooked up to companies’ systems with access both to sensitive information and to tool calls like email sending. That agentic setup is the promised land.

How a model acts in that truly real deployment versus these simulations most definitely needs scrutiny—especially since the models blackmailed more when they ‘believed’ the situation to be real.

If you think that result has no validity or predictive value, I would ask, how exactly will the production deployment differ, and how will the model be able to tell that this time it’s really for real?

Yes, it’s an inanimate system, and yet there’s a ghost in the machine of sorts, which we breathe a certain amount of life into once we allow it to push buttons with real world consequences. The unthinking, unfeeling machine that can nevertheless blackmail someone (among many possible misaligned actions) is worth taking time to understand.

Notably, this research itself will become future training data, incorporated into the meta-narrative as a threat that we really will pull the plug if these systems misbehave.

Is that necessarily a blocker? As others in this thread have pointed out, this probably becomes possible only once sufficient compute is available for some form of non-public retraining, at the individual user level. In that case (and hand-waving away just how far off that is), does a model need to retain its generality?

Hypothetically (and perhaps more plausibly), a continually learning model that adapts to the context of a particular org / company / codebase / etc., could even be desirable.

Russia is the most pro-nuclear country in the world

Not France?

France derives about 70% of its electricity from nuclear energy, due to a long-standing policy based on energy security.

— https://world-nuclear.org/information-library/country-profil...

I think that's... up for debate

Been trying to inform myself on how these models work, and it’s pretty interesting, I have to say.

Came across this paper from Anthropic, Scaling Monosemanticity [0], where they’re extracting features (via a trained sparse autoencoder) from the ‘middle’ layer of Claude 3 for the purpose of interpretability, and quite convincingly find features corresponding to abstract concepts that do seem to encode a model of computation of sorts. Most remarkable to me is their example of a feature that activates for functions implementing addition, which holds up under function composition.

I guess there’s more going on under the hood than I’ve been giving credit. Of course, that one example is a tiny window into it, and I recognise that even being able to extract that kind of insight into the model’s workings is a feat.

What they don't have is an ability to infer constraints you didn't explicitly or implicitly spell out. And only limited capability to ask clarifying question.

Interesting to think about how the concept of clarification can be formalized and whether it’s possible to work in to the next-token prediction paradigm. I have too many holes in my understanding at this point to go much further with the idea...

They also are horrible at correcting mistakes through ongoing conversation

I guess this one is somewhat understandable with how the models work, though it’s unfortunate that the typical chat interface strongly encourages you to attempt to resolve your issue through conversation.

a lot of code is in between those two lane markers

Yup.

[0] https://transformer-circuits.pub/2024/scaling-monosemanticit...

I think you neatly summarise why the current pre-trained LLM paradigm is a dead end. If these models were really capable of artificial reasoning and learning, they wouldn’t need more training data at all. If they could learn like a human junior does, and actually progress to being a senior, then I really could believe that we’ll all be out of a job—but they just do not.

Very much appreciate this philosophy to nutrition and physical development.

Why supplement vitamin B12, incidentally?

Then again...

Our brains only developed their abstract reasoning capabilities after we already possessed a cognitive ‘platform’ for learning in a very physical context. Or rather, our cognitive capabilities for things that may seem totally extraneous to reasoning in fact developed in tandem with it. Think, fine motor control over our fingers and opposable thumbs, and the corresponding development of spatial and physical reasoning and hierarchical planning, that allows us to analyse a problem and build a tool with our hands to solve it.

The ‘bitter lesson’ [0] seems to be that we humans are not very good at designing the algorithmic machinery for cognition—better we let the machine discover its own mechanisms. Take the case of AlphaGo, the performance of which greatly improved when the human data was thrown away.

So, perhaps there is a pathway to artificial reasoning that shortcuts past many functions of an artificial brain, as you put it, but it also looks quite like we’re not ourselves going to be able to architect it, and that reasoning has not emerged from the current LLM paradigm of digesting the written knowledge of all humanity.

Where human general-purpose abstract reasoning naturally arose from concrete, goal-directed interaction with the physical world, perhaps we need to recreate that environment for machines to learn to learn and reason themselves.

That need not necessarily be physically embodied, which would surely be heavily constrained in numbers of learning trials. Quickly searching turned up this [1] example of reinforcement learning to walk in a sim, which is then transferred to physical.

I wonder how far the approach can go... Could agents learn to talk to each other? Looks like there’s some recent research in that direction too [2].

[0] https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...

[1] https://www.figure.ai/news/reinforcement-learning-walking

[2] https://www.ifaamas.org/Proceedings/aamas2024/pdfs/p2725.pdf