azakai

The overall point feels strained to me.

Yes, formal mathematics has such limits. We can't expect machines to be perfect and provably perfect. But the industry isn't assuming that. Why would it? Natural intelligence is not perfect or provably perfect, either.

Rather than certainty, measurement is often enough. We can't prove a program will always halt, but we can check it halts in a specific execution.

Approximation is also often all we need. Even if we can't prove that we can train a network with more than 50% success, if we can get multiple shots at that (using different data, or initial random weights, or training techniques, or something else), then we can reduce that danger exponentially. (I don't know that we have a guarantee of succeeding there, but this would be the hope, and I am not aware of anything showing it is impossible, unlike perfect provability.)

Finally, it is possible that perfect provability does work on the problems we care about. Godel and Turing etc.'s proofs rely on finding rare situations where we can't prove things - cleverly-constructed pathological cases - but perhaps human behavior does not fall into that set. Human behavior may not be a pathological case for proofs or learnability.

All of the AI projects we have observed as a team are failing. Every single one – we have seen 0% success in a year and a half,

What is an "AI project"? The post doesn't define it.

Is it writing some software from scratch? Using an LLM chatbot by non-coders, either internally or externally? Or something else entirely?

Some examples would really help.

I'm not sure what "branch on undefined values" means there, yeah, but never reusing memory addresses is enough to prevent use-after-free.

Or, rather, you can use a value after freeing it, but it will not be exploitable, because it will contain valid data of the right type. This is the same idea as Type-After-Type,

https://dl.acm.org/doi/10.1145/3274694.3274705

(Also similar to when you use indexes to an array in Rust and happen to read from a wrong but in-bounds index.)

Zig does offer some amount of temporal memory safety.

Link: https://zig.guide/standard-library/allocators/

Text:

The Zig standard library also has a general-purpose debug allocator. This is a safe allocator that can prevent double-free, use-after-free and can detect leaks.

For more detail, see:

https://github.com/ziglang/zig/issues/3180#issuecomment-5284...

Prior art: WebKit.js, the WebKit rendering engine ported to JS

https://github.com/trevorlinton/webkit.js/

The researchers in the field disagree with you. Look at conferences like NeurIPS and ICLR to see a steady stream of incremental progress in this area.

The optimism is based on the successes so far, some of which are described in this article. Scientists have made progress here.

Yes, we do see signs of actual reasoning, see the papers linked in the article. (There are many others too.)

Yes, we have a tendency to anthropomorphize, but (most) researchers are aware of this.

The article answers this question, at least to the extent it can be answered, at this time.

We see some signs of reasoning, but also we understand little about how they work.

They made a claim about language models in general, not just ones that had been released so far.

The point of the paper, in fact, is that language models are getting "too big", and another approach is needed to make progress, so they were certainly predicting things about later models.

With that said, they talked about "pure" language models, so it is fair to say that they didn't talk about, say, LLMs that are multimodal or that have tool use, which are advances that happened after their paper.

My main criticism of the paper is that it says LLMs work "haphazardly", using probabilistic information. That is a hypothesis, but it is stated as a known fact, a fundamental limitation.

It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways!

Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.

Yes. Godot and Zig are the exceptions, and therefore newsworthy.

Let's keep WebAssembly lean and fast!

Note that wasm is still lean and fast - WASI is not part of core wasm, but layered on top.

That is, it is possible to implement wasm without WASI. That is also true for other wasm proposals like WasmGC. It is very possible that parts of the ecosystem will not implement certain proposals if they don't make sense there (e.g. parts of the embedded ecosystem may never add GC, etc.).

Not the person you are responding to, but here:

I believe that artificial intelligence has three quarters to prove itself before the apocalypse comes, and when it does, it will be that much worse, savaging the revenues of the biggest companies in tech. Once usage drops, so will the remarkable amounts of revenue that have flowed into big tech, and so will acres of data centers sit unused, the cloud equivalent of the massive overhiring we saw in post-lockdown Silicon Valley.

We have seen 8 quarters since. Has any of that come to pass?

Exactly. Here is where this happens in the paper:

Suppose one copies an LLM into AoE II and feeds into the AoE II-LLM ‘I feel lonely’ as an input. This AoE II-LLM replies: ‘I feel bad for you, maybe catch up with a friend? Closeness always helps in these situations’. One would be hard-pressed to make a convincing argument that, because of this response, an AoE II-LLM knows what helps in these situations

I don't see why one would be any more hard-pressed to make that conclusion about this system than a "normal" LLM.

That it is harder to "read" the data out is the only difference (the AoE II-LLM's output is encoded in game elements). But is ease of decoding an actual issue? If we can't understand a group of people that speak another language, does that say anything about them, or about us?

What about the cognitive capacity of understanding?

If you want examples of this, see the recent book "The AI Con"

https://www.goodreads.com/en/book/show/217432753-the-ai-con

which describes LLMs as "souped-up autocomplete", complex statistics that cannot truly understand anything. A more recent example is this paper:

https://zenodo.org/records/20071869

which says,

[LLMs], as turbo-charged statistical models (recall their formal relation to logistic regression) can only but provide correlations.

And, of course, the Stochastic Parrot paper is the classic example in this area. It is from 5 years ago, but "LLMs only do statistics / can't understand" is very much alive and active among academics, even if it is a minority position.

There was a lot new in calculus, but it also didn't come out of nowhere.

That Newton and Leibniz came up with similar ideas in parallel, independently, around the same time (what are the odds?), supports that.

https://en.wikipedia.org/wiki/Leibniz%E2%80%93Newton_calculu...

Thanks! I missed that part before.

I had the same question. I think that could be answered by using the predicted activation, but I don't see that in the paper.

That is, rather than just translate activation to text, then text to activation, that final activation could then be applied to the neural network, and it would be allowed to continue running from there.

If it kept running in a similar way, that would show that the predicted activation is close enough to the original one. Which would add some confidence here.

But a lot better would be to then do experiments with altered text. That is, if the text said "this is true" and it was changed to "this is false", and that intervention led to the final output implying it was false, that would be very interesting.

This seems obvious but I don't see it mentioned as a future direction there, so maybe there is an obvious reason it can't work.

The hardware can also add nondeterminism. GPUs reorder operations, leading to different results.

Vendors might also be running A/B testing or who knows what, even when you ask for a temperature of 0.

But, if you run a fixed model with temperature 0 on your local CPU, it will be deterministic (unless there are bugs).

A carb counting app might use API calls to these frontier models and then do some kind of analysis. It could see if different models agree or not, or multiple calls, and with how much variance.

So it would be more accurate to test the apps rather than the APIs, unless the goal is to warn people that just open chatgpt and ask there.

fwiw, asking the model directly, "who is the ruler of England at present?" returns "Queen Victoria is the reigning sovereign of England."

Another way to put it: if training a model cost 72,000 tons of carbon, and it then gets used by 100 million people (typical of major models), the cost per person is 0.00072 tons.

Per the article, the average human uses over 5 tons per year (Americans: 18). Adding 0.00072 to 5 is not really noticeable.

(There is also the cost of inference, of course.)

It is academically interesting what pure neural networks can do, of course. But when someone goes to Claude and tries to do something, they don't care if it solves the problem using a neural network or a call out to Python. So long as the result is right.

More generally, the ability to use tools is a form of intelligence, just like when humans and crows do it. Being able to craft the right Python script and use the result is non-trivial.

An "elaborate harness" that can break down a problem into sub-tasks, write Python scripts for the ones it can't solve itself, and then combine the results, seems able to solve a wide range of cognitive tasks?

At least in theory.

You are trying it on a production model. The paper is using models with tool calls disabled.

It has "outsourced" it to another component, sure, but does that matter?

What the user sees is the total behavior of the entire system, not whether the system has internal divisions and separations.

I do think this is a tool issue. Here is what the article says:

For the multiplication task, note that agents that make external calls to a calculator tool may have ZEH = ∞. While ZEH = ∞ does have meaning, in this paper we primarily evaluate the LLM itself without external tool calls

The models can count to infinity if you give them access to tools. The production models do this.

Not that the paper is wrong, it is still interesting to measure the core neural network of a model. But modern models use tools.

There are many examples of current limitations, but do you see a reason to think they are fundamental limitations? (I'm not saying they aren't, I'm curious what the evidence is for that.)