bjornsing

The concept is called static analysis.

Will Anthropic/OpenAI really hire anyone who can fine-tune an LLM?

I ran a small ISP around the same time that used this behavioral pattern to bring down the customer acquisition cost to near zero. Essentially we sold ADSL connections with Wi-Fi and a second SSID where anybody could connect and sign up for internet access. If too may signed up we sent out personal offers for ADSL service to some of them and wired up their homes too. Fun project, but stressful and not very profitable.

To some extent I think that’s just human nature, or even animal nature. The optimal explore / exploit tradeoff changes as we age. When we’re children it’s beneficial to explore. As adults it’s often more beneficial to exploit. But you need cultural and organizational safeguards that protect those of us who are more childish and explorative from those that are more cynical and exploitative. Otherwise pursuits of truth aren’t very fruitful.

Yeah that’s sort of how I understand the OP too: The CPU will execute speculatively on the assumption that the next element in the linked list is consecutive in memory, so it doesn’t have to wait for L1 cache. It needs to check the real value in L1 of course, but not synchronously.

Yes. But I don’t think the OP is suggesting this as an alternative to using an array. As I read / skimmed it the linked list is just a simplified example. You can use this trick in more complex situations too, eg if you’re searching a tree structure and you know that some paths through the tree are much more common than others.

But that works on a different level, right? At least as I understand it data speculation is about prefetching from memory into cache. This trick is about using the branch predictor as an ultra-fast ”L0” cache you could say. At least that’s how I understand it.

Yeah I think this is a general principle. Just look at the quality of US presidents over time, or generations of top physicists. I guess it’s just a numbers game: the number of genuinely interested people is relatively constant while the number of gamers grows with the compensation and perceived status of the activity. So when compensation and perceived status skyrockets the ratio between those numbers changes drastically.

Yeah the whole methodology depends on forgetting about state and treating it as a long-running program. If you need to look at the state then you connect a debugger, etc.

Sure. But that’s just compression, right? I guess you could argue that some information is stored outside the genome, in the structure of proteins etc. But the counter argument is that that information is quickly lost in cell divisions. Only DNA has the error correcting mechanisms needed to reliably store information, is my impression.

Swedish banks (even the Riksbank linked above) regularly refuse to turn cash into digital money unless you can ”prove” where you got it from. It’s not sufficient to say (with immense credibility) that you worked hard all your life and saved it. Entire inheritances are regularly wiped out due to this, when high denomination bills are obsoleted by the Riksbank. So I’d say it’s not only a common sentiment but also government policy.

You’d have to explain where that innate knowledge is stored though. The entire human genome is less than a GB if I remember correctly. Some of that being allocated to ”priors” for neural circuit development seems reasonable, but it can’t be very detailed across everything a brain does. The rest of the body needs some bytes too.

The challenges around idempotency remain to some extent, yes. But you have that problem even in non-workflow code, so the usual patterns will just work with no extra mental effort from the developer.

Sure you get more control with explicit state management. But it’s also more work, and more difficult work. You can do a lot of writes to NVMe for one developer salary.

"Exactly-Once Event Processing" is possible if (all!) the processing results go into a transactional database along with the stream position marker in a single transaction. That’s probably the mechanism they are relying on.

Yes, in any durability framework there's still the possibility that a process crashes mid-step, in which case you have no choice but to restart the step.

Golem [1] is an interesting counterexample to this. They run your code in a WASM runtime and essentially checkpoint execution state at every interaction with the outside world.

But it seems they are having trouble selling into the workflow orchestration market. Perhaps due to the preconception above? Or are there other drawbacks with this model that I’m not aware of?

1. https://www.golem.cloud/post/durable-execution-is-not-just-f...

But couldn’t an LLM search for documents in that enterprise knowledge base just like humans do, using the same kind of queries and the same underlying search infrastructure?

Exactly.

I doubt it was due to network effects. I’m probably one of those top quartile potential H1-B holders that never applied. My main reason for not applying is that the random nature of the lottery reduces the RoI on finding a US job so much that it becomes rational to stay in Sweden and focus on local / remote consulting opportunities. As I understand it the way those Indian IT firms get around that is that they hire locally in India and send in applications for more staff than they want visas for. So they can get around the lottery dynamics through collective action, something I can’t do.

I guess it needs to get across the blood-brain barrier. But that shouldn’t take 10+ hours of surgery, I don’t think.

But it doesn’t take 10+ hours to surgically get a virus across the blood-brain barrier, right?

Why does it have to be delivered through brain surgery?

I’d say an LLM is a form of Markov chain, where the state is defined to consist of the entire context window, and each state transition adds one token to that context window.

So for data centers, training is just as important as inference.

Sure, and I’m not saying buying Nvidia is a bad bet. It’s the most flexible and mature hardware out there, and the huge installed base also means you know future innovations will align with this hardware. But it’s not primarily a CUDA thing or even a software thing. The Nvidia moat is much broader than just CUDA.

CUDA has been a huge moat

The CUDA moat is extremely exaggerated for deep learning, especially for inference. It’s simply not hard to do matrix multiplication and a few activation functions here and there.

While that doesn’t mean their cosine distance is large

There’s a lot of devil in this detail.

I agree the OPs argument is a bad one. But I’m still optimistic about the representational capacity of those 20k dimensions.

Good read. I’ve thought about this distinction myself, but this was definitely more clearly articulated.

This view is kind of like saying that engineers are just people who weren’t smart enough to do physics, or physicists aren’t smart enough to pure mathematics.

To some extent that’s true though. :) Source: I have an MSc in Engineering Physics.

So how do you best store and iterate on the spec? One way I guess would be to work on a branch an modify Claude.md to reflect what the branch is for. Is that a good approach? Are there others?

Not saying they are wrong, but my feeling is that selection effects probably dwarf any causal effects here. You don’t need much skew in the health distribution of people who are drawn to an intervention like this to get this kind of correlation and effect size.