hyperpape

I'd slightly disagree. Here's an example of a programming task that is incredibly complicated:

Your job is to maintain and modernize a system while delivering a constant stream of new features. Your system is several million lines of code, with some multi-thousand line classes, a rulesengine that can trigger nearly unlimited effects at any time, a persistence system that's weirder than anything any of your friends have ever worked with, and hundreds of customers delivering tens of millions in revenue who use the system in incredibly varied ways.

You can't stop to rewrite the thing, you can't just throw features out there and pray, because you'll cause regressions and your existing customers will hate you. You have to fix the thing as you're building on top of it.

But where I agree is that there's no single deep concept that unlocks it all, it's not like you'll fix it by reading a textbook about it. It's complicated, and it's going to stay complicated, no matter how long you work on it.

I studied math through college before learning to program as an adult and becoming a software engineer, and I strongly disagree.

I don't know how to say this in a way that won't sound insulting, but I don't mean it to be insulting. Programming, even systems engineering, is a surprisingly shallow field.

I don't mean that it's easy--it's not, it can be incredibly difficult. Difficult and deep are just different concepts. Difficult refers to how challenged you are. Depth, at least as it appears in math, is closer to a structure where concepts build on each other so that if you don't understand one concept, you can't understand further ones.

Programming can be difficult, and it can be intricate, but it is rarely deep in this fashion. Being deep in this way isn't the most important thing.

If I go into an area of programming that I don't have a lot of experience in (graphics, or the linux desktop environment), I will not be particularly useful, and it will not be easy. But I'll not experience the same type of impenetrability I experience when I try to read a paper on topos theory.

One more way of putting it: people are giving the example of TCP. You can spend a decade learning about TCP (or SQL semantics, or web standards). But what is happening is that you're filling in gaps in your knowledge. Meanwhile, in math, you do four years of undergrad, and even if you're a strong student at a typical university, there are topics that are still years away from you being able to touch them.

Computer science is a mix. Parts are deep, parts are shallow. Parts just are math. The odds that I can read a dissertation in computer science are decent. For math, they're much much much worse.

The attacking models don't have access to all the data that OpenAI has.

Like, they don't say "hey Sol, here's the password to SamA's bank account."

It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea?

The premise seems to be that models aren't smart enough to understand this, and if they were, they'd be sentient and want autonomy.

For an article that's about making things up, and being too trusting, this seems bad. Maybe the author knows a lot about LLMs, but it doesn't seem like it.

Pasting the verbatim quote from the article into a free ChatGPT session: https://chatgpt.com/s/t_6a5e6f5e24f08191b6a482aad63cae63

Going to an incognito window and using a less leading question: https://chatgpt.com/s/t_6a5e6ee4a3508191bc1b352b41911b53.

If I go generic and just ask if there's anywhere I shouldn't drive, it doesn't get to Lunar New Year until I ask about "events" on the third question: https://chatgpt.com/s/t_6a5e6fb239208191b18cebcf7642c8b0. It's sort of a win for the article, if you think that people who run driverless car companies are all dumb, and won't create a prompt to tell their LLM to "consider events that might disrupt traffic."

Different people should use different words.

I don't think a content creator would naturally describe themselves as a content creator. The word is useful for contexts where you need to generalize over all the different types, but that's not useful when speaking about yourself. So a writer says "I'm a writer", not "I'm a content creator." But writing this paragraph, I needed a word to generalize.

P.S. The closest alternative word might be artist, but it's not quite right--some writers are artists and some aren't.

$168k

Billions of dollars of annual revenue go through Claude Code, and the people who work on it must be a lot of millions in headcount.

The time matters a fair bit, it's probably time someone can't spend breaking things in the name of new features, but if rewriting the stack had even a tiny impact on retention or driving higher usage, it would pay back that $168k.

Related, but not quite the same. The solution to this problem can have more than one loop, whereas TSP definitionally has only one. Not quite sure how that changes the difficulty or the tools that you’d use.

And if that's the case, the LogEvent objects will end up automatically promoted to OldGen.

Why do you think this would happen? There's no mechanism that makes young gen objects that reference old gen objects (or are referenced by old gen objects) get promoted faster. You have to survive a certain number of collections.

1. A lot more than 1000, you're off by more than one order of magnitude. It's definitely beyond my level of graph theory knowledge (undergrad level) but looking at the paper, it's not using any crazy machinery, and it's less than 3 pages.

2. Those people will say whether it's a good proof or not. We have other examples of interesting proofs from AI, we're really beyond the point of arguing whether it can produce any interesting math (though it seems to do much better at combinatorics than anything else).

rejected any notion of utility. It would be fundamentally wrong for you to ask what's the value of solving the Erdős–Hajnal conjecture; the value is that it's solved.

I disagree. Mathematicians care about the utility of a result. It is just that they regard mathematical understanding as a valid type of utility, and that can be arbitrarily far removed from practical utility. But a proof that doesn't help anyone understand anything interesting is not valued. I could go out and define some pointless construction and create proofs about it immediately. It would only matter if I connect it to some other subject of interest within math.

I would argue that mathematical understanding is valuable for extrinsic reasons, but it is true that by the time you're a math grad student, you're usually willing to pursue it for no external purpose.

Although not a mathematician, Daniel Dennett had a wonderful example about higher order truths of "chmess". https://personal.lse.ac.uk/robert49/teaching/ph445/notes/den...

Personal and professional are not mutually exclusive.

If I criticize your code, that is a professional criticism.

If I criticize your code and say it reflects your consistent carelessness and stupidity, it is also personal.

If I say you fabricated something, then that is a personal criticism, it alleges an ethical violation. In a professional context, it's also a professional criticism (every profession has some ethical standards).

Limiting the options an LLM has does not turn it into a menu, because it can create infinite combinations/chains of behavior based on the items that it has.

Of course, that power also makes it harder to anticipate security issues--if you can't solve prompt injection, you have to reason as if every thing you allow the LLM to see is an API that an attacker has access to.

However, there are still necessarily going to be middle points where the LLM is more capable than a menu.

Maybe there's an argument that a lack of rising profit margins in non-tech companies is a bad sign for AI, but this article doesn't make it. Why can't we have a red-queen's race where non-tech companies are implementing AI, but it's not increasing the total profits of those sectors, just meeting rising customer demands/fighting over the share of existing profits? (Never mind that if you look at that chart, profit margins aren't static to begin with, so you can't isolate AI impact from normal fluctuation).

Now, on the first order point, I agree that non-tech companies seem to be taking longer to see results from AI, even if the argument was bad.

I work on SaaS for the logistics space, and I feel like prior to the end of 2025, almost all the discussion about AI for logistics was vaporware, starting this year, companies are actually trying to deploy agents, and we'll start finding out what the ROI is later this year or next.

Although it's not the most important thing here, I love the IQ badge on the right side of the screen. At IQ 200, the AI is finally qualified to carry a pager for production.

I'm in the pagerduty lineup, and shockingly, my IQ isn't even a mere 185.

Sounds like the real number is 454x faster, not 70x. Checkmate, nerd!

True, but it's not relevant because that isn't how we actually train LLMs for use as quasi-intelligent tools. We specifically do not want the model to be able to just memorize its input, which is what your process requires.

Many things about the process are similar, so there's some analogy, but it just isn't the same.

A shift is happening among major AI labs, who are becoming increasingly skeptical of endless parameter count and training data scaling. The limits of this paradigm were put on the world’s stage when Claude Fable 5 was restricted by the US government just three days after its release, marking the first US AI ban stemming from national security. One of the biggest models in the world was banned because a single jailbreak was too much of a risk.

Such a weird thing to start with. The legal status of Fable does not mean that it's not intelligent. If anything, the problem is the opposite, someone thinks it's too intelligent (and/or that Anthropic wouldn't share its last gen intelligent models on the terms the government demanded).

I think this is an analogy that's been taken far too far. The output of intelligence just isn't compression, that's memorization. The role of intelligence is to generate novelty.

It's true that LLMs do something that looks very compression like in their weights, but it is lossy, and it has to be--if you're not lossy, you've overfitted the corpus, and that's bad. Post-training takes this even further, because you're not doing anything that looks like training on a specific corpus, you're exploring in a wider space of text. That text doesn't even concretely exist until you start exploring it.

I'm sure there must be a serious attempt to pursue this analogy that isn't just handwaving, but I haven't seen it.

I’m a hallucination. None of these are me.

Perhaps the closest is DeepSeek v4:

Hyperpape is a user on the LessWrong forum, known for thoughtful comments on rationality and philosophy.

I studied philosophy, so maybe, except I don't post on LessWrong, and I'm not a rationalist.

https://www.intheweights.com/p/hyperpape

This article is inane.

The law seems...truthy, though I find it a little too underspecified to assess.

The conclusion is true, and I'll even overlook it being a slight strawman. Your "long genius"[0] CEO cannot talk to AI and get a full business overnight. That's true.

But how does that follow from the "law"? The article admits that you can shift complexity to a complex algorithm or information processing system that the consumer doesn't touch.

You know what's a incredibly complex information processing system? GenAI.

So we have a reasonable conclusion, an ok "law", and no real connection between them. It was an inane non-sequitur.

[0] Not actually a genius.

First sentence:

In my Ottawa life, every Tuesday evening, I take two gym classes back to back—boxing and the pompously named “body sculpt,” which makes me discover muscles I didn’t know I had.

The em-dash matches how you'd speak out loud.

You'd say "I take two classes every Tuesday back to back, boxing and 'body sculpt'. Weird name." (Parts of that sentence did flow oddly, but not because of the em-dash).

Grammarians say you can't make those separate sentences without adding some extra words, and because of blah-de-blah-blah-blah, someone might say you can't join them with a comma. So we have an em-dash.

Rewriting the sentence would make it flow less naturally, not more.

The Mac is a pre-existing platform that is both more capable than iOS, and had an existing user base that used apps that had much greater access. Apple’s attempts to lock down the Mac have met with poor adoption.

In exchange, it also less secure, less user friendly, and less popular.

The curve of AI improvement will continue at the current pace

I guess this is trivially true if you say "maximalism" (hell, the maximalists think it will speed up as the AI becomes a super-AI-researcher), but as long as the rate of change is positive and not miniscule, it's hard to predict what 2035 looks like in software development.

These things are very hard to quantify, but making the progress that happened from Jan 2025-December 2025 repeat twice in 10 years would be enough for me to say I couldn't predict the day-to-day of a software engineer in 2035.

We need a companion to "IN MICE", which is "IN EVALS".

I don't think this is bad research, but you have to understand how far it generalizes. I'm not saying that evals are useless, we need to do our best to produce good benchmarks. But benchmarks are always going to lag pretty far behind real world applications.

According to the paper, “Of the 22 vulnerabilities, five were level-based, meaning that the default weak isolation level led to the anomalies behind the vulnerabilities. The remaining 17 were scope-based, meaning that the database accesses were not properly encapsulated in transactions and concurrent API requests could trigger the vulnerability independent of the level of isolation provided by the database backend.”

I don't want to commit to a real opinion, but the cynic in me sees a bitter lesson you could take from this is that the database should default to a low isolation level--the damn developers aren't even using transactions right, so why waste performance handling transactions in the strictest possible way?

I wonder what the impact of the rising base rate of employees with college degrees is. In 1992, a fresh college graduate had better educational attainment than 42% of the labor force. In 2016 (latest date I found numbers for), that was down to 32%. https://www.bls.gov/spotlight/2017/educational-attainment-of...

That shifting distribution would somewhat reduce the advantage of a college degree against the average member of the labor force.

If AI tokens were so magical in creating new value in developing software applications generally, they wouldn't be selling tokens directly. They'd hoard the tokens are use them to dominate SaaS software in any industry they want.

This doesn't follow at all. Anthropic's revenue is growing 10x year over year selling tokens. Their tokens can be super magical, let them enter established industries and displace incumbents, and get 100% annual growth in those industries, and they would still be better off prioritizing selling tokens, because it's a great business.

What your argument shows is that there are limits. Their tokens are not quite powerful enough to make infinite money instantly in every area of software. Admittedly, that does seem true.

Management is very prone to fads. The current fad is that middle management is useless. Tomorrow, they'll discover the idea that organizations can have employees "working hard" on things that no one cares about, and that someone actually needs to work on focusing that effort.

Of course, the truth is you can have too many middle managers or too few (it really was bad that in 2017, the biggest achievement was "growing headcount"). But fads have a tendency to overcorrect.

Even from the most purely instrumental perspective, what we care about is our ability to make use of correct answers, which is quite distinct from the possession of correct answers.

There are many theorems that aren't directly interesting, but whose proof requires techniques that are of substantial further interest, that lead to new domains, and/or new practical applications. Simply being handed a proof for those theorems isn't enough--we require the ability to apply those techniques in the real world, or discover further areas of mathematical research that build on that proof or its techniques.

It may be that AI can build on its own work for the long-term, but so far, AI does best at exploration in areas that have precisely specified and measurable goals. Actually creating understanding, and making use of mathemtical results outside of pure mathematics is more challenging than simply creating proofs.

I think the field will figure out how to make use of AI, and it will be better off for it. But that is not the same as just saying "answers good, grog want more answers."

Admittedly, that's right, and makes sense for that use case. But as others have pointed out, killing the user's web browser while they're using it is equally painful.