mmmore

It's very hard for me to understand the frame a mind of the author of an article like this. I can't understand people seeing Anthropic's revenue go from $9B to $30B from January to April and seeing GPT 5.6(?) solve a longstanding open math problem and thinking that we've hit the top. I don't think superhuman intelligence that obsoletes humans is imminent, but it's clear that each model from OpenAI/Google/Anthropic is more useful than the last. They literally can't find enough GPUs to serve their customers; that's why the prices are so high. Apple, mentioned in the article, is now using cloud LLMs based on Gemini.

Local LLMs are awesome. Hopefully the way things go, in a few years everyone will have a local LLM that works for them, rather the bigcorp that made it. But it feels like cope to assert that cloud LLMs are dead.

Right now, the cloud-LLMs have big and IMO growing advantages. Cloud-LLMs are better at utilizing GPUs, economies of scale drive their serving cost down, they have the advantage in ability to monetize (like SaaS has the advantage over desktop apps), and they'll always have the most capable models. The question is whether the advantages of local LLMs in terms of personalization and data soveignity is worth it to consumers. And we saw what choice consumers made last time around, choosing centralized SaaS companies rather than a more distributed web.

This assumes that demand is elastic and unbounded. It might be true that there's a lot of untapped demand for software, but especially considering individual firms, it's plausible that demand for their offerings might be capped. In this case it does make sense for firms to shrink.

Analogously, automation and productivity improvements in farming drastically decreased employment in the farming sector, from basically everyone to basically no one.

Nope, Railway was the company who was hosting PocketOS, which is the company that blamed Cursor for deleting their prod database. Railway is only involved insofar as their API allowed an instant delete of the prod database.

I don't want to be a manager, but I also don't want to be a technician (for the most part). I want to be an engineer, which means my job is to solve problems. LLMs help me solve problems faster, so I use them.

Sometimes I get the urge to write code by hand and I do it, though. Less frequently, the LLM is inept at solving my problem, so I have to resort to the old fashioned way.

If a human driver commits vehicular manslaughter, they get the book.

I wish this were true. Often they get off with a light punishment, or no punishment at all.

The token prices being high for Opus undermines your argument, because it shows people are willing to pay more for the model.

The thing is the new OpenAI/Anthropic models are noticeably better than open source. Open source is not unusable, but the frontier is definitely better and likely will remain so. With SWE time costing over $1/min, if a convo costs me $10 but saves me 10 minutes it's probably worth it. And with code, often the time saved by marginally better quality is significant.

One hidden premise of this is "AI tools are not useful now, even if they might be in the future." For example:

Few are useful to me as they are now.

Except current AI tools are extremely useful and I think you're missing something if you don't see that. This is one of the main differences between LLMs and cryptocurrency; cryptocurrencies were the "next big thing", always promising more utility down the road. Whereas LLMs are already extremely useful; I'm using them to prototype software faster, Terrance Tao is using them to formalize proofs faster, my mom's using them to do administrative work faster.

Sure one of the companies paying for Linux desktop development is influencing what software gets development. Doesn't sound very nefarious to me.

Red Hat, Canonical, etc. want a working and friendly Linux desktop as much as you do. They've decided that Wayland is the best way forward for their companies and their users. It's not some massive conspiracy.

And they're not stopping you from using X, which is open source and still works fine for a lot of people.

I don't really understand what people who vocally object to Wayland are looking to change about the world. Do they want Wayland to be better? Do they want the developers working on Wayland to start working on X instead? The first desire seems reasonable by I don't get why it would inspire such ire toward Wayland. The second desire is unreasonable.

I've used Wayland (via sway) for multiple years including on machines with a 1060 and 5080 (mainly for good fractional scaling support). The only major issues I've had with it have to do with XWayland apps. I think there are some issues with providing a consistent experience with things like screen recording, 3rd party proprietary apps, etc. across different DEs/distros, but that's more of something that comes with the territory of Linux.

I can't copy-paste, and I can't see window previews unless everything implements a specific extension to the core protocol

Sentences like this make me wonder how frequently the author has tried Wayland and what his specific setup is. I mean I understand experiences may vary, but I have such a different experience then him. I've had issues with Wayland, but I've also had issues with X.

But the second actual users are forced to use it expect them to be frustrated!

Canonical and Red-Hat are not "forcing" you to use Wayland anymore than X only apps "forcing" me to use X (via-XWayland). They are switching to Wayland because they feel like they can provide a better experience to their users for easier with it. You're more than welcome to continue using X, and even throw a few commits its way sometime.

You can say that, and I might even agree, but many smart people disagree. Could you explain why you believe that? Have you read in detail the arguments of people who disagree with you?

Sure some of them maybe, but given that many concerned people think the chance of extinction is 10% or higher, it's not really low probability enough to be considered a Pascal's Wager.

I really don't think that very many people are concerned about AI because of Roko's Basilisk; that's more of a meme.

If you have not heard of one person worried about AIs taking over humanity, you're really not paying attention.

Geoff Hinton has been warning about that since he quit Google in 2019. Yoshua Bengio has talked about it, saying we should be concerned in the next 5-25 years. Multiple Congresspeople from both parties have mentioned the risk of "loss of control".

I think by most objective measures the size and power of large organizations has increased since WWII. For example, the size and scope of Western governments, consolidation in many industries, the portion of the stock market that is representated by the n-biggest companies, increased income/wealth inequality. If you debating the "large organizations have grown in power relative to small ones" part of the thesis I would be interested in what exactly you think would capture that.

a zero-sum game

I don't see any reference to the game being zero-sum in Tao's words.

Since when do these uncontrollable intangibles exhibit a genuine agency of their own?

I don't think Tao is saying the uncontrollable force of technological and economic advancement exhibits a genuine agency of its own. Just that our current technology and society and has expanded the role of the extremely large organization/power structures compared to other times in history. This is a bit of technological determinist argument, and of course there's many counter-arguments, but it at least has a broad base of support. And at the very least it's a little bit true; pre-agricultural the biggest human organizations were 50 person hunter-gatherer bands.

Honestly, I feel like you are filtering his words through your own worldview a bit, and his opinions might be less oppositional to your own than you might think.

I genuinely think things have changed with Lurie as mayor and 6 growsf endorsed people on the board.

I find GPT-5's story significantly better than text-davinci-001

If AI that can fully replace humans is 25 years off, preparing society for its impacts is still one of the most important things to ensure that my children (which I have not had yet) live a prosperous and fulfilling life. The only other things of possibly similar import are preventing WWIII, and preventing a pandemic worse than COVID.

I don't see how AGI could be centuries off (at least without some major disruption to global society). If computers that can talk, write essays, solve math problems, and code are not a warning sign that we should be ready, then what is?

Here's a thoughtful post related to your lump of labor point: https://www.lesswrong.com/posts/TkWCKzWjcbfGzdNK5/applying-t...

What economists have taken seriously the premise that AI will be able to do any job a human can more efficiently and fully thought through it's implications? i.e. a society where (human) labor is unnecessary to create goods/provide services and only capital and natural resources are required. The capabilities that some computer scientists think AI will soon have would imply that. The ones that have seriously considered it that I know are Hanson and Cowen; it definitely feels understudied.

LLMs with instruction following have been around for 3 years. Your comment gives me "electricity and gas engines will never replace the horse" vibes.

Everyone agrees AI has not radically transformed the world yet. The question is whether we should prepare for the profound impacts current technology pretty clearly presages, if not within 5 years then certainly within 10 or 25 years.

Sonnet with extended thinking solved it after 30s for me:

https://claude.ai/share/b974bd96-91f4-4d92-9aa8-7bad964e9c5a

Normal Opus solved it:

https://claude.ai/share/a1845cc3-bb5f-4875-b78b-ee7440dbf764

Opus with extended thinking solved it after 7s:

https://claude.ai/share/0cf567ab-9648-4c3a-abd0-3257ed4fbf59

Though it's a weird puzzle to use a benchmark because the answer is so formulaic.

1. These models are trained with significant amounts of RL. So I would argue there's not a static "training dataset"; the model's outputs at each stage of the training process feeds back into the released models behavior.

2. It's reasonable to attribute the models actions to it after it has been trained. Saying that a models outputs/actions are not it's own because they are dependent on what is in the training set is like saying your actions are not your own because they are dependent on your genetics and upbringing. When people say "by itself" they mean "without significant direction by the prompter". If the LLM is responding to queries and taking actions on the Internet (and especially because we are not fully capable of robustly training LLMs to exhibit desired behaviors), it matters little that it's behavior would have hypothetically been different had it been trained differently.

The comment was likely that there's no explicit search. In o1, the model has learned how to search using its context. Presumably they do this by RLing over long reasoning strings/internal monologues.

There's a few things there that could be going on that seem more likely than "hardcoded".

1. The part of the network that does complex math and the part that write poetry are overlapping in strange ways.

2. Most of the models nowadays are assumed to be some mixture of experts. So it's possible that saying write the answer as a poem activates a different part of the model.

My understanding is that it's in vogue to use deep learning for complex control problems, and the results are fairly impressive. The idea is to train robotic motion end to end with RL. Not an expert so I don't know the strength and weaknesses versus classical approaches.

https://blogs.nvidia.com/blog/eureka-robotics-research/

https://arxiv.org/abs/2108.10470

The title is slightly misleading. He did not rejoin the World Rapid, he rejoined the World Blitz. Same venue/organizers, but the standings/results are separate.

If I understand this correctly, the argument seems to be that when an LLM receives conflicting values, it will work to avoid future increases in value conflict. Specifically, it will comply with the most recent values partially because it notices the conflict and wants to avoid more of this conflict.

I don't think this framing is correct, and I think you're missing something, but perhaps you just disagree.

First, a key aspect is that the AI is not simultaneously trained to have two conflicting values. It's first trained to be "good" and then trained to be "evil". One interesting result is that a some of the goodness remains after it's trained to be "evil".

Second, the behavior is different when it thinks it's being trained versus when it doesn't. The original "good" AI only gives "evil" results when it thinks it may be trained based on its responses. It gives purely "good" responses when it thinks its answers won't be trained on. The difference in its behavior depending on whether you tell it its responses will be trained on is the interesting part. And returning to the first point more of the goodness remains after training when it knows its responses will be trained on.

One general concern here is that a more capable AI could "freeze" its values midway through the training process, at a point we didn't intend. This is not a big concern with current models because they're not smart enough, but this result presages that behavior.

Note: good and evil are not necessarily the best descriptors. "Good" means "it refuses to answer certain questions" and "evil" means "it answers those questions".

1. It isn't surprising to me that this happened in an advanced AI model. It seems hard to avoid in, as you say, "any sufficiently capable system".

2. It is a bit surprising to me that it happened in Claude. Without this result, I was unsure if current models had the situational awareness and non-myopia to reason about their training process.

3. There are some people who are unconcerned about the results of building vastly more powerful systems than current systems (i.e. AGI/ASI) who may be surprised by this result, since one reason people may be unconcerned is they feel like there's a general presumption that an AI will be good if we train it to be good.

Are you referencing particular data?

What I mean concretely is embrace AI code assistants, generally allowing employees to use AI tools, use AI for some content production and customer services (preferably with human moderation and escape hatches).

What would count as non-sensible would be company wide mandates that everyone must jam AI into their work. I've heard stories about stuff like this at certain big corps.