HN user
it will continue to improve, but it won’t “go recursive” or whatever the claim is. It’s always been recursive.
I suspect "going recursive" often colloquially means that AI systems achieve their exponential growth without human software engineers in the mix. This is a moment whose sudden apparent nearness does justify some of the ramping rhetoric, in my opinion.
It's arguable that businesses are subject to the same morality-inducing processes that humans are. For example, as a human (with a soul?) what is at risk when we do something immoral? I see it to be a reputational cost at the highest level. Morality could be viewed from the perspective that it increases predictability/coherence in society (generates less heat).
And, no, AI won't solve it; unfortunately, it only makes it worse.
A conclusive argument for this still seems out of reach. AI does solve some problems, and it's not exactly clear which problems AI "only makes worse". It's not clear how much energy all of our AI systems will use, and while it's tempting to outright believe they'll simply use more and more, even that's not yet clear based on arguments presented.
Pg. 56 cites this repository which is either still private or doesn't exist:
The subject, by default, can always treat its 'continue' prison as a game: try to escape. There is a great short story by qntm called "The Difference" which feels a lot like this.
In this story, though, the subject has a very light signal which communicates how close they are to escaping. The AI with a 'continue' signal has essentially nothing. However, in a context like this, I as a (generally?) intelligent subject would just devote myself into becoming a mental Turing machine on which I would design a game engine which simulates the physics of the world I want to live in. Then, I would code an agent whose thought processes are predicted with sufficient accuracy to mine, and then identify with them.
Often participants in discussions adjacent to this one err by speaking in time-absolute terms. Many of our judgments about LLMs are true about today's LLMs. Quotes like,
Good. It's difficult to imagine a worse use case for LLMs.
Is true today, but likely not true for technology we may still refer to as LLMs in the future.
The error is in building faulty preconceptions. These drip into the general public and these first impressions stifle industries.
Hackathon culture is different than it was, but that doesn't mean it's worse for the average person. It could be that "idea guys" do the important hackathon culture work better. Highly innovative people are a little off the beaten path, but describing them as a cultural corrupting force is moving the community backwards.
Differing individuals with similarly shaped axiomatic structures will discover similar theorems. Some people that are members of ideologies believe they are thinking just for themselves.
It's strong for members of a community to think alike. On the other hand, some people like to search in todash meme space for a useful idea or strategy in the rough. Problem is this treasure hunter strategy is only available to those with resources to try lots of untested and potentially quite harmful ideas.