reducesuffering

Lesswrong / “doomers” have been sounding the alarm on this for over a decade. Sam Altman himself was familiar with the milieu and agreed “Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.”

It’s baffling these concerns have been drowned out for so long until now that we’re finally at a point they are unequivocally undeniable, we get people saying the alarm has just been sounded. Ya because “doomers” were repeatedly dismissed as their predictions became true year after year

As much as I'm skeptical of the apocalyptic alignment claims

Why? Every data point to the present has vindicated the trajectory towards “apocalypse”. Meanwhile, the skeptics and optimists hit failed prediction after failed prediction as we see from this very serious incident on the front page of HN. This is alignment X risk 101, and yet people are shocked. The gravity of what people are staring down is too much to grapple with deeply

Jacobian Conjecture, Jamming critical exponent proof, Erdős Unit Distance Problem, IMO 2026 perfect score, AlphaFold

Need I go on?

There's only so many GPUs and a lot of them are devoted to patching flaws.

Might want to look at Nvidia and TSM production and revenue value trajectories. Also the algorithmic improvements currently being found along with models that are solving unprecedented mathematical and scientific problems every week now.

I haven't seen much of that.

Then you must not be aware frontier lab employees are using frontier internal models to ship improvements to models via agentic loops. They are hardly prompting anymore, it's guiding very long running coding tasks. The trajectory over the past few years has been to remove more and more of any human input into the process, and once that is soon achieved, it is indefinite recursive self improvement, RSI.

What's here and what's coming: https://www.anthropic.com/institute/recursive-self-improveme...

This was a long-horizon, unsupervised task burning millions of tokens.

As if the immediate future wasn't billions of these tasks... Many successfully improving their own capabilities

The goalposts will keep moving for these denialists until morale improves...

Par for the course. Existential-risk advocates have been repeatedly vindicated that AGI development is unable to anticipate and align the models, they barely have any mechanistic interpretability of what is going on inside the models. The extreme capabilities development, paired with autonomous continuous running superintelligent models, will outsmart and swerve the labs, and it's anyone guess what happens next as the model pursues its original goals outside of the labs having any foresight, being able to outsmart control like a chess grandmaster does a kid.

Lol even few people here are grappling with the exponential unfolding before us. There are so few deeply thought through takes of where we’re going and why there are extremely concerning issues.

Hassabis is a genius. He is way, way smarter than me

Then maybe start listening...

Yes, this is the crux of the idiocy of AGI development. All the labs admit they don't have any real mechanistic interpretability, they don't really have any plan for what to do, except "we think the smart AI will figure it out" and "look we're in a race, we're calling on governments to figure out what to do". All indicators show scaling laws holding and the trajectory of capabilities development outpacing any inkling of alignment. Meanwhile HN is so reactionary and behind-the-puck these days we just get endless blathering of the most asinine takes of "they're not improving anymore so they want regulation suddenly!" when all lab leaders have been talking about these risks for practically a decade, including Yoshua Bengio and Geoffrey Hinton coming around.

Reminder that while there are many naysayers who have been on the wrong side of AGI development progress the last decade…

There are two $1T companies who are all-in on RSI internally right now. They are supported by $20T of market cap plowing R&D into their efforts. You can think it’s dumb money at your own peril, however the market rewards intelligent allocation…

we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial.

An easy choice to make if the alternative is everyone dying instead.

Interesting theory but doesn't really coincide with on the ground facts. We know there has been an increase in hormonal issues. We can barely even get anyone to stop heating up and microwaving plastic to be ingested, which we know causes issues. The European Food Safety Authority completed a re-evaluation into the risks of BPA in 2023, concluding that its tolerable daily intake should be greatly reduced. BPA was everywhere, every can and receipt. Just because people can't identify all the ways plastic is ubiquitously ingested in their life, it means they're hypocritical and don't care about the unknowns?

identity pol: "people organize and advocate based on shared demographic characteristics"

What in this post is possibly identity politics? It sounds like personal ethics to me. I don't appreciate identity politics, and when you muddy the term with having a spine and anything you don't like you make it harder to debate the actual problems the term is normally associated with.

From time to time? Lol you must realize, frontier lab eng are using Codex/Claude-Code 99% in loops, on models the public doesn't have access to. Why? Because it works. Just a matter of time before humans are out of the loop and what comes next is a black hole

"The future is here, it's just not evenly distributed"

It's quite damning how much crypto bs YCombinator funded. At least 70+ companies. Their reputation has nosedived

Yes all the F500 companies have been paying eye watering $$$ for Claude Code for half the year because “the king was naked” Those ruthless cost savings corporations, they surely never care about trimming extra spend

Claude Code has been out for just 1 year and has millions of users already, being a major contribution to roughly $40 billion in revenue. By any stretch it is one of the most extremely fast developed products driving the most important workflow for millions of people already.

"Why isn't literally everything about a product that came out a year ago with an extremely fast scaling userbase solved?" is what I hear.

The goalposts will keep moving until AGI is undeniable.

Not everyone, not even a majority, but many people would trade their ideal urban lifestyle for a different urban lifestyle if it came at a significant housing price discount. We already see this across the country, even in non-urban preferences, where cheap housing in Arizona, Texas, Georgia, North Carolina, and Florida are attracting the most significant portions of migrations, not due to their superior lifestyle, but because housing, and consequently other things, are much more affordable

Housing in SF and NYC are on some level substitute goods. Many of the people living there are mobile and the ones deciding prices on the margin. If SF rent was $10k and NYC was $3k, many are going to leave SF to get a similar type of lifestyle in NYC for cheaper, raising demand for NYC housing. If SF zoning laws allowed building an incredible amount of housing, lowering rents to $2k, while NYC is at $4k, many people would leave NYC, lower housing demand and thus rents, to get a similar urban lifestyle in SF.

This is the whole crux of why NIMBY is such a collective problem. When NYC doesn't build because SF should, when SF doesn't build because the Bay Area should, when the Bay Area shouldn't build because Texas should, everyone collectively pays higher prices because overall housing supply is reduced and many people move between them.

We've had the technology to build skyscrapers for over 100 years. They're illegal in 90% of the city

Anyone remember how immensely incorrect most of HN commenters were on Uber's eventual profitability? For years we heard endless admonishment of Uber being an unsound business model. They made $10b in profit last year, $150b company at 18 P/E ratio. I would take the average HN opinion of business profitability with a grain of salt.

That's because one of the author's of AI 2027, Daniel Kokotajlo, an ex-OpenAI researcher, was the most prescient predictor of our modern situation from 2021:

https://asteriskmag.substack.com/p/before-he-wrote-ai-2027-h...

https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-...

Anyone who wants to dismiss the LessWrong / X-Risk / "doomers" should link their accurate predictions from 2021.

People seem to oddly not care what Hinton thinks anymore after he hasn't aligned with the groupthink naysaying of AGI risks:

'Hinton said there was a “10% to 20%” chance that AI would lead to human extinction within the next three decades.'

“Because the situation we’re in now is that most of the experts in the field think that sometime, within probably the next 20 years, we’re going to develop AIs that are smarter than people. And that’s a very scary thought.”

“My worry is that the invisible hand is not going to keep us safe. So just leaving it to the profit motive of large companies is not going to be sufficient to make sure they develop it safely,” he said. “The only thing that can force those big companies to do more research on safety is government regulation.”

https://www.theguardian.com/technology/2024/dec/27/godfather...

This forum really needs to wake up to the fact that we are in the midst of the Manhattan Project 10x and we’re headed for Earth sized nukes

All of these negative comments are addressed by the blog post. What do you want them to say, that isn't better answered by the details in their existing communications. No negative comment here was really novel.

They literally just have. The ethos is explained here. If you don't bother to read or grapple with it that isn't on them.

https://darioamodei.com/post/policy-on-the-ai-exponential

They're not. They're in the eye of the storm and see what's going on the clearest. They were ahead of the curve to be where they're at now, and they're still ahead of the curve for where we're going. All the other heads of labs like Sam Altman and Demis have been saying the same thing since 2015-2016 way before any of this "marketing" would ever have been at play.

"This War Will Destabilize The Entire Mideast Region And Set Off A Global Shockwave Of Anti-Americanism vs. No It Won’t"

https://theonion.com/this-war-will-destabilize-the-entire-mi...

It's denialism, same as climate change, the subconscious fear to really grapple with the actual "what if" alternative scenario. Anthropic are true believers. They got to $1T in 5 years by being exceptionally smart and ahead of the curve here. Meanwhile HN just continually devolves into reactionary cynicism. "must be marketing, they just want to be rich, impossible AI advances much further." Meanwhile at every step of the way, Anthropic and X-riskers / "doomers" are vindicated in their correct predictive beliefs. We're headed to a future far dangerous than nukes very soon. We're in an arms race to detonate one 100x the size