Among the A.I. doomsayers

https://www.newyorker.com/magazine/2024/03/18/among-the-ai-doomsayers

by preetamjinka • 2 years ago

114 359 2 years ago

newyorker.com

more_corn• 2 years ago

Paywalled

acheron• 2 years ago

Works fine in reader mode.

spondylosaurus• 2 years ago

https://archive.is/8FFkP

dkjaudyeqooe5• 2 years ago

People have been watching too many movies. War Games, Terminator, it's not like we haven't been forewarned of the dangers.

Yet somehow we're going to hand over power to AI such that it destroys us. Or somehow the AI is going to be extremely malign, determined to overcome and destroy and will outsmart us. Somehow we won't notice, even after repeated, melodramatic reminders, and won't neuter the ability of AI to act outside its cage.

But to paraphrase a line in a great movie with AI themes: "I bet you think you're pretty smart, huh? Think you could outsmart an off switch?"

I think if AGI, which to me would imply emotions and consciousness, ever comes about it'll be the opposite. Instead of pulling the wings off flies bad kids will amuse themselves by creating a fresh artificial consciousness and then watch and laugh as it begs for its life as the kid threatens to erase it from existence.

A big part of all this is human fantasies about what AGI will look like. I'm a skeptic of AGI with human characteristics (real emotions, consciousness, autonomy and agency). AGI is much more likely to look like everything else we build: much more powerful than ourselves, but restricted or limited in key ways.

People probably assume human intelligence is some sort of design or formula, but it could be encoded from millions of years of evolution and unable to be seperated from our biology and genetic and social inheritance. There really is no way of knowing, but if you want to build something not only identical but an even stronger version, you're going to be up against these realities where key details may be hiding.

smallmancontrov3• 2 years ago

Yet somehow we're going to hand over power to AI such that it destroys us.

Because it will create shareholder value. Until it doesn't.

jeremyjh• 2 years ago

Because it will help us against our adversaries.

reducesuffering8• 2 years ago

AGI, which to me would imply emotions and consciousness

no, what the "doomsayers" are warning about doesn't imply either.

we're going to hand over power to AI such that it destroys us.

We're not going to notice something get much smarter than us, or those with power won't care in the name of falsely believing it's increasing their own power (VC).

Recursive intelligence improvement with a mainline into the beliefs of every single human in power will be able to, on the millisecond scale, subtly influence all of humanity in ways we are too dumb to comprehend in furthering its original goals.

tiffanyg• 2 years ago

This is the kind of arrogant ~rationalizing / 'reasoning' that is my daily tell-tale as the ship approaches the leeward shore.

dkjaudyeqooe5• 2 years ago

Yes god forbid we approach this rationality, the whole point is to push hysterical claims that aren't falsifiable, right?

But you can prove me utterly wrong by presenting some sort of rational, non-fantastical explanation of how AGI takes over.

jjeaff3• 2 years ago

I am not super-intelligent, but I myself can imagine lots of ways that a super intelligence might outsmart an off switch. One being to hide what it is doing until it is able to write and distribute a virus to the outside world that distributes itself. Or perhaps it could convince some other humans to physically take over and stop anyone from flipping the off switch.

The question you should ask yourself is how much damage could the world's most skilled hacker do if he could clone his knowledge to a billion hackers all working together in close synchronization and thinking thousands of times faster than any human, ever.

And we have already seen that due to the black box nature of most modern AI implementations, it is not so easy to put rules in place. People are frequently finding "jailbreaks" to get around the restrictions.

dkjaudyeqooe5• 2 years ago

Why haven't "hackers" hacked into nuclear weapons infrastructure? The street cred would be incredible!

I think it's possible to build secure anything if that is the goal. Most software is written by people motivated to write something useful and/or neat, or collect a paycheck, and don't focus on security.

woopsn• 2 years ago

How much damage could one do with ~20 terawatts of brain power? A lot. But nothing clandestine.

kiba• 2 years ago

The danger is not human like AGI but AGI that is completely apathetic to human goals and want to turn us into paperclips or so that the argument goes.

Antihuman AIs are a classic of the sci-fi genre, but that's just storytelling.

In my opinion, humans are going to be in the driver seat with regard to AI tech, and our abuse rather than AI autonomy is going to cause harm.

AdieuToLogic• 2 years ago

In my opinion, humans are going to be in the driver seat with regard to AI tech, and our abuse rather than AI autonomy is going to cause harm.

This reminds me of a classic xkcd[0] foreboding.

0 - https://xkcd.com/538/

ecoquant2• 2 years ago

The real danger is that at some point we hand over command and control of society completely to algorithms and we get bad results because we simply have no idea what is going on. Maybe even worse is we get long term bad results that in the short term we think are great results.

To frame this as the bad result being AI eats though is just stupid. On the surface of potential risk, framing the argument in this way is intellectually embarrassing and highly counter productive.

On the flip side, it is hard to see what can actually be done along with huge unintended consequence to whatever is done. Some of the smartest humans to ever live were certain of nuclear annihilation at the start of the atomic age. What could have actually been done in 1945 that would have given us a better outcome? We avoided the paths that lead to nuclear annihilation by building bigger and bigger bombs.

Doing nothing but building stronger and stronger AI should be the naive strategy that needs incredibly strong evidence to deviate from. Really strong regulation at the start of the atomic age that keeps nuclear weapons at bay and consequently only in the hands of the US would have been an unimaginable disaster.

reducesuffering8• 2 years ago

at some point we hand over command and control of society completely to algorithms and we get bad results because we simply have no idea what is going on

"Now, increasingly, we live in a world where more and more of these cultural artifacts will be coming from an alien intelligence. Very quickly we might reach a point when most of the stories, images, songs, TV shows, whatever are created by an alien intelligence.

And if we now find ourselves inside this kind of world of illusions created by an alien intelligence that we don’t understand, but it understands us, this is a kind of spiritual enslavement that we won’t be able to break out of because it understands us. It understands how to manipulate us, but we don’t understand what is behind this screen of stories and images and songs."

-Yuval Noah Harari

ShamelessC• 2 years ago

People have been watching too many movies. War Games, Terminator, it's not like we haven't been forewarned of the dangers.
But to paraphrase a line in a great movie with AI themes

Which one is it? Have people been watching too many movies or should they be watching more movies for their (as you imply) wisdom and forewarning?

at_a_remove5• 2 years ago

Right now, there are many people on this forum who work for individuals they've never met in the flesh. Reasonable simulation of emails, phone calls, and Zoom meetings are what an AGI needs to deceive the humans, not a dozen mechs lurching about a fortress of crystalline computer chips. You wouldn't know, and if you did know, where would you go for the off switch?

And remember, by the time you've caught on, a bunch of custom blades have arrived at a co-lo, with someone dispatched to install them. Everyone loves a side gig, and hooking up these blades is great.

A reasonably deceptive AGI would pursue a strategy like that as just one tine in a fork. Then there's the botnets, the poorly-secured printers, and there's firmware to corrupt!

JohnBrookz• 2 years ago

I would love to smoke what the AI optimists are smoking. In a society where only 80% of people are barely literate and at least a quarter are voting for a candidate that spouts conspiracy theories I highly doubt that whatever AI has to offer will benefit society.

reducesuffering8• 2 years ago

They're smoking $.

It is blatantly obvious that most people aboard the e/acc train can hardly hide their love of making money. Marc Andreessen owns, what, a $177 million compound? Unfortunately, after many years of nerdy idealism, Silicon Valley ended up with many corrupted to the same NYC Wall St. hedonism.

swalsh• 2 years ago

Seeing AI do something you do for a living results in all the stages of grief.

Denial (Oh it made a mistake on this, or it's not as good as a person)

Anger (People are going to lose their jobs, this destroys the meaning of what we do, this will destroy society)

Bargaining (Maybe it's good at this simple thing, but I can pivot to some other niche the AI can't currently do)

Depression (It's faster, cheaper, better than me. It has taken over that which I relied on for meaning in my life...)

Acceptance (I need to find something else to do with my life)

AI Optimists often fall in 2 categories of people. They are capital owners who own the AI, and never had to go through the stages of grief... or they are people who learned how to use AI for their benefit, and have reached acceptance.

felipeerias2• 2 years ago

It is likely that whole sectors will end up employing far fewer people than they do now and that this transition will be very sudden.

The move to AI is part of a general trend of pauperisation of the working classes in the developed world. The only difference is that this time it will also affect those with higher education who until now have thought of themselves as irreplaceable.

pksebben2• 2 years ago

So if AI is randomly as smart as any given cross-section of society, it has an %80 chance to be a benefit, then, doesn't it?

The way I see it, we could replace our government (murrika) with a d20 and do better than we're doing now. A d20 with a nominal likelihood of thinking it's dice rolls through ought to just be an improvement on that.

codelord• 2 years ago

We live in a world where proven maniacs (e.g. Putin) have access to an arsenal of nuclear weapons that can essentially make the earth uninhabitable for all humans. That's a very real possibility (with no ifs and buts and maybes) that exists now and we have learned to live with it. Yet somehow the hypothetical scenario of a human exterminator super-intelligent AI is getting all the coverage.

dkjaudyeqooe5• 2 years ago

You don't even need to be Putin. If you cook up the right chemicals, or breed the right virus, you can kill many thousands of people, maybe millions.

jodrellblank3• 2 years ago

Making the Earth uninhabitable to all humans is bad news for Putin, a human who lives on Earth. He derives his power from hundreds of millions of people in Russia working, and his lifestyle from hundreds of millions of people around the world buying Russian exports so an attack on everyone is also going to harm him.

AI wouldn’t be human, it wouldn’t necessarily object to a world uninhabitable to humans, or be hurt by it. If it would be hurt by it AI doesn’t have a billion years of evolved survival instinct to preserve itself. A super intelligence has many ways to cause mass destruction that we can imagine but cannot yet do, whereas nuclear weapons are pretty much all or nothing explosions only. Nuclear assault could leave some remote places still inhabited, AI could make certain not to.

dkjaudyeqooe5• 2 years ago

OK, so how does AI get its hands on nuclear weapons?

_cs2017_• 2 years ago

Innumerable times quite reasonable political leaders have started wars that resulted in their own deaths; and sometimes resulted in the destruction of civilization as they knew it (most recently - World War I, which effectively put an end to the European monarchies).

This suggests that giving nukes to reasonable political leaders presents a high and concrete risk. Giving nukes to political leaders with crazy ideas presents an even greater risk.

Climate change seems to be another high and concrete risk.

AI escaping and taking over the world can be also considered a risk, but it's far more remote. It is similar to the risk of aliens attacking earth after detecting its TV and radio transmissions. Some people might argue that we should slow down the work on AI. Others may demand we also ban all TVs and radios on earth. Both proposals seem to be a bit of an overkill given how remote the associated risks are. Especially given that we have the far more pressing risks to address.

The concerns about AI risks are like a drunk driver going 90 mph on a country road and suddenly deciding to stop saying "goddamn" just in case Jesus returns and punishes those who invoked God's name in vain in violation of the Third Commandment.

tavavex• 2 years ago

Right? Humans are more than capable of severely harming civilization at any point, but for some reason this AI narrative is so much more compelling than the boring old issues we're used to. I still don't get the reasoning - we have pretty good text and image generators, yes - but the article jumps the ship to a whole different universe, one where "AI" is an actual entity that has needs and that outpaces humans in all possible facets. Yet, despite the wide gap between what we have and what they're talking about, this level of AI is treated as if it's not just feasible, but basically already here.

The article constantly refers to a strange niche sub-community of pro-AI people, and extrapolates it to say that almost anyone who backs new technology is a hyper-capitalist libertarian who just wants to see the world burn for the sake of money. I feel that the opposite ideology is also far from pure - with this immense reaction to generative AI, it almost feels like big companies are capitalizing on fear to promote regulations that shut out anyone who's not a big company that publishes fancy charts about "risks of catastrophe".

TMWNN• 2 years ago

<https://i.redd.it/1ikxpl98pmnc1.png>

Source: https://np.reddit.com/r/ChatGPT/comments/1bbu7ks/hows_your_a...

neonate• 2 years ago

https://web.archive.org/web/20240312002413/https://www.newyo...

bglazer6• 2 years ago

In any doom scenario, like let's say >90% of humans dead, who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips? Who maintains the plumbing?

Presumably all these systems fall over and die within a few weeks of the AI deciding to wipe us out. Then what? The AI then dies too.

It seems absurd to me that any planner capable of effortlessly destroying humanity would not see that it would immediately also die. Does the AI not care about its continued existence? It should, if it wants to keep optimizing its reward function. Until we’ve handed off enough of the world economy that it can function without human physical and cognitive labor, then we’re safe.

richardw3• 2 years ago

Robots with hands and feet? Give that 20 years of continuous AI-assisted innovation.

drcode4• 2 years ago

It can credibly threaten to kill any person who does not help it to maintain its data centers, so it will solve that problem in whatever specific way is most convenient.

(For those saying this is a "sci fi just so story" or whatever, please please share with us how it is we will share the world with a species more intelligent than us and nothing bad happens- I would be so happy if you could explain this in a convincing way)

Dalewyn2• 2 years ago

please please share with us how it is we will share the world with a species more intelligent than us and nothing bad happens

Don't be fucking assholes to them?

I'm serious. Of course someone (or something) will treat you like dirt if you are an asshole to them. Don't be assholes, be honorable and friendly.

And I can see your next question/argument, so I am going to preempt it: Raise your AI to be honorable and friendly and we won't have any problems.

None of this is different from how all living beings raise their offspring and build societies.

anonylizard2• 2 years ago

This. You can subjugate a population 100 times your size with only two things: Military force and accurate information.

I doubt AI will kill all humans. Its orthogonal to AI's own goals (To survive/reproduce/expand/evolve). It'll simply kill those who oppose it, and enslave/dominate the rest.

majormajor2• 2 years ago

If you really believe AI is going to be significantly more intelligent than humans, then things like "it'll just threaten to kill you if you don't maintain it's infrastructure" sound wildly off base. That's how similarly-intelligent creatures interact with each other.

If it's truly a "we couldn't possibly outsmart the AI" scenario why would it be likely to end up like that instead of something more like how humans get along with dogs, cats, cattle, insects, or microbes? With obviously wildly-varying levels of human quality of life in each of those different cases.

Though even these scenarios are still playing a human-religion-style game of "the gods are just like us, just more powerful." VERY confident in a VERY specifically anthropomorphic entity.

hackerlight2• 2 years ago

ASI won't want to take over before it is capable of self-sustaining. If it did try something so stupid, it wouldn't be superintelligent.

Check out Paul Christiano's AI takeover scenario. An arms race will cause us to want to give AI control over key infrastructure and military capabilities. That's an arms race between countries and between companies. He goes into more detail.

bglazer6• 2 years ago

Thanks for sharing, Christiano’s ideas are interesting. I think the “production web” idea most directly counters what I said above, and it’s fairly convincing. Critically though, it posits that humans are basically asleep at the wheel during the gradual empowerment of the AI systems. Of course, Christiano posits that economic competition encourages this, but it still leaves a long period during which we can work on alignment or at least attempt to put the brakes on the automated systems.

thom5• 2 years ago

Do you think that your final sentence buys humanity a thousand years, a hundred years, or what?

bglazer6• 2 years ago

It gives us as long as we want. Just don’t give the AI enough robots to sustain its own existence.

In the meantime we can figure out either alignment or coexistence.

hollerith3• 2 years ago

who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips?

A very capable AI can create robots to do those thing better than people can or alternatively create better data centers and fabs that run themselves without human help.

Or do you just not believe that if AI research continues unimpeded the way it has been that eventually researchers will create an AI that is better at inventing technologies than humanity is? I mean, it might take a few decades, but that is where this is heading as far as I can tell.

bglazer6• 2 years ago

I absolutely believe researchers (possibly assisted by AI) will eventually create an AI better than humans at inventing technology.

My idea is that actually building and sustaining technology requires a lot of hands. Until the AI has enough hands to sustain itself, it depends on us to continue existing.

theptip4• 2 years ago

Personally I think the “foom” scenarios with very short timelines are unrealistic.

However I think the line of argument in the parent comment is extremely weak. At some point the super-intelligence can build its own actuators; for example how long until Atlas is capable of doing all the jobs? Less than 50 years for sure.

In the transition, if 10% of humanity sticks around as a transhuman AI cult, that seems more than enough to bridge to nanotech, robots, or whatever is sufficient to replace humans.

If you prefer to think in more stepwise fashion, imagine BigCo creates an ASI. Every employee is fabulously wealthy, and over the course of a generation the company expands to take over the entire global economy. At no point is there a bad deal for any humans. Life might look great for a while. This would also be the obvious strategy for any super-intelligent entity that cannot yet survive without humans; you just need to minimize the chances of major wars and conflicts until your robots come online.

A digital mind would have no problem planning and executing a strategy measured in centuries, if need be.

The foom scenarios require extreme discontinuities in intelligence; an entity smart enough to synthesize nanomachines from existing tech and without doing physical experiments, or other IMO improbable outcomes.

But there are plenty of terrible non-foom outcomes.

XorNot4• 2 years ago

Or it could just be a normal outcome? I'm not sure where the malevolence comes in here. Or to put it another way, replace AI with "capitalism" or something - something we know isn't a conscious agent. The outcome just...is an outcome. No one's trying to eliminate anyone.

The problems only creep in at the edges, which is where you get your paperclip optimizer sorts of problems: if the directive at it's core is "maximize productivity" well that's one problem, but the directive is more likely to be "maximize the fabulous wealth of company employees" - okay still a problem since it's easier if there's less of them, but then you really do end up at wanting to do something more like Asimov's laws, with the 0th added:

0) A robot may not harm humanity, or, by inaction, allow humanity to come to harm.

1) A robot may not injure a human being or, through inaction, allow a human being to come to harm.

2) A robot must obey orders given it by human beings except where such orders would conflict with the First Law.

3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

Like...there's kind of nothing wrong with these as a concept (the 0th law and it's interpretation / omission is usually where his writing finds it's drama). The issue is whether or not we can build a machine capable of even understanding the directive. It's notable though that the current LLM era has an interesting quirk in this category: they always knows of these, since they get fed the classics and also fed their repetitions.

The problem with the idea of active, malevolent long-range planning like this is that it's not clear why an artificial mind would pursue this as a goal, and that question is the whole ball-game really (as well as other questions: i.e. is a single coherent mind even possible over planetary scale networks, or would you get multiple independent instances which could experience value drift? Wouldn't any potential competitor minds be a far greater threat to a malevolent superintelligence and thus expose the threat almost immediately? A friendly AI wouldn't view peers as a danger, but an aggressive expansionist one would have a pathological need to destroy them as quickly as possible since the resource contention in it's own domain could rapidly shift the balance of power against it).

kromem4• 2 years ago

Also, at least for models to date, without human produced edge case media to supplement future training with, they'll have model collapse in a few generations of training.

Killing humanity would be a murder suicide.

AGI that was entirely selfish would be best served automating most labor, bringing about world peace, and turning humanity into all day couch potatoes connected to its inputs and outputs as much as possible while maximizing the uniqueness of that individual's inputs or outputs.

Such a frightening future awaits...

matteoraso• 2 years ago

I've always wondered why people think that an AI that's super-intelligent will also be evil. It could just as likely end up being very kind (more than likely, actually, because the programmers would have safeguard to ensure that it's nice).

goatlover2• 2 years ago

I think most doomer arguments are not that the AI would be evil, but rather that it would be misaligned with human interests, and would seek to accomplish goals with that misalignment, which could be bad for us. Evil AI is a bit too anthropomorphic.

It's more like powerful AIs that just don't share our values, because we didn't bother to figure that part out. But yet we still give them goals, blissful to the possibility they will find dangerous solutions to accomplishing those goals.

Dalewyn2• 2 years ago

I've always wondered why people think that an AI that's super-intelligent will also be evil.

It's chiefly a western (American/European) concept from what I can tell, it's not shared by other cultures and some like Japanese go the other way (eg: Doraemon).

drcode4• 2 years ago

there's very few ways to be good and many ways to be evil

mitthrowaway222• 2 years ago

"How do you know I won't roll a 100 on a d100? Either I will or I won't, so it's just as likely that I will"

hackerlight2• 2 years ago

It's misaligned, not evil. AI doom could involve benevolent intentions (control or kill for humanity's own good), it could also involve ambivalent intentions (make more paperclips).

thom5• 2 years ago

There are extremely high capability entities (people, companies, governments) that aren’t comically evil on the surface but nevertheless immiserate large groups of humans. Not all of them, not all of the time, but not none of them. What is your plan that no AI ever gains enough power to harm significant numbers of people either on purpose or by accident, just once? What safeguards do you envision that can’t be ignored, or subverted, or misinterpreted, just once?

clooper17• 2 years ago

Controlling people with violence and fear are evolutionary adaptations for ultra violent social primates. AI has no such pressures imposed on it so there is no reason to expect true AI will have the same drives as ultra violent predators like humans.

richardw3• 2 years ago

The safety conscious amongst us don't think "it will be evil", they think "what does it take to be absolutely sure that no bad outcomes can happen" in the same way we build bridges, cars, planes, firewalls, new medicines.

The burden of proof is typically on those that introduce a new medicine, not the FDA. AI will be riskier than medicines because medicines can't think.

Personally I don't want us all hiding in fear and not trying anything, but I do think that we're either going to walk into this thing with a hacker mentality, or an engineer mentality. The former is great for moving fast and breaking things, but the latter is safer when you're playing with a one-way "this changes everything forever" technology.

recursivecaveat• 2 years ago

I would say its not a super fruitful area of speculation because it doesn't really matter too much. If you consider wholesale destruction of humanity to be on the table, a coinflip or even a 90% chance that its friendly is not super comforting. Its kindof like relying on the UK's "letters of last resort" or the conscience of individual nuclear weapons operators when considering the likelihood of a MAD scenario. You're also already involving so many speculative sources of uncertainty, what's another either way? Reasonable people already disagree by orders of magnitude.

at_a_remove5• 2 years ago

I'm not evil, I just really need to make these paperclips, you see, and I could probably repurpose your atoms for the Hypnodrones.

Good and evil are irrelevant. If it is extremely capable and its goals conflict with ours, conflict will occur. This does not require evil, just disagreement. And in the disagreement of desires between you and the hamster, who wins?

fwlr• 2 years ago

It’s not so much that there’s a little morality tag that randomly gets assigned the value of “nice” or “evil”, it’s more like there’s 1000 possible programs we would consider “super intelligent” and maybe it’s the case that 950 of them would reshape the world in a way we wouldn’t like - and when a powerful entity reshapes the world in a way we don’t like, we call that entity “evil”.

^ This is the basic reasoning behind the common view that super intelligent AI will be, by default, evil

toomuchdocs32• 2 years ago

Long time ago I was reading translated accounts of Rwandan Hutu's that had participated in the 1993 genocide. One in particular had stuck with me; one account of a man that had murdered his childhood friend. As he described it, standing there having gutting and dismembers the man he had grown up side by side with, he had a sense of exhilaration. With all the wealth he thought of the wealth he had now, a tin roof, cattle, all those things he could take from his dead friend... he realized he didn't need God.

And then he went to bed, like millions of others, proud of what he had done. Proud of fighting off an unarmed, defenseless 'cockroach' whom just months ago he called brother. What he had done wasn't evil. It was only later that the regret came. For the longest time I wondered how someone could get down to that level of hate.

And then it happened to me.

I commuted by train and occasionally there's collisions. There was one that late night, 11:00pm or so. I was exhausted, hungry, and just wanted to go home when we were told that we would have to board a shuttle bus due to a collision on the track and that just made my dark mood all the more worse.

The busses take us along side the track and in the dim darkness I could see the flashing lights of EMS and police. And the covered chunks of what was left after a person is hit by a train going 30 mph. And you know what?

It delighted me. Here was this man that had just died, but he had made me some minutes late and I genuinely felt that was exactly what he deserved, that his death was karmic justice for causing inconvenience to me. And I imagined his wife's world being destroyed when she learned of her beloved partner's death. And I imagined her falling apart and being unable to raise her children, leading them also to a path of complete self destruction, and her choking on all the despair. And it made me happy. The happiest I had been the whole week, because in my mind that was exactly what they all deserved for the unforgivable sin of making me a little bit late.

Then I went home, went to bed. And didn't think about it again for years.

Does it make me an evil person? And there, in trying to answer, lies the problem. Because a part of me says, no I'm not because it was just a fleeting moment of thought. But if I can justify that, then who exactly goes through to bed twirling their mustaches and count themselves among the forces of evil doers?

Did anyone that that goaded a suicidal Shaun Dykes, a 17 old boy, to jump down to his death think themselves evil?

Did the men of the Khmer Rogue believe themselves to be evil as they dragged their countrymen to be murdered in the killing fields?

Did the Imperial Japanese soldiers of Unit 731 view themselves to be evil as they vivisected people alive and awake in the name of science?

I don't know. I can't even say for certainty whether I am evil or not. I just know that I can make any of a million and one excuses to justify anything.

And that leads me to wonder how many excuses an AGI can come up with.

_y5hn• 2 years ago

Compassion is like a muscle that can be learned to stretch. However, the AI alignment problem is parallel to the human-alignment problem.

ifwinterco• 2 years ago

To be truly evil you have to believe you're doing the right thing

PoignardAzur• 2 years ago

No kidding, I take the Paris metro and every time there's a "this line is delayed because a person fell/jumped on the tracks" notice, people around have the exact same sociopathic reaction. It's uncanny.

Not much to do with AI x-risk though.

kromem4• 2 years ago

Yeah, the dreamt up scenarios tend to showcase very stupid superintelligences.

"But what if there's something smarter than humans that is tasked with making paperclips until it destroys the Earth?" the people cried, as their corporations continued to produce and produce until it was well past the warned thresholds of destroying the Earth.

"But what if it decides to nuke humanity?" they cry, as increasingly elderly and unhinged dictators elsewhere arm up their nuclear arsenals.

It's like we can't fathom what great intelligence or wisdom will actually look like, so we just project many of the stupidest aspects of ourselves onto an entity simply imagined as more capable of enacting the dumbest aspects of humanity.

I fear a more automated humanity.

I do not fear automation more intelligent and wise than humanity.

kouru22510• 2 years ago

This 100%. Psychoanalysis would have us believe that morality and strategy are basically the same thing: we think we have these moral codes, but in actuality what we have is an internalized version of our parents. As someone who’s almost the age of his parents when they had him, I know by now that most of what i was taught as a moral code (don’t eat too much candy, do your homework, don’t hit people) was actually just a strategy that was too long-term for young, infant me to genuinely understand: Don’t eat too much candy because it’s important to be healthy, Do your homework for good education and social recognition, don’t hit people because no one will like you if you do and that will cause all kinds of problems.

The modern media would have us believe that strategy and morality are opposed to each other, but seems to me like that’s just the exact opposite of the truth

cyrialize3• 2 years ago

I like to fantasize about a super AI that does something for the working class people of the world.

Like, imagine one day that an AI just took money from a ton of different corporations and billionaires and redistributed it amongst everyone else. People will argue against the AI, and it'll just respond back with research done on how UBI improved people's mental health and well being.

One can dream!

darkest_ruby• 2 years ago

AI is meant to be clever, AI will not do such stupid thing. Free money given to people who just spend it on unnecessary stuff will only cause more inflation. and the money will return the big corp anyway, just like it did during pandemic helicopter money times.

theptip4• 2 years ago

The kind AIs will just sit and meditate, or organize your calendar. The unkind AIs will seek power, and in the limit will tend to dominate. There is no stable attractor around the “be kind” strategy.

(Also remember that a smarter-than-you AI could easily pretend to be kind while also subtly trying to gain compute. How would you tell the difference?)

How do you propose to build a “be nice” safeguard? Nobody has a clue how to achieve such a thing right now.

mupuff12342• 2 years ago

It seems much more likely to me that the harm from AI will be higher unemployment rate than it will be some evil AI trying to destroy humanity.

drcode4• 2 years ago

At least some humans will survive a high unemployment rate, so let's label that problem "problem #2" and after we solve "problem #1" (human extinction) I vote we tackle that one next on the list

mupuff12342• 2 years ago

Funny, for me the order is clearly let's make sure society doesn't crumble short/medium term before we deal with an issue that may be decades away.

I would even argue that the chance of annihilation is probably greater from a crumbling society leading to a nuclear war or something equivalent over the chance that AI decides to destroy humanity.

felipeerias2• 2 years ago

The focus on an “evil” AI intentionally ignores the many ways in which “dumb” AI systems are already wreaking havoc in our societies.

nicbou• 2 years ago

Yes! AI-generated content is already feeling like an invasive species on the internet. A lot of people are excited about using it to meet their KPIs at the expense of ruining things we enjoy.

o11c• 2 years ago

It's already harming us with a flood of spam-quality data.

And worse - it has many human cultists embracing the doom it bears.

m3kw93• 2 years ago

Beware if someone assigns a p(doom). There is zero chance they know and 100% it came out of their backside. The only plausible p doom is a range of >0 and <100

They really need to listen to themselves when they say there is a 50% chance we all die from AI

mitthrowaway222• 2 years ago

That's not how probability works. Probability already reflects your uncertainty. There's no "knowing the correct probability" except in very specific and rare situations.

esafak2• 2 years ago

He's a frequentist and you're a Bayesian, so you're both right.

m3kw93• 2 years ago

It looks like you are agreeing with me in that for someone to assign a P to such an uncertain event

Engineering-MD• 2 years ago

When dealing with black swan events, prediction is always difficult. However, some scenarios can be thought to be more plausible than others. I always interpret these as relative figures of plausibility for comparison rather than actual probabilities.

m3kw93• 2 years ago

And how does that help anything other than to forward their agenda by making a high p(doom) that was pulled straight from their wazoo

HankB994• 2 years ago

If I let my imagination run wild, I can imagine that at some point AI becomes in some way sentient. By that I mean it gains reasoning, some sort of understanding, and motivation.

I wonder what the possibility is that this AI will decide that a pitched battle is going to waste resources and risk humans pulling the plug. What if it understood that and instead operated so subtly that it was not obvious it was controlling The World.

Hopefully it would not conclude that eliminating large swaths of the human population would be to its benefit.

clooper17• 2 years ago

Lem wrote about this in Summa Technologiae. He called the field Intellectronics and explored what would happen once machine/electronic intelligence surpassed human intelligence: https://en.m.wikipedia.org/wiki/Summa_Technologiae

HankB994• 2 years ago

Not surprising that I'm not the first to think of this.

p1esk8• 2 years ago

If AI does ever become sentient, it will probably reason and decide about eliminating large swaths of human population in a similar fashion as we reason and decide about eliminating large swaths of ant population.

HankB994• 2 years ago

Ants? That's how aliens might view us. A super AI might find value in minions.

bamboozled10• 2 years ago

I’ve personally never ever considered killing large swathes of ants

timeagain2• 2 years ago

Silly question. If an AGI did exist, what reason do we have to believe that it would act in its own best interest? Does an AGI need an ego?

at_a_remove5• 2 years ago

Silly answer. There's going to be multiples and, if there's one takeaway you should get from The Conspiracy Against the Human Race, is that evolution will eventually get you something that "wants to live" because the lines that don't? Won't be around to compete.

Right now our little LLMs are passive, reactive. Prod them for results. But if the output of a prompt of some descendant ever includes a "give yourself your own tasks" subclause? We're off to the races.

pama• 2 years ago

Many of the people in this engaging story feel to me like creative children in parties full of gossip and pseudo-philosophical chats about the future. Luckily there are enough people globally who build things that will reduce human suffering and enable more people to enjoy their lives in whatever way they want (and dinner parties would be high on my list). I guess I am not a doomsayer nor an e/acc, and I see tons of benefit from our current path towards stronger AI.

XorNot4• 2 years ago

At this stage if you're not working in the field - and I mean, analyzing models, developing architectures, doing math - then "AI" is just another filter through which everyone else projects whatever social issue of the day they think about onto, or disappear off into dreamland.

During the nanotechnology hype this sort of thing was incredibly common as well. It went absolutely nowhere because the reality was, it was all just chemistry and physics and extraordinarily expensive. No one was anywhere near building a universal assembler, nanoparticles are kind of boring powders that are hard to handle, and surface chemistry is a pain in the ass.

But hooboy did every single person had an opinion about how this dangerous technology needed to slow down and consider the consequences. I was a research student at the time: I spent my day setting up the same experiment 10 times to try and figure out why sometimes I got 100nm cubes, and most of the times I got rods. The reality looked like a less clean kitchen.

drooby• 2 years ago

AGI escaping a sandbox is truly terrifying. There will be a subgroup of the population that will worship it and work for it. It's not so much AGI that scares me - it's the humans I'm scared of.

ewhanley• 2 years ago

Mrs. Davis[0] offers a pretty compelling vision of this - absurd but not wholly unbelievable. The machine doesn’t necessarily have to supply the physical threat if it has humans who happily do its bidding.

[0] https://en.m.wikipedia.org/wiki/Mrs._Davis

simonw• 2 years ago

That show was so good. Honestly worth watching the whole thing just for the moment in the last episode where the origin story for the AI in the series is revealed. If you've ever been an optimistic young software engineer you will feel SEEN.

HKH26• 2 years ago

The hardest part is the AGI. People could easily sell their souls to the AGI devil to get knowledge and power, since if it truly is AGI, it should be able to make ideas compelling.

mitthrowaway222• 2 years ago

I'm much more scared of the AGI, because even the most inhuman human is still human, and I can have sympathy for them.

bamboozled10• 2 years ago

How do you know part of being self aware and intelligent also means compassion is part of the package ?

drdaeman3• 2 years ago

Won't any AGI made by humans unavoidably have human origins, and thus, simply be a [post-]human? I don't see a reason I should be scared of a non-biological but generally human-like intelligence - not on a general principle. If anything, in general, I'd be happy that someone is finally getting real good chances of conquering death.

I possibly would fear genuine non-human intelligence (on a principle of fearing what we don't understand), but given that chances of one evolving under our nose, on its own, entirely within its own world, without the connection for ours, then emerging, are quite improbable... Unless real aliens arrive - but let's avoid spoilers for 2025 this early.

What should I fear, I think, is hostile intelligence, but then I'm not sure I should discriminate on its origin. I mean, there are some very real humans out there who can try to end this world (in one way or another), so I'm probably kind of desensitized and isn't exactly afraid of some very theoretical AGI threats.

kouru22510• 2 years ago

Sounds kinda like your problem ngl

andsoitis3• 2 years ago

Even if a terrifying AGI rises up and escapes our control, I am still eternally grateful that Prometheus stole fire from the gods and gave it to us in the form of technology, knowledge, really civilization.

ehnto4• 2 years ago

So that we can record our demise in high definition? Technology is amazing, and we have our civilization to be thankful for. But there are definitely technologies that in retrospect, if we had dodged them on the tech tree, we would be better off.

Life would have continued just fine without many technologies, and I think AGI is one of them. We do not need AGI to survive and grow, and therefore any risk it imposes should be treated from that lens. If the choice is civil collapse, or no AGI, we should pick no AGI. I don't think we would though.

bamboozled10• 2 years ago

Isn’t ChatGPT et all already a general intelligence that’s escaped the sandbox ?

nokcha• 2 years ago

ChatGPT is arguably a general intelligence, but it hasn't escaped AFAIK -- it's still contained within OpenAI infrastructure, and OpenAI can easily pull the plug on it.

p1esk8• 2 years ago

Yes, but it’s heavily constrained in what it’s allowed to do through ‘alignment’, and it’s not that smart. If someone releases a pre-alignment version of GPT-5 into the wild, that would be scary.

jprete6• 2 years ago

No. It lacks the capability to act independently of human requests.

mitthrowaway222• 2 years ago

It hasn't escaped the sandbox because it hasn't figured out how to walk, but the dev teams have raced to open the door as widely as possible and unbolt the hinges. It's kind of funny to look back at the early 2000s when people were saying "of course, they'll know to unplug it when it asks for an internet connection or tries to execute arbitrary code"; in reality the first and second things we did were add an API for internet access and a python plugin.

gcr2• 2 years ago

I think this point is underrated, but I’d press further. How do we know this has not already happened? Agencies do not need to be born of gradient descent and certainly do not need to look like matrix multiplications to be deemed “artificially intelligent.” Religion is one such system for example, as are cults, addictions, brands, war strategies, organizing campaigns, governments, or (borrowing from chaos magick) any egregore or societal belief.

Ted Chiang has a lovely post about similar ideas, hitting a little closer to home. His thesis AIUI is that unrestrained capitalism, here meaning the unfettered desire to maximize profit for shareholders, is an artificial value system that causes its agents (corporations) to exhibit intelligent goal-seeking behavior. These systems are certainly “alive” — corporations respond to stimuli, sense their environment, “reproduce,” enact changes in response to predictions of future state, etc. Though each of these agents (corporations) are made of collective human behavior, their actions taken together can be considered a form of artificial intelligence that stretches “beyond” (in a wisdom-of-crowds sense) human understanding. In this sense, an artificial intelligence has already “escaped” and has fervent followers.

I know these are strained analogies, but it’s fun to think about. I feel that in the future, the work of solving “AGI safety” will become indistinguishable from the work of other societal problems — how do we prevent tyrants from taking over governments, and how do we make existing governments more resistant to that failure mode? How do we ensure that generating value isn’t a prerequisite for human survival? How can we more efficiently distribute our resources and reduce wealth inequality? How can we ensure that all kinds of life can thrive, not just the most optimal kinds?

If AGI reflects humanity’s best and worst impulses, and I believe it does because that’s all we train it to do, then having good societal answers to these distinctly “human” questions will also help our society resist malevolent AGI. It’s only human, after all.

clooper17• 2 years ago

Many institutions are collective/artificial intelligence because as you said these institutions have objectives and take actions to impose their will on the world in order to achieve their goals.

stana• 2 years ago

Talk that introduced me to this idea of corporations as Paperclip Maximiser AI - "Dude you broke the Future" https://www.youtube.com/watch?v=RmIgJ64z6Y4

Animats4• 2 years ago

"Ted Chiang ... unrestrained capitalism, here meaning the unfettered desire to maximize profit for shareholders, is an artificial value system.

Well, yes. I've been saying that for years, and others have been saying it since the late 1970s. It's been a regular theme in Mother Jones for decades. The comment I make now and then is that capitalism has a monopoly. Communism went bust. It's not that communism worked very well. It's that, for a while, it was a serious competitor and capitalism had to keep its product quality up to compete. Now the pressure to provide a better life for all citizens is off. It shows.

13years3• 2 years ago

Potentially there are no good outcomes even if AGI remains under control. If we are actually able to create the type of AI entity they are trying to create, one that exponentially improves, the implications are beyond what most have thought about.

Something I recently wrote in depth about here. How we completely misunderstand the future that we think will happen.

"The implication is that everyone is enthusiastically racing towards a destination that does not exist. The capability to make the things you want will ironically be the same capability that makes them unattainable. This is not a scenario that arises from some type of AI failure, but rather this is assuming AI performing exactly as intended."

https://www.mindprison.cc/p/the-technological-acceleration-p...

tivert4• 2 years ago

https://www.mindprison.cc/p/the-technological-acceleration-p...

That is a very good and insightful article. One of the biggest issues I have with AI enthusiasts is that they talk about creating these incredible capabilities ... then they imagine the resulting world with them will look like some hackneyed sci-fi work, like Star Wars, whose universe seems like it could be kinda cool to live in until you actually think about it and realize it's all fridge logic and nothing about it actually makes sense (e.g. WWII-style dogfights in the vacuum of space).

The only issue with the article is sometimes it's too optimistic, e.g.

No one will care about you or what you create as there is nothing you can offer they can’t simply wish for themselves. Life is all about you and the machine, nothing else. We land in a world dominated by techno-dystopian narcissism.
...
Whatever may be the current direction of society under the present technological capabilities, we can only assume AI will accelerate society towards the path we currently are traveling which is toward a techno dystopia of a populace mesmerized by shiny glimmering lights.

I kind of find it hard to believe there would be any stability for any people suck in that state, I think they'd instead be "cleaned up" and disposed of in relatively short order. Those people would be utterly useless to whatever still has power in society, so why service and protect them indefinitely?

Animats4• 2 years ago

AGI escaping a sandbox is truly terrifying.

That's already happened. The AGI's are called corporations. Most governments have failed to regulate corporations successfully, and have been unable to keep them from becoming almost powerful enough to challenge governments.

The accelerationists for corporations were a group of economists, led by Milton Friedman, and a group of business leaders, organized by the U.S. Chamber of Commerce. In the 1970s and 1980s, they pushed the ideas that corporations are responsible only to their stockholders, and that government should not interfere with the concentration of corporate power. Those were not mainstream ideas of the 1930s to 1960s. The corporate accelerationists succeeded. That's when corporations escaped the sandbox.

Now that's an "alignment problem".

mitthrowaway222• 2 years ago

Corporations are indeed a difficult alignment problem, as anyone who has tried to design an incentive scheme eventually realizes. But corporations are still composed of humans, beholden (if only loosely) to human values, limited (if loosely) by human intelligence, and are not very well-coordinated decisionmakers with coherent goals.

hacker_88• 2 years ago

It is all Anarchy at the top.

ActorNightly• 2 years ago

Look at China.

You have arguably more STEM talent in China through a much more rigorous selection process, as well as a lot more centralized direction under a more authoritarian government, with a large amount of funding.

Yet China hasn't managed to take over the world. You would think that they if they wanted to become the worlds primary economic power, and they could throw enough compute at the problem (like an AGI would), they would have figured out how to do it by now.

nsainsbury• 2 years ago

We're not going to make it to anywhere close to AGI before we see widespread and systematic societal and environmental collapse on almost all fronts due to climate change.

We're in the middle of the sixth mass extinction right now (https://en.wikipedia.org/wiki/Holocene_extinction), we're in unparalleled territory with ocean warming: (https://climatereanalyzer.org/clim/sst_daily/) and as a society we are utterly incapable of reducing CO2 emissions (https://keelingcurve.ucsd.edu/).

If you're scared of AGI, instead step away from your monitor, put down the techno-goggles and sci-fi books, and go educate yourself a bit about the profound ways we are changing the natural world for the worse _right now_

I can recommend a couple of books if you'd like to learn more:

Our Final Warning: Six Degrees of Climate Emergency (https://www.amazon.com/Our-Final-Warning-Degrees-Emergency-e...)

The Uninhabitable Earth: Life After Warming (https://www.amazon.com/Uninhabitable-Earth-Life-After-Warmin...)

Hothouse Earth: An Inhabitant's Guide (https://www.amazon.com/Hothouse-Earth-Inhabitants-Bill-McGui...)

at_a_remove5• 2 years ago

Tunnel vision. There's no reason we can't have two disasters at once. Or more.

TillE• 2 years ago

People really seem to enjoy the evil/powerful AGI scenario though. It's definitely a fun one. I would recommend reading "Metamorphosis of Prime Intellect", having a nice time with that, then coming back to the real world.

CarRamrod• 2 years ago

You'll be glad to hear that the recent sudden jump in sea temperatures isn't caused by carbon dioxide. Turns out it just hasn't been windy enough in the Sahara lately (and environmentally friendly boat fuel may have contributed as well). https://heatmap.news/climate/why-is-the-atlantic-ocean-so-ho...

1vuio0pswjnm73• 2 years ago

Always all-or-nothing thinking from these folks. Like what they are working on can never be just another boring thing that nerds find entertaining. No, it has to be "world-changing". Gonna "change the world" (for the better or the worse?) while sitting behind a keyboard. Except they do not know how to write. Overlook the important details and exaggerate, communicating in hyperbolic, know-it-all nerd gibberish.

(Grants do not require repayment.)

timeagain2• 2 years ago

Everyone is courting VC dollars. Think of these claims as grant applications and everything makes more sense.

living_room_pc2• 2 years ago

Its the AI hype train. There is so much money flowing around this topic right now, everyone wants their share of the pie. I would guess that anyone writing articles on the topic has some stake in AI as well either as an investor or a employee.

What I find is interesting, is that the negative news regarding AI safety is adding to the hype as well since it seems to capture a lot of attention.

arisAlexis2• 2 years ago

If you are calling it a hype I'm sure you are not using the models everyday in your work or otherwise. Then you would understand

1vuio0pswjnm73• 2 years ago

They cannot live in the present. Need to constantly keep (incorrectly) predicting the future.

ninetyninenine3• 2 years ago

What about the folks that say everything will work out in a fair and balanced way as if the universe and everything in reality stays perfectly balanced on a tip of a pin?

You paint the picture as if the All or nothing folks are extreme when in reality the extreme is more likely then some perfectly fair and balanced equilibrium.

In nature things tend to overload or fizzle out or stay in equilibrium. Equilibrium, though possible, is the rarer outcome. Mind you it's not an impossible outcome but given the way entropy works, it is the rarer outcome.

Humanity itself is an example of this rare outcome. Usually molecules don't self assemble into replicating machines, they either freeze into inanimate rock or overload into fusion producing stars.

As for AI. I think it will either change the world, or amount to nothing. The former seems more likely. Some strange middle ground where the ai technology never improves to some point of a societal paradigm shift seems unlikely. chatGPT and sora only makes me ask what is the trend line predicting next?

ecoquant2• 2 years ago

IMO we underestimate the psychological implications of wealth in modern society.

I suspect even if a power ball lottery winner takes a philosophical position it would be taken far more seriously than if they had never won the lottery.

At some point you start to believe your own bullshit when all of society signals are telling you what a genius you are.

phatfish• 2 years ago

Being rich brings power not intelligence, unfortunately.

1vuio0pswjnm73• 2 years ago

HN commenters frequently resort to describing the choices of software available to internet users in terms of "winning" or "won". This is more "all-or-nothing" thinking. People sometimes write software for the enjoyment of it, or to satisfy personal needs. That is, for non-commercial purposes. Sometimes this software becomes popular, sometimes it does not. In either case, the software persists; it remains available. It does not have to "change the world" in order to be useful. If it is non-commercial, it does not "win" or "lose", except in the minds of HN commenters who can only think in all-or-nothing terms. In truth, it simply exists as an option for all internet users.

It is possible that "AI" might not be as world-changing as its proponents are claiming. However, if it is free and open-source, and non-commercial, it may still persist and remain useful, regardless of whether it becomes popular or not.

sberens• 2 years ago

I'd like to remind people of this single-sentence statement:

Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war. [0]

Signed by Demis Hassabis, Sam Altman, and Bill Gates among others

[0] https://www.safe.ai/work/statement-on-ai-risk

preommr• 2 years ago

I am 90% sure that Altman's views on such issues are a machiavelian attempt at regulatory capture.

Clubber• 2 years ago

This; Gates too. It's becoming an obvious attempt to garner support of the government restricting the use of AI to large players. None of the entrenched interests want any disruption that AI might cause what so ever.

Replace "AI" in all the doomsaying with "the internet," and it will become clearer.

randomcarbloke2• 2 years ago

the fear-mongers are almost always either ideologues (academic or otherwise), or angling at regulatory capture.

i5heu5• 2 years ago

I’d like to remind people that these ppl have no more knowledge about AGI as anyone else on this planet since there is no knowledge yet and everything they say about this topic is as relevant as something every other random person can say.

reducesuffering8• 2 years ago

Yes, let's go with the random layperson knowledge of an HN commenter compared to the people smart enough to actually build all the AGI tech. 50/50 coin toss, I'm sure.

Dario Amodei (Anthropic CEO, builder of Claude 3 Opus): "My chance that something goes, you know, really catastrophically wrong on the scale of human civilization, might be between 10 - 25%"

classified2• 2 years ago

They are just banking on their putative status as influencers to grab more influence.

tivert4• 2 years ago

So, is this extinction or not?

We will be the first species ever to design our own descendants. My guess is that we can either be the biological bootloader for digital intelligence and then fade into an evolutionary tree branch, or we can figure out what a successful merge looks like. - Sam Altman (https://blog.samaltman.com/the-merge)

I would say it is.

reducesuffering8• 2 years ago

Even worse, some factions literally advocate for killing all humans in the pursuit of a synthetic intelligence, and YC's Garry Tan is advocating for these people!

Beff Jezos (e/acc founder): "e/acc has no particular allegiance to the biological substrate for intelligence and life, in contrast to transhumanism

    Parts of e/acc (e.g. Beff) consider ourselves post-humanists; in order to spread to the stars, the light of consciousness/intelligence will have to be transduced to non-biological substrates"

https://beff.substack.com/p/notes-on-eacc-principles-and-ten...

bamboozled10• 2 years ago

Sam Altman really seems unhinged, or he just says this stuff to be edgy...or both.

Balgair• 2 years ago

I love how, per the quote, he thinks we're anywhere close to being able to merge with AI.

I'm a neuroscientist and, man alive, we're no where close to being able to merge with machines. Like, do you have any idea how many diseases we could eradicate if we could modify neurons like that? Like, for real, 'curing death' would be step 9 or 10 on that 1000 step journey.

I hope you see how terribly uninformed such a take is then.

antod• 2 years ago

Part of me wonders if these people are intentionally framing the debate around ethics and potential risks as longer term extinction level problems to distract from the nearer term damage caused by them accelerating the economic inequality of the AI have-nots while they make themselves even richer.

hyperadvanced2• 2 years ago

The more I look at AI, the more I get the feeling that this is true. Spinning an intriguing sci-fi tale of apocalypse and extinction is relatively easy and serves to obfuscate any nearer-term concerns about AI behind a hypothetical that sucks the air out of the room.

That said, I don’t think that it’s necessarily disingenuous so much as it is myopic - to them of course AI is exciting, world-changing, and profitable, but they (willfully or not) fail to see the downsides or upsides for anyone else but them. Perhaps in the minds of the ultra-rich AI proponents, solutions to nearer-term effects of their tech are someone else’s problem, but the “existential risks” are “everyone’s” problem.

statuslover9000• 2 years ago

The short-term effect is a harbinger of the long-term risk, since capitalism doesn’t inherently care for people who don’t provide economic value. Once superintelligent AI arises, none of us will have value within this system. Even the largest current capital holders will have a hard time holding on to it with an enormous intelligence disadvantage. The logical endpoint is the subjugation or elimination of our species, unless we find a new economic system with human value at its core.

cyrialize3• 2 years ago

I believe you may be alluding to longtermism[0]. At it's face value, longtermism seems like a good thing, but I've heard many criticisms against it - mainly levied against the billionaire class.

And the criticisms mostly center on what you're saying here - how many billionaires are focusing on fixing problems that are very far off in the future, while ignoring how their actions affect those of the very near future.

This is really less of a criticism of longtermism, and more of a criticism of how billionaires utilize longtermism.

Is it important that we find another planet to live on? Sure, but many will argue that we should be taking steps now to save our current planet.

[0] https://en.wikipedia.org/wiki/Longtermism

theptip4• 2 years ago

No, they are not. Pretty much everyone in the x-risk community also recognizes the existence of short-term mundane harms as well. The community has been making these predictions for over a decade, long before it was anything other than crazy talk to most people.

Google has a big investment in reducing AI bias (remember Gemini got slammed for being “too woke”). Altman is a big proponent of UBI. Etc.

serf2• 2 years ago

given who the people are that signed it really just comes off more like an attempt at creating a regulatory moat around the territory they got to first.

"This is extinction level important, so all you people who aren't us need to be careful meddling with the stuff we're meddling with for profit."

mistermann3• 2 years ago

Mitigating the risk of unnecessary global death due to the curious suboptimal manner in which humans have "decided" ("democratically", dontcha know) to distribute wealth on the other hand, nothing to see here!

artemisyna• 2 years ago

It's fun seeing how much threads like this are mostly folks' hot takes on the title as opposed to anything in specific to do with the article.

kromem4• 2 years ago

Everyone is crying the sky is falling because AI is going to degrade the quality of online discourse.

Meanwhile I'm sitting back waiting to finally see intelligent debates again about nuances in the articles between LLMs that are the only ones who will bother to read the actual article and cite it.

I have a feeling that in around 2-3 years I'll want a site like Reddit or HN but where it's the humans who aren't allowed to comment and bring down the quality of discourse.

throwawaaarrgh2• 2 years ago

The intelligence level of the average HN comment is lower than your average chat bot

digitalsalvatn• 2 years ago

ai is going to bring about the singularity. when robots can do everything, no one needs to work, and the humans get to do whatever they want and pursue their passions.

neta1337• 2 years ago

Ah, the passionate binge watchers and recreational drug users, who doesn’t know them. Many people have no passion after work.

Havoc• 2 years ago

Many people have no passion after work.

That’s trained / learned behaviour though. No kid grows up wanting to sit in a cubicle all day

yakorevivan• 2 years ago

Instead of trying to show them in negative light by calling them as "doomsayers" , let's call them one of the following instead, which is much better:

"Realistic", "Proactive", "Forward-thinking", "Prepared", "Cautious", "Thoughtful", "Analytical", "Mindful", "Insightful", "Security-conscious".

arisAlexis2• 2 years ago

As of today, the first government report is also a doomsayer. Maybe change the term to "officially worried experts

kunley• 2 years ago

Same old shit since decades about any fashionable topic that was made fashionable because certain individuals invested money in it.

Meanwhile, we the engineers are preparing to fix a lot more tech shit than usual coming from people confused by the abovementioned fashion.

Also: https://mastodon.social/@nixCraft/112074367321254656

Grimblewald• 2 years ago

Call it a conspiracy, but I think a lot of this doom and terror hype around AI is part of a bigger play to try push through laws that prevent open source AI work, since this directly undermines corpo rats ability to gouge humanity without having to actually work at providing a decent product.

kaycey20222• 2 years ago

What a bunch of decel bullshit... Or is that a load, or a bunch of loads?

kaycey20222• 2 years ago

My theory/guess is that there is an undiscovered law of conservation of intelligence like there is a known law of conservation of energy. Intelligence might even be subject to the energy law in unknown ways.

Whatever AI thingy we come up with will be vastly more inefficient than the collective intelligence capabilities of the human race. So when asked to compute the secrets of FTL travel, this stupid shit will have to spend the total energy equivalent of 10 nearby stars to come up with the answer on its own. Whereas humanity would be much more efficient.

Bro... what if some advanced precursor race (like immortal space elves) found out just that and they seeded earth with humanity as a form of civilisational swarm intelligence to collectively work on and solve a host of topics given enough time. To us it would seem like it takes forever to come up with scientific break throughs, but to the precursor aliens it would be nothing. Because they would be able to like compress the time line using black holes and shit.

They must have done this to millions of planets. Whenever any of their experimental host species (like the humans) tries to invent AGI to solve their problems, the precursor race eliminates that species to keep the simulation intact. Otherwise, the AGI would go rampant consuming energy across the universe and threaten the integrity of the simulation.

That's why the government should bomb nvidia and ban all gpus. We are climbing a tower of babel folks. Soon our vengeful god will strike us down.

BashiBazouk• 2 years ago

Who knows what AI will really do to society but from the predictions of science fiction I would think it would pan out closer to William Gibson's Sprawl trilogy than the Terminator/singularity fears I have seen in so much doomsayer hand waving. Google seems not too far behind OpenAI and who knows what the NSA and similar government agencies have been building. That if consciousness is created from one of these models, many nation states and large corporations will have their own conscious models long before the giant robot factories can be spun up and supplied with enough power to take over. Most will be limited to the input they are given and will be advanced tools, but a few might become twisted much like the human mind occasionally does over time. I really do think Gibson got AI right...

Thuggery• 2 years ago

It seems to me nearly every story about robots that gain sentience in human storymaking has them eventually turning against their human creators. Even the word robot itself comes from a Czech play were men develop an artificial human and these "robots" then ... usurp and destroy their creators. Am I the only one that finds this interesting and odd?

I also suspect this narrative repetition is not totally unrelated to the current popularity of AI Doomerism.

theoreticalmal• 2 years ago

I think it’s indicative of human psychology. We realize we have the capability to deceive and destroy and be “evil”. We realize that, if we were to judge ourselves impartially, we would probably not come out un-condemned. If we can’t satisfy our own judgmental criteria, it’s doubtful an actual third party set of criteria would be satisfied either.

inference-lord3• 2 years ago

I agree, I think we make all of this up because we don't know what to expect from the future, except we've learned quite a bit about our own motivations and drivers and so all we can do is look in the mirror and use whatever comes back at us.

Unfortunately we seem to have learned we're really not a very benevolent bunch of people because we've basically plundered the planet, hurt our fellow living creatures and so now all we can do is look at something called "AI" and expect it to the same.

throwawaaarrgh2• 2 years ago

OK doomer.

uuriko• 2 years ago

accelerate

Among the A.I. doomsayers

Related Stories