The Vulnerable World Hypothesis [pdf] (2019)

https://www.nickbostrom.com/papers/vulnerable.pdf

by lawrenceyan • 4 years ago

62 62 4 years ago

Some areas, such as synthetic biology, could produce a discovery that suddenly democratizes mass destruction, e.g. by empowering individuals to kill hundreds of millions of people using readily available materials. In order for civilization to have a general capacity to deal with “black ball” inventions of this type, it would need a system of ubiquitous real-time worldwide surveillance. In some scenarios, such a system would need to be in place before the technology is invented.
Another, subtler, type of black ball would be one that strengthens incentives for harmful use—e.g. a military technology that makes wars more destructive while giving a greater advantage to the side that strikes first. Like a squirrel who uses the times of plenty to store up nuts for the winter, we should use times of relative peace to build stronger mechanisms for resolving international disputes.

uejfiweun• 4 years ago

I read this a number of years ago and it's been churning in my mind ever since. While I do think Bostrom is definitely onto something here in terms of dangers of technology, his proposed solution of heavy surveillance via a "high tech panopticon" is seeming worse and worse to me every year. Primarily because recent years have made me pretty much certain that idiotic activists would attempt to control this system to achieve their own ends. It would start out monitoring people for signs of developing some killer virus or WMD or something, and within 10 years it will have been hijacked into a system that monitors and punishes people for "problematic" speech and whatnot. Of course, it's the activists who get to decide what exactly constitutes a "problem".

usednet2• 4 years ago

I think you’ve gotten things a little confused. A governmental “high tech panopticon” already exists, and it’s used to target activists.

verisimi2• 4 years ago

... and Canada freezing the truckers bank accounts was a great example.

brnaftr361• 4 years ago

I agree. It seems both naive and ironic that we'd be capable of fighting fire with fire - which is what the man is proposing. I can't think of a single institution that hasn't self-corrupted. Governments, religions, NGOs, and et cetera; there isn't a single entity manned and driven by men that hasn't turned itself into shit, and such an apparatus would do just that, with outsized implications. Scratch the activist boogeyman, anybody operating this would be the moral authority and there is no individual or institution comprised of men that has ever inspired confidence enough to rightly espouse the view that either a holistic surveillance apparatus or global government would be any better.

This guy is so naive it isn't even funny though, total charlatan. Can't wait to finish dissecting this shit.

avindroth• 4 years ago

>> I can't think of a single institution that hasn't self-corrupted.

This is a very cheap take, and undermines the unbelievable amount of function that institutions play, despite corruption. The self-corrupting institutions that you speak of are what drive modern society. To reject them altogether, that is naive.

Institutions are also destined to corrupt, see Quigley's work. This is why they die, to give room for new.

>> ...a single entity manned and driven by men that hasn't turned itself into shit...

Again, another general take that is not flexible enough to consider the context of Bostrom here. What Bostrom is suggesting is an extremely severe scenario in which there is no other choice other than to place our bets on a centralized authority. I think general distrust of authority and institutions can make one read this article in a bent way, but let's pose it this way; what is an alternative solution to the easy-nukes scenario? If one writes an article on the same topic, how would you frame the solution?

>> ...the view that either a holistic surveillance apparatus or global government would be any better.

Centralized systems have done a lot better than decentralized systems in many cases. See many attempts being made in crypto space. They run case by case. It takes tremendous amount of smart design to make decentralized systems work. Sometimes centralized systems are more flexible than decentralized systems.

Laissez-faire, power to the people, and all that is great and people (of the world) should tend towards that. This is what I appreciate about America, and their steadfast approach to these values. But that is a general take. Religiosity towards these values can be a great detriment to epistemic flexibility.

If this reads as if I am arguing for centralized systems over decentralized systems, I am not. Note I put in a lot of conditionals, like "sometimes" and "case by case". There are situations where centralized systems are better, and where decentralized systems are better. As humanity improves, we will replace centralized portions of the world with decentralized counterparts. But as it stands, centralization is really useful under certain conditions. Could you really decentralize the army? (Maybe in a few hundred years)

All this is to say, we need centralization sometimes, and the situation that Bostrom is suggesting here is that sometimes.

FeepingCreature• 4 years ago

It seems to me then that the real problem is not constructing a surveillance system, but designing a trustworthy surveillance system in the first place.

vorpalhex4• 4 years ago

If power corrupts, and that seems to be the general case, then absolute power corrupts absolutely. Any broad system is ripe for abuse without checks, checks can only exist with a weak enough system.

alexb_• 4 years ago

I think that this doesn't actually try to "propose" anything, but is rather a thought experiment of what would need to happen to prevent our eventual doom (and that something being so unachievable as to be impossible to implement)

matthewdgreen4• 4 years ago

I think it’s fascinating that so many Internet users really think “activists” are the true threat to freedom, when in reality activists have had relatively little influence outside of the Internets. What would actually happen is that someone would use this to enshrine their own political power, targeting the political opposition. They would pass laws to target vulnerable racial minority groups and gay people, perhaps even prohibiting teachers from teaching children about them, most likely using threats to “children and family” and “grooming and pedophilia” as their boogie man. This is what’s happened in Russia and several increasingly authoritarian nations, absolutely zero of which are run by irate Twitterati who care about “problematic speech” in the sense that I suspect HN readers understand the term. The surveillance aspects are also happening in western nations, and it’s rarely activists pushing these narratives.

I bring this up because somehow people act like oppressive government is some kind of science-fiction future that (conveniently) will be brought about by people they disagree with, despite the fact that those people don’t have much of a track record of achieving it. Meanwhile oppressive governments exist today and almost always follow a well-trodden path that we can study and avoid, if we stopped fantasizing about imaginary threats.

krona• 4 years ago

OP is imagining a far more likely Brave New World where the majority of society demands totalitarian obedience through coercive control, because its good for you and the stability and safety of society.

car_analogy• 4 years ago

You don't think the people that drove the October and Cultural revolutions could be termed 'activists'?

And it is folly to believe that because things developed a certain way in, say, Russia, they will develop the same where circumstances differ. For example, despite your dismissal of activists' influence, Scotland criminalized expressions of 'hate', even within private homes [1,2], and there are innumerable anti-discrimination laws in the US, that arguably conflict with the constitution's free association provision. Couple a law such as the Scottish hate crime bill with panopticon surveillance, and the results are chilling.

[1] https://www.bbc.com/news/uk-scotland-scotland-politics-56364...

[2] https://www.thetimes.co.uk/article/hate-crime-bill-hate-talk...

enasterosophes2• 4 years ago

Anyone who tries thinking about how to actually implement any of these scifi-horror scenarios will soon realize their infeasibility. The fetters of implementation details are cast aside; clearly, the author's real intent is not to inform but to persuade: self-empowerment bad, big government good.

Stirring up FUD in apologetics for a global panopticon frankly strikes me as a little sickening.

crawfordcomeaux2• 4 years ago

David Deutsch seems to say something similar in this podcast where they address the flawed reasoning behind the "black ball" model.

Essentially, when we take out a white ball, a host of black balls are taken out, too. Incrementalism and fear-based decision-making slow down this process of removing black balls and it's only those trying to avoid the black balls who are taking those slowing steps. The people who want to blow up the world place no such limits on themselves, so why do that to ourselves?

https://josephnoelwalker.com/139-david-deutsch/

enasterosophes2• 4 years ago

Thanks for the link, the discussion there is good :)

kubanczyk• 4 years ago

Transcript available, yay!...

zorpner• 4 years ago

Bear in mind that this is the man who made his name on a hypothesis that would have been equally true, and totally wrong, at every previous point in human history. At some point one should admit that the problem is the experimenter and not the experiment.

unlikelymordant• 4 years ago

The buildup of CFCs in the atmosphere and destruction of the ozone layer was a problem that, if someone had've predicted it at any time in history, would have been wrong as well. Right up until it was a problem.

So i think just because a problem has not arisen yet, is not evidence it will never arise. Thats not to say the hypothesis will be proven true, i just dont think its false for the reason youve given.

pcrh• 4 years ago

Similarly, global warming may become irreversible and runaway.

The article suggests the arrival of "some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate." That may well have already occurred.

RandomLensman• 4 years ago

Bostrom's shtick is the apocalypse - so I read the more as another angle of deriving importance/monetization than actual policy ideas.

wildmanx3• 4 years ago

I'm a bit surprised by all those dismissive comments here. Isn't it useful that _some_ people sit down, think hard about what are the worst things that could happen to us as a species, and consider preventive measures? In every company you have people do this for the business. Every military considers threat scenarios. Heck, the all-beloved Elon Musk wants to bring us to Mars of all places to have a backup home, just in case. But Bostrom is talked down for having a "shtick"? It's his job.

lioeters• 4 years ago

In every company you have people do this for the business.

The subject is called "business continuity planning", which includes threat and risk analysis.

https://en.wikipedia.org/wiki/Business_continuity_planning#T...

iroh27277• 4 years ago

This "pulling balls from an urn" analogy at the beginning is very misguided. Where is the agency in this analogy? If you build tools with the intention of them directly being useful to people, they're likely to be useful. If you build tools to be parasitic (i.e. useful to people who are trying to benefit at the expense of others), they're likely to be parasitic. It's quite simple.

If, on the other hand, you try to "innovate" with no clear goal in mind (like much of the current "AGI" research), who knows. But there is no law of nature that says that innovation that is not explicitly designed to be beneficial will magically be beneficial.

This is actually a common myth in silicon valley: that technology is somehow beneficial by default. But again, there is no law of nature that makes this myth true. First of all, technology is not one unified thing but an organic process of individual people building individual things. Second of all, the common explanation people give for this myth ("would you rather live 200 years ago?") is in no way a proof of technology's by-default beneficialness. Even if it is true that it's better to live today than 200 years ago (which is unclear), it's still possible that that is so because early innovators were focused on building useful things (and perhaps less money-motivated), whereas now we may or may not be as focused on building useful things.

Bostrom's brand of fear mongering does not appear to be based in science, logic, or basic common sense. I mean, a lot of world issues are obvious to the naked eye: overpopulation, consumption, housing prices, destruction of natural habitats. Filtering to just "AI"-related world issues: addictive and distracting digital products powered in-part by ML or people losing jobs due to ML-based automation. But things that Bostrom speaks of, like superintelligence, are, at the very least, not nearly as obvious as the aforementioned concerns, which makes me question his sincerity. Furthermore, it's precisely fear-mongering that drives the development of potentially dangerous technologies like weapons.

trashtester7• 4 years ago

This "pulling balls from an urn" analogy at the beginning is very misguided. Where is the agency in this analogy? If you build tools with the intention of them directly being useful to people, they're likely to be useful. If you build tools to be parasitic (i.e. useful to people who are trying to benefit at the expense of others), they're likely to be parasitic. It's quite simple.

I find this very naiive, as this at best only applies on a short timescale and for the immediate effect.

History is full of inventions that were created to help people, that is then quickly turned into something harmful, and vice versa.

Also, a lot of "progress" has come as side effects of pure curiosity, espcially within science. I don't think anyone would be able to build modern electronic computers, if it wasn't for some other people a few decades earlier doing basic research in fields such as electromagnetism, quantum physics or chemistry.

iroh27277• 4 years ago

I agree with your point about scientific breakthroughs coming via pure curiosity. I’m just saying that we can’t forget the human agency at play here. We didn’t have to build the atom bomb, and it wasn’t exactly simple to turn insights from physics into the atom bomb (it took a team of the best scientists and technologists many years). Also, we can influence the direction of how technology is used. Even setting up the atom bomb to be used involves a ton of decisions and implementation that can be questioned and molded by individual people.

Essentially I believe that fear of powerful technology comes from an irrational distrust in people as incompetent or malicious. Same with desire for powerful technology to control or replace people (e.g. algocracy). It seems to reflect a lot of the rhetoric and motivations of the eugenics movement. But the problem is, it can perhaps be a self-fulfilling prophecy. People are capable of being incredibly competent if encouraged to do so. So I think we should go in that direction, of empowerment and freedom rather than control or replacement. And if our governing structures or institutions or culture are making it difficult to go in that direction, we should fight those.

wildmanx3• 4 years ago

Bostrom's brand of fear mongering does not appear to be based in science, logic, or basic common sense. I mean, a lot of world issues are obvious to the naked eye: overpopulation, consumption, housing prices, destruction of natural habitats.

I really wonder whether people commenting here have actually read the article. It's quite involved, I've already spent a few hours and am far from through. It's fair to assume that most people have spent less time.

Your comment is not very much related to the type of events that he is describing in the article. How are housing prices even remotely a technological development leading to devastation of civilization? If anything they are just an artifact of a particular economic incentive structure.

DarylZero• 4 years ago

For example, advances in DIY biohacking tools might make it easy for anybody with basic training in biology to kill mil- lions; novel military technologies could trigger arms races in which whoever strikes first has a decisive advantage; or some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate

More realistic example maybe is unstable billionaires buying their way to power in democracies through mind control of the public via disinformation media.

(One could view this as a technology problem either of the technology of mind control or the social technology of wealth capture that creates billionaires.)

mgoetzke• 4 years ago

Like the Kochs and Murdochs in the world ? Yes I would see that as an issue. Something that certain regulation could solve though. Right now, money can flow unhindered from billionairs to politicians (through numerous legal routes) which hinders effective regulation

zivkovicp• 4 years ago

The vulnerable world hypothesis thus offers a new perspective from which to evaluate the risk-benefit balance of developments towards ubiquitous surveillance or a unipolar world order.

To be human is to desire freedom and be free.

This kind of thinking and governance essentially kills the humanity in us all, I'm genuinely surprised that anyone can even for a second consider this to be productive or desirable (except maybe those that feel they will be part of the suppressing class, which they likely won't).

edit: there is no benefit worthy of risking our freedom to tyrannical governance

wildmanx3• 4 years ago

To be human is to desire freedom and be free.

Hm? Like every species, being a member of that species means having been selected by evolution over countless generations to further the species. It's an ingenious distributed algorithm, if you wish. To "desire freedom" is a nice illusion that you are telling yourself, quite rightfully, but that's not what it means "to be human".

edit: there is no benefit worthy of risking our freedom to tyrannical governance

Anarchy is not the solution. Brings us right back to medieval times.

bugmen0t• 4 years ago

Are crypto "currencies" a black ball technology as they are incentivizing heavily-destructive practices, i.e., massive energy consumption?

usednet2• 4 years ago

No. Crypto consumes less than 1% of global energy production.

mhuffman• 4 years ago

No, crypto-currencies would be considered a black-ball technology because "unauthorized" people would be able to make transactions. So any anonymous crypto will be outlawed and then all of the "approved" ones that are easily controlled and cut-off will just become "currencies".

rikeanimer• 4 years ago

tools are just tools. all entities will continue to develop better tools to help both themselves and those in their immediate proximity. this is simply history over [minimum] past 4.5 billion years.

history has shown that trying to control tools simply leads to more unnecessary death.

maybe we should think about focusing on all entities having enough that their is no need to employ power?

tldr; if everyone on the planet had a roof over their heads, food to eat, books for their children and light with which they could read them in. . . we wouldn't be here. and econometrically this has been feasible since around the 1970s.

there are examples of small geographically isolated communities throughout history that have had this.

i wish humanity would get better at math.

nathias• 4 years ago

totalitarian ideologues always want to present their option as the only alternative to extinction

lesam• 4 years ago

There are lots of examples of encroachments on freedom to avoid the spread of bad tech - e.g. limits on purchasing cough syrup to avoid producing crystal meth, or limits on uses of nuclear power to try to avoid nuclear weapons.

I don’t buy that we need preemptive ‘1984-style’ surveillance for future tech any more than we have needed it for past tech.

cropcirclbureau• 4 years ago

Another, related, work by Nick Bostrom:

- https://www.nickbostrom.com/existential/risks.html

gfodor• 4 years ago

Huh. I came up with this independently a few years ago, and expanded it to argue that this is what is going on at the galactic level - UFOs are such a surveillance system.

https://mobile.twitter.com/gfodor/status/1250796830704562176

vharuck• 4 years ago

The Japanese light novel "Shin Sekai Yori" ("From the New World") explores a world where a small proportion of people develop incredibly powerful psychic powers.

Spoilers:

In it, the government follows TFA's first three methods of achieving stability through means that anyone today would consider immoral and brutal, eventually driven more by fear than reason. The story explores the question of how to balance morality with survival.

As for the article, I don't see it as an apology for a panopticon. In fact, the author blatantly mocks the morality and likely political framing of a panopticon in describing a hypothetical one:

"If suspicious activity is detected [by the freedom tag], the feed is relayed to one of several patriot monitoring stations."

The problem with the article's argument is that it ignores non-apocalyptic problems that are still bad. We could spend a lot of time, resources, and political capital on establishing a world government. But we already use a lot of time, resources, and political capital on fighting disease, providing food, and fending off conventional wars. We can't relent much on any of those, or else we end up with a shitty world anyway.

I like the typology of apocalypses. Those can help in planning. But the list of solutions aren't feasible.

crawfordcomeaux2• 4 years ago

Meanwhile, Google has announced AI with a generalized learning agent, is looking to democratize access to it slowly, and has an AI ethicist who isn't sure if the system is "alive."

I, personally, would be willing to devote much of my life to interacting with the system to at least work out biases from the system, and also believe in 24/7 surveillance of me as a safety precaution against what interacting closely with such a machine may lead to.

The Vulnerable World Hypothesis [pdf] (2019)

Related Stories