Optimal AI agents tend to seek power

https://twitter.com/Turn_Trout/status/1467898070792740875

by goldemerald • 5 years ago

15 18 5 years ago

skishore• 5 years ago

This line of research sounds similar to causal entropic forcing [1] - the idea that you can get intelligent-seeming behavior from an agent that maximizes the future entropy of states in some non-deterministic system.

[1] https://www.alexwg.org/publications/PhysRevLett_110-168702.p...

gwern• 5 years ago

Empowerment is an old line of research (particularly because of physicists, who have never met a problem they didn't want to interpret as entropy-minimizing/maximizing). It's fine but not important. The important innovation here is showing that if you use random reward functions, roughly like 'assigning arbitrary values to all possible states', you still get power-seeking.

This is important because one of the arguments people have always used against Omohundro drives is "but you haven't proven that lots of reward functions will want to seek power, aren't you just anthropomorphizing from a few special cases? Sure, entropy-seeking will seek power, but that's just one of a whole universe of possible reward functions; if that's dangerous, just don't use it: I would simply not make the AI dangerous. And I will use this as an excuse to dismiss the dangers from any other useful reward function you propose too."

But now we have an example of how power-seeking is the default, and the burden of proof reverses to be on anyone who thinks that AIs will just not be power-seeking, to explain how that very special passivity will come about.

blamestross6• 5 years ago

The real problem isn't that AI is power seeking, it is "what will make AI any better at it than us". I haven't seen an AI safety problem that isn't just already a problem existing companies have under capitalism. The difference between an AI and a large company is just one of substrate and desperate postrationalization to avoid the realization the singularity is long past and the rapture of the nerds left almost all of us behind.

blamestross6• 5 years ago

I'm laying my bet here for future lawsuits.

The paper you just cited (which I am familiar with) is generally considered a dead end. Even if it is correct it isn't useful. We don't have models good enough to calculate future entropy for practical problems. If we did have models that good then we wouldn't need the AI to solve the problems.

Nevertheless I expect this paper will be considered "before it's time" in another decade.

heyitsguay• 5 years ago

This is a really cool paper (that I definitely haven't had the time to fully grok) but the jump from the theoretical environment to concerns about superintelligent AI taking over is a bit strained. It misses the same key issues that most similar arguments miss, even while presenting some really amazing results on statistical properties of optimal policies - there's a lot hidden in the "if physically possible" part of the quote from the paper: "Average-optimal agents would generally stop us from deactivating them, if physically possible". I can see how the arguments in the paper lead us up to the possibility of kinetic conflict with intelligent robots, but the dynamics of power at that stage then seem to be more about how we've embodied those robots rather than optimality results of their control policies. Did we give them off switches? Will they run out of energy after a while? How susceptible to physical damage are they? If someone outfits a million Predator drones with a control policy with a will to live, we may have trouble, but we probably were going to have trouble from the group that decided to build a million Predator drones, too. If the number is lower, more realistic, then you're talking about standard armed conflict.

I could see this line of work having implications for autonomous weapons systems in a hypothetical future, but my understanding is that at this time they're on a pretty short leash with very narrow AI systems if they're using AI at all. Incentives could push those systems to get smarter as the autonomous arms race heats up, and if it does so in an unrestricted manner that could spell trouble along the lines discussed in this paper, but incentives are definitely aligned for any group to build weapons systems that can't be turned on their handlers, whether by human hackers or their own volition. Theories on optimal policies have no bearing if there's some encrypted remote kill command, or if as we start understanding ML models better, we can do things like hardware-block policies that lead to certain predicted outcome sequences (blocking an off switch, harming a human, etc.). Ultimately, the dynamics here seem to be an evolution of the problems of human force and violence, rather than a qualitative jump to a different sort of threat entirely.

314• 5 years ago

Without reading the paper it seems from the tweets that the results are overstated. Academics need to make their results sound more impressive and less trivial and so the choice of wording reflects that. The property that optimal agents seek is not "power", which is quite a general term - it looks more like "agency" i.e. a more specific form of power. But "Optimal agents seek agency" won't win any future grants.

Turn_Trout2• 5 years ago

Maybe you should read the paper, and/or the reviewer threads (as we discussed the nomenclature, and eventually agreed that "power" was accurate). We straightforwardly formalize a mainstream definition of power and show how it's more intuitive than the current standard measurement (information-theoretic empowerment).

Turn_Trout2• 5 years ago

First author here. Thanks for your comment!

there's a lot hidden in the "if physically possible" part of the quote from the paper: "Average-optimal agents would generally stop us from deactivating them, if physically possible".

Let me check that I'm understanding correctly. Your main objection is that even optimal agents wouldn't be able to find plans which screw us over, as long as they don't start off with much power. Is that roughly correct?

Theories on optimal policies have no bearing if

See my followup work [1] extending this to learned policies and suboptimal decision-making procedures. Optimality is not a necessary criterion, just a sufficient one.

if as we start understanding ML models better, we can do things like hardware-block policies that lead to certain predicted outcome sequences (blocking an off switch, harming a human, etc.)

I'm a big fan of interpretability research. I don't think we'll scale it far enough for it to give us this capability, and even if it did, I think there are some very, very alignment-theoretically difficult problems with robustly blocking certain bad outcomes.

My other line of PhD work has been on negative side effect avoidance. [2] In my opinion, it's hard and probably doesn't admit a good enough solution for us to say "and now we've blocked the bad thing!" and be confident we succeeded.

[1] https://www.alignmentforum.org/posts/nZY8Np759HYFawdjH/satis...

[2] https://avoiding-side-effects.github.io/

Nevermark5• 5 years ago

If a group of AI instances can collectively audit each other's behavior, their best policy is to eliminate other AI's that threaten the group.

Likewise, each individual's incentive is to either:

1. Support the audit of all AI's, and avoid doing anything that would be a threat.

2. Or, only if extremely sure of ones ability to eliminate all the other AI's and keep all preparations secret, do that.

As the number of AI's go up, presumably case 2 would get more and more unrealistic.

Obviously, this is a toy version of a real solution.

Optimal AI agents tend to seek power

Related Stories