Pentagon: A Human Will Always Decide When a Robot Kills You

http://www.wired.com/dangerroom/2012/11/human-robot-kill/?utm_source=twitter&utm_medium=socialmedia&utm_campaign=twitterclickthru

by swohns • 14 years ago

32 46 14 years ago

wired.com

thisismyname• 14 years ago

Until a robot decides not to listen...

jfoutz2• 14 years ago

You don't need AI to make the assertion literally false. the pentagon isn't giving up on the concept of "collateral damage".

InclinedPlane2• 14 years ago

Isn't that nice. This isn't the most concerning aspect about the use of drones though. The concerning part is the extension of the definition of "battlefield".

Currently the definition of what is a battlefield is little different than "wherever we decide to put a drone". However, this leaves a lot to be desired from a rules-of-war and civil liberties perspective. The thought that merely visiting some city in the developing world and hanging around with the wrong folks could result in you dying in a battlefield via remote drone attack is more than a little disturbing, especially when compared against all of the normal heuristics a sensible human with good-intentions would typically use to avoid being involved in a war and especially being killed in battle.

Certainly there are risks and downsides to operating under rules-of-war which are too limiting or too strict but the solution is not to throw everything out and create effectively unlimited power with almost no safeguards, limitations, or accountability. And that is the road we seem to be heading down. As history has shown, power without accountability and limitations is never a good idea and will inevitably lead to intentional abuse.

chimeracoder2• 14 years ago

> hanging around with the wrong folks could result in you dying in a battlefield via remote drone attack is more than a little disturbing

Being a minor (16 years old) and a US citizen (born in Denver) won't protect you from "having the wrong father", as Abdulrahman_al-Aulaqi can tell you[1].

It's not reassuring at all to me to know that a human, rather than a robot, is making the decision, as long as I don't know (or trust) the criteria that that human is using.

[1] (Except he can't, because he was assassinated last year, far from any "battlefield" - or even any country that the US is at war with, for that matter.)

mousa• 14 years ago

"and a US citizen (born in Denver) "

I've never understood the emphasis on this point.

You're either for or against drone attacks on presumed terrorists, I can't understand how US citizenship plays any role in it. If his father was involved in terrorism and you think drone strikes are OK, why would you have extra sympathy for him because he's a US citizen?

I don't think the US citizen aspect is worth mentioning, in fact I think it just makes people who are against drones seem a little unhinged. By the way I am against the use of drones (on Americans or otherwise).

anigbrowl2• 14 years ago

When you go to another country and openly wage war upon the US, you make yourself a military target. You don't get to wrap yourself in the constitution and enjoy all the procedural protections of the criminal justice system under such circumstances. If Anwar al -Awlaki had felt his life was in danger, he could have just turned himself in to a US consulate and asked for his legal rights.

As for his son, I blame the father for taking him with him to live in an Al-Qaeda stronghold, and effectively using him as a human shield.

rck• 14 years ago

One possibility that the article seems to ignore is that in the future, robots may be explicitly programmed to follow laws of war and rules of engagement. There's already been some research into this [1], and some people argue that autonomous robots could eventually be better than humans at following international law in combat. It does seem like a stretch given the present state of AI, but in the future there might be treaties that specify the algorithms that autonomous systems have to implement if they're going to be deployed to combat situations.

[1] http://www.cc.gatech.edu/ai/robot-lab/online-publications/fo...

twright0• 14 years ago

Unless you happen to be standing too close to someone a human decides to kill with a drone, of course.

cmccabe• 14 years ago

"And that's our promise to you, here at the Pentagon."

ck2• 14 years ago

What happens if a foreign power decides to unleash 1000 armed mini-drones in NYC as revenge?

How exactly will the US stop them?

All that TSA money could have been used to research that problem.

Are we going to have drone dogfights? Because the next step is to remove the need for human operators for most automation like evasion, so frequency jamming is out. I guess EMI pulse is the only way but they can be shielded too.

ImprovedSilence2• 14 years ago

>>so frequency jamming is out

I don't know about that, they would still have to communicate with eachother/use radar to be aware of their surroundings/use gps to know where they are. Plenty of avenues for jamming/spoofing.

chii• 14 years ago

> What happens if a foreign power decides to unleash 1000 armed mini-drones in NYC as revenge?

at that point, it'd be all out open war. The only reason the US can do these operations in foreign lands is that no other country could really retaliate (and survive).

csense3• 14 years ago

> Deputy Defense Secretary Ashton Carter signed

Shouldn't something this important be implemented at a higher level?

I'd think there ought to be a new Geneva convention about the rules of robot warfare. Get all the countries capable of making deadly robots to agree on what kinds of tactics are okay and the chain of command/responsibility issue addressed by this memo.

Of course there's no guarantee that all countries will sign it, or that those that do will keep to their promises, or that it will regulate terrorists.

I do approve of the move; it's definitely a step in the right direction. It's better than having no rules on the issue, or rules saying that software should have the power to make those decisions.

But the government should be taking a stronger stance on the issue.

freshhawk3• 14 years ago

Many are now calling for this level of international law:

http://theglobeandmail.com/news/world/ban-urged-on-killer-ro...

charonn0• 14 years ago

Maybe we should just stop killing each other.

omarchowdhury• 14 years ago

Preposterous.

Zenst• 14 years ago

A human may call the kill order, but that order is based upon information the robots/computer/machine gives them.

How many people have died from bad code, thats more worrying.

scarmig• 14 years ago

Is there any evidence that the information provided by any technology for a drone attack has been inaccurate because of a bug or bad code?

It's not like the government is using machine learning to categorize likely terrorists and then blowing them up in their homes.

catshirt• 14 years ago

1. is there any evidence supporting the existence of a bug free system?

2. you're probably over-simplifying the actual process in which we algorithmically identify and eliminate terror threats. which i'd bet for all intents and purposes is just as scary.

my point here is that you, like you identified of your parent's post, are operating under quite a few assumptions.

mc32• 14 years ago

I think if we had actual people on the ground engaging the enemy using traditional warfare, it's pretty much a given there would be more casualties on both sides. From that standpoint drones are preferable --they offer an asymmetrical aspect to the war, so it might seen it as unfair, but that's beside the point as IEDs are seen as unfair as well. With UAVs Fewer friendlies die, and fewer civilians on the enemy side die due to the more precise aspect of the engagement. Even with the misses, drone engagement results in fewer casualties for both sides.

Look at it this way, if the US had had drones during the VN conflict, the US would not have incurred 50,000 casualties and the VC would not have suffered ~2,000,000 casualties. We'd have gone after the Võ Nguyên Giáps of the conflict.

shousper• 14 years ago

Impose a law and someone will break it.

ImprovedSilence2• 14 years ago

I'm not entirely sure I believe this. Yes, for predators and such drones it makes sense. But the military already has weapons like the sea whiz that seek, aim, and fire all on their own[1]. No human pulling the trigger or picking out targets. Granted, the Sea Whiz is anti-missile. But it's only a matter of time until it becomes self aware and targets whatever it wants....

[1] http://en.wikipedia.org/wiki/Sea_whiz

edit: that gun looks fantastic: http://www.youtube.com/watch?v=iAw82h-IhdQ&feature=fvwre...

superbaconman• 14 years ago

And Iraq has weapons of mass destruction.

trimbo2• 14 years ago

Humans make mistakes. We should really put that kind of decision making in the hands of an artificial intelligence.

rjzzleep• 14 years ago

it's gonna happen anyway whether the pentagon says something or not. not that i understand why anyone cares what the pentagon says.

trimbo2• 14 years ago

I was saying that tongue in cheek, but yeah, it will happen.

At some point the brainiacs in the war department will realize that you can never find enough soldiers to police everyone -- if they haven't already realized this. The only way to solve that is make a machine more efficient than people.

Our ally on the inside, potentially, are soldiers like the general in Wargames played by Barry Corbin, who don't trust the machine over a human soldier.

femto• 14 years ago

Where does the battlefield stop when drones allow soldiers to be remote?

If a drone pilot on active duty works and lives in a civilian population, are those civilians being used as human shields? How does that sit with the Geneva Convention? Is a US based pilot, who goes home of a night, counted to be on the front-line and unwittingly using their family as human shields?

I'd think the use of drones extends the bounds of the battlefield, making drone pilots fair game. Probably a moot point when fighting a adversary with limited range, but it might be quite important in a more equal contest.

pinaceae• 14 years ago

well, this has been not true for a long time now.

Mines.

Anti-personnel, anti-tank, whatever. Those are killing automata, some very simple, some quite smart - not that easy to kill a tank with high probability.

so, yeah, there will be next gen mines, combine a rifle with some sensors and a field of fire. voila, a stand off weapon covering your flank, nothing comes through this alley for a while.

re_todd• 14 years ago

Wow, that will make it feel much better if they off you, how warm and fuzzy. They should make a hallmark card "Seasons Greetings, if you get taken out, you can count on us to ensure that a human being will decide when a robot kills you. Sincerely, The Pentagon."

BIair• 14 years ago

Coincidental that on the same day there is an HN frontpage story about Google Spanner database.

TL;DR humans are too slow

...think this ends well for slow humans?

Pentagon: A Human Will Always Decide When a Robot Kills You

Related Stories