Using Metadata to Find Paul Revere

http://kieranhealy.org/blog/archives/2013/06/09/using-metadata-to-find-paul-revere

by decklin • 13 years ago

457 73 13 years ago

kieranhealy.org

msandford2• 13 years ago

This is a really great counterpoint to the "if you have nothing to hide..." crowd.

mseebach• 13 years ago

Well, Paul Revere did have something to hide.

msandford2• 13 years ago

Yes he did. The point is that having something to hide doesn't necessarily mean you're a criminal or a terrorist. Or that in some situations what our government today would classify as "criminal" or "terrorist" is in fact an act of patriotism.

We all know that history is written by the victors so I won't suggest that what Paul Revere and the other revolutionaries did was an unmitigated good.

But I will argue that needing to expend the resources it takes (salary, benefits, management, etc) to have one human being monitor another does put a pretty good check on the ability of those in power to stifle dissent. It also provides some kind of a critical threshold over which the revolutionaries can't reasonably be stopped. In many ways that's a good thing provided it's a high enough level. Once the public opinion swings the best thing to happen is for those in power to give up on suppression and start packing their bags.

quchen2• 13 years ago

While being an interesting read, the "old world" scenario was kind confusing me every couple of lines. (Could be because I'm not a native speaker.)

aidenn0• 13 years ago

If you're not a native speaker, you're probably also not as well versed in the folklore of the US revolution. Paul Revere is a popular hero of the US revolution, so using him as the target terrorist invokes "one mans terrorist is another mans patriot" argument at the same time as it demonstrates the usage of metadata.

quchen2• 13 years ago

Indeed I didn't know about him, thanks.

gwern• 13 years ago

And it cuts doubly, because a lot of the tactics of the rebels were what we would call terrorism these days - tarring-and-feathering is simple torture and terror tactics.

jroseattle• 13 years ago

Makes me think of the ramifications of false positives.

danso2• 13 years ago

I don't know if David Simon's (creator of "The Wire" and "Treme") defense of the Verizon phone records request ever made it onto HN's page:

Here it is, if you hadn't seen it: http://davidsimon.com/we-are-shocked-shocked/

Apparently it was so controversial that his site crashed from the traffic, and he had to tell everyone to chill out: http://davidsimon.com/nsa-and-fisa-commentary-calling-it/

Anyway, why I thought of that in relation to the OP was, that I think some defenders of the NSA and general government surveillance policies are just unaware of how technology can fundamentally change things...As Google leaders have been known to say, "Speed is a feature"...and so it's not the finding of information that makes the establishment of Google time in human civilization, but how fast Google allows us to do it.

So that said, Simon is one of the journalists I have the absolute highest regard for...I'll be one of the many who think "The Wire" is the best TV drama ever, both for its artistic take and for its illustration of how institutions -- the police, the schools, the drug trade -- corrupt even the best of individuals. "The Wire" is heavily based off of the year that Simon embedded himself in the Baltimore homicide department...the book (which spawned a network TV show) is the best book about the practice of journalism I've ever read. After a year following the detectives, you'd think Simon would be pretty much in cahoots with the police...but he followed up "Homicide" with "The Corner", in which he spent a year embedded with drug dealers and their customers...apparently most of the friendsships he made in the Baltimore Police department evaporated after he published a book bringing sympathy to Baltimore's downtrodden.

Anyway, I don't think Simon has a love for government or authority. But I do think he's a little naive when it comes to advances in technology and their consequences. When "The Wire" started, the police were focused mostly on tapping pay phones. By the time "The Wire" ended, the police were surprised at the advent of camera phones. So when Simon says he thinks the NSA and other law agencies won't abuse their wiretap authority, I believe him...because in much of his experience, the practical obstacles (such as, having to have an officer watch a payphone all day) made it basically impossible for blanket surveillance.

But technology is different...I think Simon's -- and others who I respect -- mistake is to think that the game is being played the same as it always is. It may be the case that the NSA is staffed with as people as good and conscientious as anywhere else...but it's naive to think (as was the primary lesson in "The Wire") that the power they have will lead them astray...and to those of us affected by it, it makes no difference if the violations were intentional or accidental.

Anyway, back to Paul Revere and the OP...I think it's a great example. But of course, what makes that educational scenario feasible is technology and the ability to record information (metadata or whatever you want to call it) in an organized way.

Frankly, I kind of thought anyone who read 1984 would understand how technology changes everything. But yeah, I do think there are some well-meaning people don't grasp the technology, and if they did, they'd have a different opinion about the dangers of unchecked surveillance.

* edit: misspelled conscientious as 'contentious'

* edit: as an example of how much Simon continues to challenge the police as a citizen, here's an essay he wrote after the success of the Wire, in which he tried, as a citizen, to get the basic details of a cop-involved shooting, something that has always been public record. He eventually succeeded, and the revelations about the officer involved ended up jeopardizing the prosecution: http://davidsimon.com/in-baltimore-no-one-left-to-press-the-...

But as you can see if you read the piece, Simon is not to thrilled with how the Internet has displaced newspaper journalism

colin_jack• 13 years ago

Good point, and you don't even need to be very imaginative about the future, imagine Google Glass but with the government quietly able to pull down what you are or were seeing.

yesbabyyes• 13 years ago

In the words attributed to Joseph Stalin: "Quantity has a quality all its own."

coldtea• 13 years ago

Well, this particular quote might come from Stalin, but it's based on an old Hegelian idea, that of a surplus of quantity becoming quality (add enough mass and you get a black hole, temperature over 100oC and you get boiling water, etc).

(Makes sense, since Marx, Lenin, Stalin et co were heavy readers of Hegel).

From Wikipedia: "One important dialectical principle for Hegel is the transition from quantity to quality, which he terms the Measure."

http://en.wikipedia.org/wiki/Dialectic#Hegelian_dialectic

Also check the ancient greek notion of the "sorites paradox", which deals with a similar idea:

http://en.wikipedia.org/wiki/Sorites_paradox

gohrt2• 13 years ago

Simon wrote in the comments section on his post:

Next, if you’ve read carefully you know that I’ve saved my criticism not for this data dump, which I do indeed support, but for the all-enveloping secrecy that prevents proper civilian oversight of the FISA court.

This is called being disingenuous. The problem has two halves: (A) the data, and (B) controls over the data.

Simon is calling everyone foolish because he thinks the problem is B, not A. He completely fails to acknowledge that the problem is A+B, and if haven't won on B, we have to fight to defend A.

I don't mind an NSA with all data and no abuse/breaches. But that's not going to happen!

I don't care if the NSA is completely independent and covert, if they don't have any data to analyze!

And from what I know of human nature, I know where I want to plant my defenses. Just like the people who wrote the Constitution, I prefer to limit the power of the government officials, not give them unlimited trust.

notdrunkatall• 13 years ago

I know this is frowned upon, but I wanted to commend you on crafting such an eloquent rebuttal.

grey-area3• 13 years ago

I agree that technology changes the game because it's a multipler.

There's also an insidious quality to current surveillance in that it is not just pervasive over space, but over time. Once they have a database like that envisioned in PRISM and the Verizon records, it could be used to travel back through any person's intimate life looking for weak points, failures, emotional attachments, and transgressions, very useful in discrediting political opponents or inconvenient journalists, and very useful in quelling any revolts by citizens before they even begin. These are currently the methods that intelligence agencies use to turn assets and use them for their own ends on a small scale, but should we allow them the tools to use these techniques against an entire population simultaneously and retrospectively? What would the long-term outcome of that sort of power be?

I also do wonder why the methods they use should be secret in the first place? I completely understand the need for operational security on specific operations or details of methods, but if a huge broad program like this is done in a nation's name, surely they should know what is being done and approve or disapprove of it? I can't imagine there are any terrorists the professed target of these programs, who are not aware of widespread monitoring, so I find it hard to believe the top secret nature of the programs is about security so much as avoiding oversight. The avoidance of providing specific details to congress from former BAH executive Clapper is a good illustration of this - if congress can't get broad figures out of them, who are they accountable to?

So the combination of power multiplied by technology and the lack of accountability is the difference between this and tapping telephone lines on an individual basis.

unreal37• 13 years ago

This reminds me of the Wired article on the big NSA data center that is presumably recording every packet it can get its hands on for later analysis.[1]

This idea that the big coup here is that when they decide they want to target you, they already have a huge cache of intel on you. Like the Boston marathon bombers - at 9am they identify you, and at 9:15am they know everything there is to know about you since they've already collected it on everyone.

[1] http://www.wired.com/threatlevel/2012/03/ff_nsadatacenter/

dfc4• 13 years ago

Where does it say the revelations about the officer ended up jeopardizing the prosecution? And do you know what the final result of the prosecution was?

danso2• 13 years ago

Sorry, that came about after Simon's essay:

http://articles.baltimoresun.com/2009-08-05/news/0908040089_...

A man who was jailed without bail for months because a Baltimore police officer said he tried to disarm her was set free Tuesday after Officer Traci L. McKissick changed her story during emotional courtroom testimony.

Earlier this year, McKissick told prosecutors that Joseph A. Forrest was the man who stepped on her hand as she held a gun and wrestled with Forrest's 61-year-old uncle, who was killed by police during an altercation in February.

jabbernotty• 13 years ago

but it's naive to think (as was the primary lesson in "The Wire") that the power they have will lead them astray...

Is this what you meant to say? I must admit, I had expected

"but it's naive to think (as was the primary lesson in "The Wire") that the power they have will not lead them astray..."

bmelton• 13 years ago

I just read his recap, which seems to mirror season 1 of "The Wire" fairly closely. What I remember distinctly from that season (which I've seen in total at least three times) is the great degree of difficulty they had in obtaining the warrants, in meeting the burden of proof, and in convincing a judge that the burden was met to a high enough level to allow the kind of carte blanche access to such a broad tap.

Further, they were constantly in fear of running into the expiration date, and the battle was waged on whether they could get data of enough substance before the tap's expiry that they would be able to extend it.

The problem in reconciling the two situations is that in the Baltimore case, assuming the show faithfully retold that portion, it was hard, but in the case of the NSA, their approvals come from the FISC, which numerically is reduced down to a rubber stamp agency. Since their inception in 1979, up to 2012 or so, they have approved over 30,000 requests, and rejected only 11. Further, it seems that they're somewhat unique in a warrant-issuing agency in that they've provided assistance on those 11 such that requests were amended, reissued, and in many cases, granted. So of those 11 denials, I am led to believe that the majority of those proved to be slightly delayed approvals in reality.

Beyond that, I think that Mr. Simon overlooks the capacity of metadata at that scale, and how effective it could be in painting a picture in a way that smaller scale, Baltimore-wide data could never do. If, for example, the British had been able to capture 'only the metadata' of our founding father's meetings in Philadelphia, it wouldn't have taken much analysis at all to have proven what was happening (famous statesmen gathering in Philadelphia court house for overnight meetings, etc.) Just knowing the attendee list alone would have likely been enough to condemn them all for treason or assassination, and the United States might never have existed.

csears• 13 years ago

So could we spoof bogus metadata to hide Paul Revere?

In other words, if a large enough number of people started start calling and texting random Verizon customers, tweeting with random people from the middle east, inviting random people with Muslim-sounding names to Google+ Hangouts, and commenting on every Facebook like with "This ____ is the bomb!"... could that tip the signal to noise ratio enough to defeat this type of analysis?

maxerickson• 13 years ago

I think if you had enough people engaged that you could do anything to affect the analysis that you might as well work to eliminate the injustice (rather than just smearing it out across more innocent people).

blktiger• 13 years ago

If you haven't read it yet, check out Cory Doctorow's book Little Brother (http://craphound.com/littlebrother/about/, http://craphound.com/littlebrother/download/). He explores ways that average citizens could fight back against technology that watches our every move.

crgt• 13 years ago

Maybe a simple mail client that adds 'extra' metadata to everything you send? Extra recipients, etc? Would add more hay to the haystack.

antoko• 13 years ago

This is very clever and all, but from the British perspective - this is exactly what they would have been looking for and Revere would be considered a revolutionary and a real threat, so this is the system working as it is supposed to.

This is the equivalent of saying that the NSA can use such metadata to find muslim extremists that want to kill your children and implement sharia law. (hyperbole courtesy of American media)

Changing the timeline such that "americans" are the underdogs rather than the establishment isn't really helpful... is it? Or is this attempt to stoke revolutionary sentiment?

ant5122• 13 years ago

It spins the story around in such a way that Americans can relate to the dangers of a surveillance state.

You're suggesting that the NSA is just looking for bad guys and that they should be able to capture evil muslim extremists.

But now the US government decides to bring back alcohol prohibition. Drinking, they say, is immoral, and so are the people who drink. Do you still have nothing to fear?

Abortion gets outlawed, and its supporters are branded as immoral. Still got nothing to fear?

Creationism gets accepted as a viable, scientific alternative to evolution and gets taught in all schools, and people who believe otherwise are dangerous subversives.

Are you now one of the bad guys? Are the NSA now interested in you?

I find it interesting that Americans can believe that the right to bear arms will protect them from the world's largest military, should it ever be run by tyrants, but at the same time believe that the government will always share the same moral code that they themselves hold.

euroclydon3• 13 years ago

It's often been said during during debates about government waste and inefficiency, that citizens do not actually want an efficient government. The typical example is: would you want a boot to appear on your wheel, in your driveway, the minute your license plate expires? Because that would be efficient.

The average person feels that they are getting more benefit out of Google searches, Gmail, Facebook, cell phones, and credit cards for that matter, then they are being harmed by volunteering so much information to the corporations who run these businesses.

If you think the U.S. is headed in the direction of outlawing alcohol or teaching creationism is all schools, I respectfully suggest you find something different to worry about -- maybe pot... wait, we are becoming more lenient on it as well.

I find it a bit silly, that when the government adopts new technology to fight crime, everyone brings up their pet boogey-man to try to strike fear in their like-minded citizens. I honestly can't understand if you are in support of, for instance, using web surveillance technology to capture and prosecute abortion clinic bombers or not.

clarkmoody• 13 years ago

Your evident bias notwithstanding, I believe what you're trying to emphasize is the danger of an over-strong, over-active federal government directed by ideologues of any flavor. When the government uses its power against its own citizens, dividing them to make them unequal, then it flies in the face of the founding documents and ideals of this country.

The IRS taking political sides is a recent egregious example, but since its inception, the tax code has been used as a political power tool to divide us, giving special treatment to some groups over others. This is why politicians hate the idea of meaningful, economically-sane tax reform.

You mention the citizenry bearing arms against its government and the full force of the military directed against the people. Why take it to that extreme? You must realize that with current technology, small arms are the great equalizer among Men (and women). Think 110 lb jogger with a .45 vs. thug rapist. Now large government types want to take guns away from the citizenry for ideological reasons, but those reasons make the assumption that the people must be taken care of and protected by the government.

And that is where the big vs. small government argument hinges. Should the government protect the personal well-being of each citizen? The Founders said that we are born with the right to the "pursuit of happiness" not to the happiness itself. Too many on the big-government bandwagon have been promising the happiness for too long. Too many judges have been siding with the government over the rights of the people.

An overactive surveillance state once again operates under the assumption that the people are unable to take care of themselves. We MUST monitor all traffic so we can catch even the smallest act of 'terror,' destroying the rights of the people in the process and giving the government huge power over them. And speaking of traffic, the CAFE standards for autos have killed[1] more Americans than the terrorists have.

Our national dialog needs to be framed in the "Freedom vs. Security" trade off, but I'm not so sure that many people even understand what freedom is / was supposed to be.

[1] http://www.americanthinker.com/2010/04/death_by_cafe_standar...

zero_intp3• 13 years ago

I think you miss the point entirely. Of course the classic Revolutionary position was treasonous to the British Empire.

The function of dialogue like this is to push the metadata conversation into the mainstream. We want the People to understand that giving away our privacy in this data is /JUST/ like taking their guns, and for the same reasons.

gohrt2• 13 years ago

Note that guns-for-defense-against-tyranny gun owners are not on the Government's side on this issue. It's the anti-gun nanny-state constituency that is supporting the government.

The group that both guns and supports domestice spying is overlap of "I just use guns for shooting rabbits for fun, and I don't like brown people, and I trust the government", not the groups that says "I need my guns in case the government tries to steal my land."

outside2344• 13 years ago

let's say in the future, Christianity is made illegal. Is the above scary now?

alan_cx• 13 years ago

Organised religion is IMHO, evil. In fact, I consider it to be the manifestation of the false gods the bible itself warns about. I cant ever get out of my head the way Bush Jr used Christian rhetoric to justify his two wars.

So, at the very least, that example doesn't really work for me.

angersock2• 13 years ago

I consider this article not as an attempt to discredit these techniques by making patriotic references, but as an excellent starting point to help people start building instincts about what their "useless metadata" could actually be used for.

The great challenge we face in educating the populace (and indeed, ourselves) is that people lack any intuition for what sorts of data they should or should not broadcast. People are hesitant (I hope!) to give out home addresses to strangers without good cause--we have no such innate hunches about the data and metadata we leak.

lettergram• 13 years ago

This would mean the larger your network, the larger your likely hood of being a terrorist

pseut• 13 years ago

From a predictive standpoint, that's not necessarily an error. Think of it as "the larger your network, the higher priority to investigate for terrorism."

rickyconnolly• 13 years ago

You ʃhould have uʃed the long ʃ in this article

surlyadopter• 13 years ago

The purfuit of happineff?

deadairspace• 13 years ago

The character is ſ, and it doesn't go at the ends of words, so it would be: The purſuit of happineſs.

LiamMcCalloway3• 13 years ago

Academic version: http://db.tt/6UA1bXZq

DIVIDE AND CONQUER: DISTORTED COMMUNICATION IN NETWORKS, POWER, AND WEALTH DISTRIBUTION Wilson Perez-Oviedo

Cornell University and Banco Central del Ecuador

Abstract In a society composed of a dictator and its citizens, what are the determinants of the political equilibrium between these two? What are the conditions for a successful citizens’ revolt? What kind of strategies do governments follow to prevent such revolts? The situation of these types of societies can be understood as a game played between a leader, who has to decide the distribution of the aggregate income, and a group of citizens who have the opportunity to revolt if they are unhappy with the distribution. Coordinated action by citizens is possible because they form nodes in a communication network. However, communication through the network is distorted, which could preclude the emergence of collective action among citizens. The network structure and the distortion level are determinants of the political equilibrium and wealth distribution. The model explains how the dictator could use propaganda, cooptation, and repression to increase his expected utility. Finally, the model is illustrated by applying it to cases in Nigeria and Zaire/Congo.

dfc4• 13 years ago

Link to version that does not require signing up for and signing into drop box:

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=522362

Big brother's little sister is just as bad. Why not give the ssrn link?

LiamMcCalloway3• 13 years ago

I didn't check whether the paper was publicly available and not on gated publishers' websites, my apologies.

dmix• 13 years ago

This paper is excellent. Thanks for sharing.

LiamMcCalloway3• 13 years ago

You're welcome. I'd be happy to read more on the same topic, if you stumble upon anything of interest.

startupfounder• 13 years ago

Paul Revere, "in the Fall of 1774 and winter of 1775, I was one of upwards of thirty, chiefly mechanics, who formed ourselves into a Committee for the purpose of watching British soldiers and gaining every intelligence on the movements of the Tories."[1]

Paul Revere was essentially one of the founding members of this country's counter intelligence program against his oppressive government. He was the first in a long line of Mark Felts, Bradley Mannings and Edward Snowdens of this country.

If we have a system where these whistleblowers are stopped before they can leak information on the system that catches them before they can whistleblow there is no turning back. This would not be the UNITED STATES OF AMERICA without Paul Revere and Edward Snowden.

[1]http://en.wikipedia.org/wiki/Intelligence_in_the_American_Re...

mpyne6• 13 years ago

You do realize that Paul Revere would by that logic be considered a founding member of the country's counter intelligence program against enemies of the U.S. as well, right? I.e. the NSA.

As far as Revere was concerned he was way past fighting against his government; he was fighting to support his country against Britain's government.

flyinRyan3• 13 years ago

What are you talking about? Britain was his government/country. By today's definitions the man was a terrorist aid. The founding fathers were all terrorists doing things like the Boston Tea Party and shooting only officers when rules of the day prohibited that kind of behavior.

tzs• 13 years ago

Did Paul Revere gather intelligence concerning both sides, and release everything he had to both sides, so that he would help his enemies as much as he helped his friends, and hurt his friends as much as he hurt his enemies?

If no, then the long line Revere is the first in does not include Manning.

BCM432• 13 years ago

Do you have any evidence that Manning hurt his enemies?

lawnchair_larry• 13 years ago

By that definition, the Manning line doesn't even include Manning.

flyinRyan3• 13 years ago

Yes, but the US government is now in the position the * British government* was then. So telling them that their actions will prevent another Paul Revere isn't going to win them to your side.

kevinpet2• 13 years ago

This is a lot of fancy math and graphics that don't differ all that much from what a very simple analysis can give you. Only Revere belonged to five of the groups of interest. Of the three who belonged to four groups, two of them are in "top scorers" on centrality, and the third, with one of the two in centrality, is in the final table.

rollo_tommasi• 13 years ago

It's an exceedingly simple example used to illustrate the technique. The methods included in the python library are robust enough to work just as well on much larger and more complicated data-sets but that would be too difficult to follow for someone who isn't familiar with the subject.

kevinpet2• 13 years ago

I think it's hard to understand what the point is without a result that differs from the naive solution. That is, my naive "who's an important terrorist" is "who belongs to the most terrorist groups".

dfc4• 13 years ago

kjh the author of this link has a greatintroduction to emacs:

http://kieranhealy.org/resources/emacs-starter-kit.html

He also has a lot of good reference material on latex/org-mode/pandoc. Definitely a good resource to have handy if you ever have a less-techie friend who wants to get away from MS word and its ilk.

Using Metadata to Find Paul Revere

Related Stories