XKeyscore: NSA program collects 'nearly everything a user does on the internet'

http://www.theguardian.com/world/2013/jul/31/nsa-top-secret-program-online-data

by sinak • 13 years ago

1688 612 13 years ago

theguardian.com

mtgx• 13 years ago

Nice pre-emptive "attack" by Greenwald today, just before the NSA hearings.

barylen• 13 years ago

I was worried the leaks may have peaked a bit early, but this was very well timed.

griffordson8• 13 years ago

The House hearing was canceled several days ago to make time for the House Democrats to meet with Obama this morning. As far as I know they have not been rescheduled yet and I will be surprised if they happen before the August recess. Although Glenn Greenwald did say yesterday he hopes they get rescheduled in the next 24 - 48 hours.

w_t_payne17• 13 years ago

So it seems.

falk3• 13 years ago

Greenwald said they cancelled the hearing on Twitter yesterday. Do you know if they rescheduled it yet?

hga• 13 years ago

This bit both somewhat limits the impact and makes Greenwald et. al.'s claims that most everything is being Hoovered up a lot more credible:

"The XKeyscore system is continuously collecting so much internet data that it can be stored only for short periods of time. Content remains on the system for only three to five days, while metadata is stored for 30 days. One document explains: "At some sites, the amount of data we receive per day (20+ terabytes) can only be stored for as little as 24 hours.""

Of course, as the article goes on to detail, anything that's found to be of interest in that window can be saved permanently, and NSA analysis do that a lot.

sixothree3• 13 years ago

"To solve this problem, the NSA has created a multi-tiered system that allows analysts to store "interesting" content in other databases, such as one named Pinwale which can store material for up to five years."

It does not describe "interesting". Maybe metadata, encrypted sessions etc? Any conversations in threads linking to these articles?

spahl• 13 years ago

But this was written in 2008. Storage capacity could/must have increased massively since then.

mpyne3• 13 years ago

But so has the volume of data that NSA would have to retain.

haakon2• 13 years ago

Notice that the presentation is from 2008 - a lot changes in five years.

griffordson8• 13 years ago

One such change is coming/has come online in Utah.

terpie• 13 years ago

These slides are from 2008... I'm sure their capabilities have increased significantly.

knotty66• 13 years ago

Tempora in the UK stores all internet data for 3 days and metadata for 30 days too. Wouldn't surprise me if most western governments use XKeyscore along with data sharing agreements with each other.

danenania3• 13 years ago

I would imagine that their top priority since 2008 has been improving storage systems. It's clear that their ultimate goal is a fully indexed archive of the entire digital universe.

mjfl• 13 years ago

Well then they are going to need that big data center. That is an unimaginable amount of data...

cpursley• 13 years ago

https://en.wikipedia.org/wiki/Utah_Data_Center

sentinel• 13 years ago

Yep. http://www.forbes.com/sites/kashmirhill/2013/07/24/blueprint...

ck2• 13 years ago

Imagine if storage limitations weren't holding back the NSA.

Those 60TB density HAMR[1] drives that are due in 2016 are really going to take invasive to a whole new level.

[1] http://storageeffect.media.seagate.com/files/2012/03/perpham...

sixothree3• 13 years ago

20 TB per day is nothing these days.

javert• 13 years ago

Imagine if the American people weren't holding back the NSA.

Oh, wait...

TsiCClawOfLight• 13 years ago

mmh... but maybe the military is also ahead in that area? Maybe DARPA has 1TB GB/s drives?

vidarh4• 13 years ago

I don't quite know what you intend "1 TB GB/s drives" to mean.

But note that you can buy off the shelf PCIe cards with SSD's mounted that will give you 1TB storage and an aggregate read bandwidth of more than 1GB/sec today. I've got three sitting in various servers. They're expensive, and frankly for the future I'll rather get a couple of extra SATA III controllers and get multiple "regular" SSDs on separate controllers for that reason, but they're available.

For NSA style data collection, though, the collection is trivially to do in parallel: Hash all keys to a "virtual bucket", and hold a map of virtual buckets to physical servers. Then when you want more capacity, you add some physical servers, reassigns some of the virtual buckets from other physical servers to the new ones, and synchronises any old data (given that NSA claims they could only hold the full data stream for three days, you don't even need the hassle of moving data, just make collection on different days map to different virtual buckets, so that on day one you "just" reassign virtual buckets the content of which is being expired on the old servers anyway, on day two, the next set etc. - you maintain full spread of read/write traffic by ensuring that in normal operation all servers have an even spread of "day 1", "day 2" and "day 3" buckets).

It's amusing they see storage as an issue, but of course this was in 2008. Today I have 6TB in my home NAS, and my perfectly off the shelf tower case can easily fit 40TB+ with current size harddisks (though I doubt the noise would make me popular at home).

w_t_payne17• 13 years ago

Who said that they were using spinning storage? You can store an insane amount of data for your dollar if you are willing t use tape.

shabble• 13 years ago

For example, IBM have a robot tape library that can store 900PB[1], Quantum can fit ~5PB per full rack equiv[2], and there are many many more.

[1] http://www-03.ibm.com/systems/storage/tape/ts3500/index.html

[2] http://www.quantum.com/products/tapelibraries/scalari6000/in...

ceautery• 13 years ago

I wonder how they store all that. Surely a side benefit of this could be NSA contributions to CS journals about database techniques.

Also I doubt the veracity of the claim that they collect "nearly everything". Wouldn't they show up on, say, Sandvine's Internet traffic reports? I think it's more likely this claim is made simply to generate FUD in the general population.

nikcub4• 13 years ago

Hadoop[0] and OpenStack[1]

I think the era of government being far ahead of commercial tech capability is over. The government mostly outsources now (a problem Snowden identified in terms of information control) or develops in-house with vendors.

[0] http://online.wsj.com/article/SB1000142412788732349560457853...

[1] http://www.youtube.com/watch?v=NgahKksMZis

286c8cb04bda• 13 years ago

The government mostly outsources now (a problem Snowden identified in terms of information control) or develops in-house with vendors.

There are a lot of fingers in that pie. Oracle, for example, has a National Security Group, whose job is to come up with "solutions" and then try to sell them to three-letter-agencies.

coldtea• 13 years ago

Also I doubt the veracity of the claim that they collect "nearly everything". Wouldn't they show up on, say, Sandvine's Internet traffic reports?

Why would they? They deal with ISPs, backbones and such.

quasque• 13 years ago

Another benefit could be realised in the future if historians and linguists manage to get access to all this data. I imagine that researchers of the social sciences would end up enjoying the same sort of large-scale collaborative projects that particle physicists or genomic bioinformaticians currently have with their huge datasets.

aryastark2• 13 years ago

I wonder how they store all that.

Well, now they have a massive data center in Utah. That's most likely where it's all going today. Standard open source tools are all you really need.

Wouldn't they show up on, say, Sandvine's Internet traffic reports?

No. If some script kiddie/hacker type installs a packet sniffer and logs all your traffic to your ISP, that won't show up anywhere. Traffic goes somewhere. You're sending packets out. Merely logging packets is entirely passive and undetectable.

codex• 13 years ago

The Guardian strongly implies this system is used to intentionally target US citizens in violation of the law, but then admits that would be "illegal." I wonder if the leaked presentation touches on this point.

jabbernotty• 13 years ago

The Guardian doesn't 'admit' anything (it wasn't hiding anything in the first place), and legality doesn't predict whether actions are being taken or not.

I wonder if the leaked presentation touches on this point.

That seems unlikely to me, as this is a technical presentation.

eksith2• 13 years ago

That UI looks awfully similar to a theme I've seen used in SharePoint Portal Server. I hope that's not what they use for the front end, but I wouldn't put it past them.

dualogy• 13 years ago

Having worked with SharePoint extensively for years, I'm highly confident this isn't the case. It's just plain-old crappy custom-coded HTML4 forms from what I can tell and similar colors, so one could easily think so. But at closer look, it definitely not any of the SP versions 2003, 2007, 2010 or 2013. Maybe 2001 but I highly doubt even that.

duncans• 13 years ago

Pretty sure it isn't - some of it looks like old ExtJS if anything.

The more worrying thing is that they're still apparently using IE6. [EDIT: OK the presentation is from 2008 but still!]

VikingCoder• 13 years ago

This, to me, is like showing the alleged UI on an Apple IIc:

It just about proves that it's not true, to me.

susi22• 13 years ago

One of the screen shots:

http://static.guim.co.uk/sys-images/Guardian/Pix/audio/video...

says: Top Secret Comm(?) REL() to USA, AUS, CAN, GBR, NZL

confirming the previous suspicions that many other governments are on board.

Der Spiegel actually has reported a few weeks back about XKeyscore [1] and that it is used by the BND (Germany's NSA). I.e. all this data is also available to the NSA equivalents of Australia, Candana, Great Britain and New Zealand.

Many Americans trust their government (unfortunately), will they also trust the other governments?

[1]:

http://www.spiegel.de/international/world/german-intelligenc...

http://www.spiegel.de/international/germany/german-intellige...

packetlss• 13 years ago

Not GER, it's GBR as in Great Britain

http://en.wikipedia.org/wiki/UKUSA_Agreement

bnmathm• 13 years ago

I'm pretty sure that's actually GBR - Great Britain.

colanderman• 13 years ago

It definitely is; look at the actual (hi-res) presentation: http://www.theguardian.com/world/interactive/2013/jul/31/nsa...

brown9-23• 13 years ago

Those are the Five Eyes countries: http://en.wikipedia.org/wiki/Five_Eyes

clicks4• 13 years ago

Good catch -- and really, I find it to be quite foreboding in terms of how indomitable it is precisely because of the secrecy of the program.

"This was a secret treaty, allegedly so secret that it was kept secret from the Australian Prime Ministers until 1973."

This is indeed a trend, and I speculate that NSA (and NSA-like entities in the other 4 eyes/countries) probably communicate information and abilities to prime ministers and presidents of the respective countries very selectively.

ceejayoz• 13 years ago

Bonus: The NSA likely can get around the "no spying on US citizens" by just requesting data from those governments, who proceed to pull it out of the NSA's web interface.

_djo_• 13 years ago

That's COMINT, or Communications Intelligence, basically the type of intelligence that XKeyscore is part of. It might say HUMINT if the intelligence was collected from human sources.

REL TO likely means release to.

As I've said before, the realisation that most countries do this sort of thing comes as no surprise.

yread• 13 years ago

This is brilliant, I love the screenshots:

Foreignness factor:

The person has stated that he is located outside the U.S.

Human intelligence source indicates person is located outside the U.s.

The person is a user of storage media seized outside the U.s.

Foreign govt indicates that the person is located outside the U.s.

Phone number country code indicates the person is located outside the U.s.

Phone number is registered in a country other than the U.S.

SIGINT reporting confirms person is located outside the U.S.

Open source information indicates person is located outside the U.s.

Network, machine or tech info indicates person is located outside the U.s.

In direct contact w/ tgt overseas no info to show proposed tgt in U.S.

It's quite easy to lose the protections of a U.S. citizen indeed!

sehugg• 13 years ago

The person is a user of storage media seized outside the U.s.

Interesting, so everyone who ever hit a MegaUpload link is potentially a foreign entity?

RyanZAG2• 13 years ago

Kind of puts into perspective why they would coordinate such a massive raid on Megaupload. The target may not have even been the data - merely seizing the data puts anybody who has accessed the megaupload website as an easy target.

mariusz79• 13 years ago

Having a 3 1/2" floppy sent to someone outside of the country 15 years ago will probably count as well.

skore5• 13 years ago

It's quite easy to lose the protections of a U.S. citizen indeed!

That, coupled with the fact that they only require 51% certainty in the foreignness factor makes me think this is intentionally designed to make every single person they come across a subject to surveillance.

I can see Weasel terms like "use of storage media seized outside of the U.s." be extended to mean pretty much anything.

kudu4• 13 years ago

The fact that they don't require a wiretap order or even a warrant to monitor foreign citizens is disturbing in itself and is based on a questionable internal interpretation of the law.

mrt0mat06• 13 years ago

On cspan this morning, the phrased it as: a secret interpretation. the fact that those two words can even sit together hurts my head. How do you secretly interpret something one way, but openly interpret another way.

junto8• 13 years ago

What is really interesting here, is that this disproves what has been said. EVERYBODY'S DATA IS COLLECTED, but to query the data of a US citizen you need to simply provide a 'mitigation reason' as to why you accessing that data.

That then provides an audit trail, where something, or more likely, nothing is done to check that decision was valid,.

randuser• 13 years ago

My favorite part is that the form assumes you need only a single line for the "Justification" field.

martindale• 13 years ago

Interesting; it appears someone failed to redact some data from the slides. In the Facebook chat example, the message is "to" 1536051595.

Using the Facebook Graph API, we can gather information based on this ID: http://graph.facebook.com/1536051595

Which leads us to the Facebook profile (https://www.facebook.com/arash.gorjipour.5) of an individual, real or contrived, named "Arash Gorjipour". His email address and phone number are all exposed in one of his uploaded photos: http://i.imgur.com/0UUk5cB.jpg

I wonder what the reason for this man being in these slides is.

junto8• 13 years ago

Probably picked at random. The screenshot 'test case' could have been you. We are all in the database after all.

junto8• 13 years ago

Of course, it would have been more amusing had the 'selector' been chosen as 'Barack Hussein Obama II'.

iandanforth• 13 years ago

I think it's a test account. The text string which reads something like 'does it still recognize me?' is very much like the kind of thing I'd type in my QA days when I was testing a new system.

If I were putting together a deck on that system I'd also probably favor test data over live data, if for no other reason than it's easy to come by.

eksith2• 13 years ago

They may have used an existing public profile since he's already displayed it openly. He's a realestate broker after all so, presumably, he's got "nothing to hide".

scottdeagan2• 13 years ago

Anna Chapman was a realestate broker too...

igravious• 13 years ago

In keeping with that line of thought would it not be better to redact the information you are presenting? I don't see why you need to write it out in full.

You could say 153xxxxxxx and "Arxxx Goxxxxxxx" just to be sure and if you need to post links you could use a URL shortener.

scottdeagan2• 13 years ago

Probably because he has dark skin, and his native tongue is probably something other than English.

I'm a little worried now because I visited his page, and this will surely be logged, hence my past online activities may now be investigated.

clicks4• 13 years ago

Don't worry, you were already on their list.

HN fields roughly 200,000 unique visitors each day, most of which have a markedly anti-gov't-spying slant[1], that's enough evidence to be in their cross-hairs.

[1]: Such that in some capacity you might participate in the creation/promotion of methods or software to get around their snooping technologies.

stordoff3• 13 years ago

Did the Guardian change the slide? I couldn't see that slide, and it has now appeared with a value of 1234567890.

jberryman• 13 years ago

The relevant slide is inline in the article, under the first appearance of the string "facebook". It was apparently redacted by the gaurdian; see nwh's link to the archived page below.

exit• 13 years ago

do you see that in this document? i can't find it anywhere and wonder if the guardian subsequently redacted it:

http://www.theguardian.com/world/interactive/2013/jul/31/nsa...

nwh5• 13 years ago

There's a small version of the original slide here, which has since been removed — http://archive.is/xcwg6

lisper• 13 years ago

He's Canadian. And he has dark skin and a funny-sounding name.

He's (almost certainly) a real person, by the way. I called his office. He wasn't in, but they offered to page him for me.

pooriaazimi• 13 years ago

Just FYI (almost certainly of no importance because this individual was chosen at random for the slides): his name (both first and surname) are Persian. I'd guess he was an Iranian (graduate) student who has decided to stay in Canada after his studies; possibly to be "free" from an oppressive government's espionage and meddling in his private life. The irony...

api2• 13 years ago

At what point do the mathematical limits of data mining kick in here? How useful is all this information?

I'm not an expert in this area of mathematics, so I could be wrong, but my impression is that as the haystack becomes larger the problem of false positives becomes more and more severe.

As a data miner, what you want is the maximum number of "hits" (of whatever you're trying to hit) with the minimum number of misses and the minimum number of false positives. My impression is that this becomes progressively harder-- the golden region between too many false positives and too many false negatives becomes smaller and smaller and harder to hit.

Eventually you either miss important hits, namely the next terrorist attack, or you get swamped with false positives that you have to manually investigate and rule out.

I'd love someone who does know more here to chip in, but my personal suspicion is that this actually has a pretty huge pork angle to it. How much money are the contractors getting for building this stuff?

runn1ng4• 13 years ago

From the article

The XKeyscore system is continuously collecting so much internet data that it can be stored only for short periods of time. Content remains on the system for only three to five days, while metadata is stored for 30 days. One document explains: "At some sites, the amount of data we receive per day (20+ terabytes) can only be stored for as little as 24 hours."

bane• 13 years ago

so let's say there's a law that says "any American company doing business with a company that does business with a known terrorist organization will have a bad day"

you don't need to use some kind of fancy data mining algorithm for this to work (generating false positives), you just need a ho-hum graph traversal algorithm and unbelievable amounts of graph data to generate "candidates for investigation".

US Company A -> intermediate 1 -> known terrorist group B

US Company A -> intermediate 2 -> known terrorist group B

US Company A -> intermediate 3 -> known terrorist group B

Each set of links is just one lead to investigate, but having a giant graph to work off of would make generating those leads simply. You might find out that intermediate 1 is a local falafel delivery place that "US Company A" uses for lunch catering. Can probably strike that one off the list. intermediate 2 is a utility (no choice but to use the local water monopoly), but intermediate 3 is a material supplier that employs several low level delivery guys from known terrorist group B, and the founder of the company is a cousin of the founder of known terrorist group B.

So I'd wager it's not as simple as just running an algorithm and automatically sending out Skynet drones to blow things up. There's some kind of more subtle assessment being made, with the systems just providing help to the analysts.

hannibal52• 13 years ago

Just like suspected. If you use encryption like PGP, you become person of interest.

northwest10• 13 years ago

Re "Person of Interest":[0]

If yesterday we were "conspiracy theorists" when we suspected things like XKeyscore, what are we today if we suspect things like "Person of Interest"-like programs?

[0] http://www.imdb.com/title/tt1839578/

TeMPOraL• 13 years ago

"You are being watched. The government has a secret system: a machine that spies on you every hour of every day. I know, because I built it. I designed the machine to detect acts of terror, but it sees everything. Violent crimes involving ordinary people; people like you. Crimes the government considered 'irrelevant'. They wouldn't act, so I decided I would. But I needed a partner, someone with the skills to intervene. Hunted by the authorities, we work in secret. You'll never find us, but victim or perpetrator, if your number's up... we'll find you".

(from season 1 opening)

It's scary how this is actually not fiction.

tjr• 13 years ago

Not surprised, but where is this statement in the article / screenshots? I am not seeing it.

LoganCale8• 13 years ago

It's in the full presentation linked at the top of the article. Under the heading "Finding Targets", "someone who is using encryption" is one of the listed means of identifying someone for targeting (along with "someone whose language is out of place for the region they are in"…).

MichaelGG4• 13 years ago

You've no information to back up that statement.

Using PGP as part of a filter makes perfect sense. If you're looking for "bad guys" that do certain activities, as a starting filter, it doesn't hurt to say "OK, show me everyone in this region doing these activities. Now filter by language, etc. etc.".

Just like if I was looking for gang members, I might start off a filter with "look for tattoos". It doesn't mean I'm saying everyone with a tattoo is gang member, it's just a way to start filtering.

The NSA analysts are presumably actually trying to get something done (find people they think are bad). How stupid do you think they are? If you were an NSA analyst, would you tag "person of interest" on everyone using PGP? How would that help your goal of finding actual people of interest?

They say they caught 300 "terrorists" with this program and other success stories. Presumably, they didn't achieve any success by wasting lots of time flagging random PGP users.

hannibal52• 13 years ago

Read the whole presentation linked in the article.

jstalin9• 13 years ago

Snowden deserves the Nobel Peace Prize and the Vatican should consider canonizing him.

jbigelow76• 13 years ago

To be canonized you have to have performed a miracle, but If he somehow get's pardoned by the DoJ or the Obama administration we could probably consider that requirement met.

skylan_q• 13 years ago

You've performed a miracle, yourself. A good joke in the comments section of an HN post! :p

jaekwon4• 13 years ago

At this rate if the surveillance programs get shut down and most of the Patriot act etc get rolled back, it would be a miracle.

nikcub4• 13 years ago

This is overwhelming. Even when you always hear the claims about we knew this was going on, somehow it is still shocking when you see it all laid out infront of you with screenshots and the capabilities described.

I can see how they get HTTP information, since they would intercept at transit hubs - but how are they getting all Facebook private messages and Gmail?

I was also looking for another unique ID that users are identified by - perhaps a machine or browser fingerprint or some form of intel that can 'glue' different browsers together and make a best guess if they are the same person (Facebook does this with device and user cookies) but couldn't find anything. It seems they rely solely on email addresses, IP addresses, cookies and HTTP headers.

So if you are browsing via 16 tor circuits and a browser that defaults to incognito with session histories being wiped, they couldn't reconstruct your history.

Users of PGP/encryption products being singled out is terrifying. The sooner we have the whole world using decent encryption tools, the better.

Edit: Gmail messages must only be captured when they leave the Google network. They are the only provider to support server-to-server TLS: https://twitter.com/ashk4n/status/346807239002169344/photo/1

They must only be getting a slice of the Facebook chat data, since the transport there is also https.

Facebook Messenger, on the other hand, uses MQTT, so it transmits and stores in plaintext. It has support for encrypted + signed messages with OTR if you are using an alternate client such as Adium or Pidgin.

Really need to go out an audit all of these services and let users know which are better.

nwh5• 13 years ago

Assume that Tor is broken. With this level of deep network monitoring, low-latency onion routing is essentially useless.

peterwwillis3• 13 years ago

Hidden services are still secure, presumably, because there is no exposed section of the network to inspect. All they can do is monitor and do statistical analysis, and maybe mess with the traffic to try to get more ideas of flow.

jaekwon4• 13 years ago

Thanks for pointing this out. Good to see a peer preacher.

brown9-23• 13 years ago

How long has https been an option with Facebook and messages? I don't think it was always required, if ever.

nwh5• 13 years ago

It's only been required in the last year or so, prior to that it was an optional extra in the security section.

kintamanimatt9• 13 years ago

Connections secured with TLS aren't effective if a) you can compromise the CA, b) have the private keys, c) have cooperation of the appropriate company (most likely), d) have compromised the server, e) are aware of flaws in the encryption algorithm, f) weak keys have been used, or g) have compromised the client computer.

znowi3• 13 years ago

how are they getting all Facebook private messages and Gmail?

Perhaps you missed the news about PRISM? :)

jgrahamc2• 13 years ago

This presentation is from 2008. According to the presentation on PRISM Facebook joined the program on 3 June 2009. That would indicate that the searches here are based, most likely, not on participation by Facebook but by passive sniffing of HTTP traffic and then session reconstruction.

minwcnt5• 13 years ago

PRISM allows them to retrieve individual users' messages via a FISA court order. It doesn't allow analysts to instantly obtain private data for any user they want. :)

jervisfm• 13 years ago

With regards to the data collection, the thing to realize (which I did so myself) is that email truly is the glue that ties together most internet services.

Take facebook for example. By default, almost any and all activity on the site is catalogued for you by email -- for your convenience. Someone mentioned you in an update, you get a notification. A friend sent you a private FB message, you can an email notification with the content in line (even with the support of replying to message via email as well).

Now, because email traffic on the internet is not encrypted by default, one is able to piece together the contents of communications just by looking at the email.

Essentially anything that you receive via email (e.g. password reset links; credit card statement summaries etc) is subject to capture and analysis. Given this, it may make sense to perhaps disable (potentially sensitive) email notifications as a workaround around this particular collection method.

archgoon• 13 years ago

The main thing that this new release reveals is not the scope of the data collection, but confirmation that analysts are given free reign to perform queries. Until this, there was an outside chance that the system required all database queries to be signed by a Judge prior to execution. This is not the case though; all queries are processed immediately, with essentially nothing more than a repo commit message as justification, and basically any analyst can do it.

nikcub4• 13 years ago

Exactly. There were a lot of people from the government that came out in the past few months and said there are checks and balances and a lot of oversight in these processes. That clearly isn't true.

It will be interesting to go back through all of those statements with this new information/evidence on hand.

Greenwald has timed this well. He put out enough information early on to give Snowden opponents enough rope with which to hang themselves.

coenhyde2• 13 years ago

I think it would be very easy for the NSA to obtain the private keys to SSL of popular sites, eg Facebook, Gmail. Either through coercion or hacking.

pavs• 13 years ago

Facebook ex-CTO joined NSA not too long ago, that might have helped?

joering2• 13 years ago

Probably not as much as FB co-founder offering his company Palantir services to government on people's intelligence.

semenko2• 13 years ago

edit: Gmail messages must only be captured when they leave the Google network.

It seems easier for the NSA to tap datacenter <-> datacenter fiber links inside Google's network.

Why worry about decryption when you can have Google's frontend servers do it for you?

MrKurtz4• 13 years ago

Why would Google (or anyone) link to them directly? with fiber no less! this stuff is alarming enough no need for FUD.

This XKS business seems about intercepting non-encrypted traffic as the references to HTTP payload quoted in the article would suggest.

m8urn2• 13 years ago

Or, the NSA could force Google to provide them with private keys to their certs for "national security" and then they can mitm all they want.

thrownaway2424• 13 years ago

Why assume that inter-datacenter links are not encypted?

cowchase• 13 years ago

Gmail messages must only be captured when they leave the Google network. They are the only provider to support server-to-server TLS

We should start lobbying for broader support for server-to-server TLS with perfect forward secrecy. While it alone is not sufficient to prevent the wiretapping of targeted individuals, it still makes fishing expeditions or "Big Data" level surveillance much harder. It would help keeping ordinary users' emails protected on the wire and secure the meta data of PGP emails.

qwertzlcoatl2• 13 years ago

I was also looking for another unique ID that users are identified by

They can use plugins / extensions installed. Fonts installed. If cookies are enabled or not, etc. Check out: https://panopticlick.eff.org/

morpher• 13 years ago

Most of the identifying information used by panopticlick requires using javascript/flash/java to obtain. As such, it isn't available when simply parsing HTTP headers and packets (as much of the data in XKeyScore appears to come from).

(That is, unless you visit panopticlick.eff.org, which then sends all of the processed information over the wire in the clear...)

quaunaut• 13 years ago

Just want to clarify something:

I can see how they get HTTP information, since they would intercept at transit hubs - but how are they getting all Facebook private messages and Gmail?

I don't know how they're getting GMail(and this is probably a slide from when GMail was accessible via HTTP and not HTTPS), but Facebook chat specifically is done over a non-secure XMPP server. The only 'secure' part of that transaction is login, as far as I remember, once you're past that none of it is encrypted.

m8urn2• 13 years ago

With Gmail, all it takes is one request to almost any Google service to leak through a non http connection and they have your Auth cookie. Once they have that, they are you. And yes it is that easy, anyone can pull it off at Starbucks, hotels, even some ISPs.

MrKurtz4• 13 years ago

You're right the slides are pre default HTTPS gmail (2007/8).

But even then gmail is the only webmail service that offers server-to-server encryption, so data can still get intercepted when communicating with someone using yahoo mail or hotmail for example: http://news.cnet.com/8301-13578_3-57590389-38/how-web-mail-p...

deveac10• 13 years ago

This is overwhelming. Even when you always hear the claims about we knew this was going on, somehow it is still shocking when you see it all laid out infront of you with screenshots and the capabilities described.

It has become a bit of a pet peeve of mine recently to see self-aggrandizing comments from users around the net about how "we should have known" and "none of this is new."

I'm a practically addicted news junkie (especially tech news) and while I've been aware of a fair amount of what has been exposed in this latest leak, it seems that every day there are revelations new to me, and what is revealed absolutely shocks the conscience. And I'm an outlier. I'm more plugged in to reporting on this subject than 99% of the globe's population, and this subject tangles with the rights and treatment of a large portion of the population of said globe.

The staggering majority had no clue, has no clue, and no, they were never informed. For all intents and purposes, the global media has been asleep or complicit.

It's staggeringly important to keep telling this story at every level specifically because "we" don't know, and still don't.

davej• 13 years ago

For all intensive purposes

FYI, it should be "For all intents and purposes". :)

rmrfrmrf• 13 years ago

First of all: keep the staggering to a minimum.

Second: realizing that "we should have known" and "none of this is new" isn't so much about reading news articles and being "plugged in", but rather having an understanding of how the Internet works. To oversimplify greatly, you're essentially playing a very precise game of telephone between around 10-20 different people, and usually about 1-3 different publicly-owned corporations. To be surprised at the possibility of storing packets is somewhat naive considering how simple it is to do.

bmelton2• 13 years ago

You're both right. If you read the PATRIOT Act, it's easy to look backwards and see that the things we're becoming aware of now are logical extensions of what was being asked for way back then.

It is, however, VERY easy not to have been able to have that foresight, and I think that the insights people were expecting the government to have been constrained by the fact that all the information of value is collected by neutral third parties. Google, Yahoo, Twitter, etc., aren't likely colluders with the government.

Plus, at the time of the PATRIOT Act's passage, there wasn't quite as much information being put on social media, or out to the public in general. Not as much was online, digital, or otherwise easily indexable.

There were those predicting this sort of possibility before the PATRIOT Act's enactment, and since, to be sure, but you shouldn't feel responsible for not having seen the signs yourself, or for having heeded the words of what probably seemed like kooky overreactionaries from back in the day.

The funniest part about this, to me, is that somewhere, very quietly, Richard Stallman is quietly telling us all the he told us so, and he's absolutely right, and always has been. Neverminding that, he's largely seen as a crazy old paranoiac who we should respect for his IT knowledge, while having to forgive the rest of his eccentricities.

Daniel_Newby2• 13 years ago

I'm a practically addicted news junkie ...

More like a news sheep. The mass market news is and has always been 49% fluff and 49% lies.

Comments from people who already knew what the NSA does are not "self aggrandizing". The are other-insulting. You should rightly be ashamed that you walk through life in a news fog of up-to-the-minute minutiae. Read books by retired insiders, talk to current insiders and contractors. That's the only way you will learn anything about anything. To wait for the newsmen to do it for you is to sign your mind over to tampon salesmen.

The NSA story is staggeringly unimportant. Every government, many companies, and rather a lot of organized criminals run intel and counterintel operations. It is just a fact of life, like antibiotics and highway construction. It is inevitable that there must be a national American signals intelligence organization.

What os staggetingly important is why the NSA alone, out of all the spy organizations, is being singled out for a comprehensive media war. The most likely explanation is that the Democratic Party needed something to distract from its pecadillos. The next most likely explanation is that a foreign government is getting themselves some payback. In any event, if you care about this non-news, you are just another mindless pawn.

doktrin• 13 years ago

I think it's just a demonstration of complacency more than any actual knowledge on the subject. I've noticed it's invariably my non technical acquaintances who are the first to pontificate on how this is all somehow boring old hat.

weland• 13 years ago

It has become a bit of a pet peeve of mine recently to see self-aggrandizing comments from users around the net about how "we should have known" and "none of this is new."

I agree that "know" is a bit too glorifying. I propose "suspected".

I don't find this surprising at all. Practically 99.99% of a normal user's Internet activity is centered on Facebook, Google (including Gmail) and a handful of other sites. The amount of data everyone is requiring in order to provide a service also includes pretty much anything you need in order to track someone.

It's not news you need to pay attention to but some of the more theoretical aspects of networking in a second-year course.

aasarava• 13 years ago

I share your pet peeve and I can only assume that the "meh, no big surprise here" response stems from two things: wanting to sound just as knowledgeable as the person who brings up the topic (despite not having any new information); and at the same time justifying their complacency about the issue.

graycat2• 13 years ago

From the slides, apparently a node in the system just connects at an ISP or peering site and grabs all the packets. Then they essentially 'parse' the packets to TCP/IP sessions, logical user sessions, e-mail messages, etc.

Then back at HQ, can send the node what are essentially 'filters' to return 'alerts' and the associated content.

So, point: As a system, it's quite obvious. As software, it's quite routine.

And, from their description of working with anomalies, they are being just intuitive and elementary and not at all advanced or powerful.

It would appear that a terrorist Internet user could do fairly well beating that system by using a proxy server also used by many other Internet users and also using a lot of strong encryption -- PGP used well might be strong enough.

commandar• 13 years ago

For me, personally, it's not about "look how smart I am" as it is genuine surprise that the story actually seems to be sticking this time.

I'm glad that people are paying attention, but especially early on, it wasn't entirely clear that Snowden's leaks were substantially different from the leaks that have been coming out of the NSA for years that never got traction in the media.

griffordson8• 13 years ago

The traditional media is complicit. And it isn't some grand conspiracy either, they just share the same interests as the rest of the establishment, being part of (and/or owned by) the establishment themselves.

There is good independent media that has been covering the story for years though. Here's a Democracy Now story from February 2005:

http://www.democracynow.org/2005/2/10/no_place_to_hide_award...

Democracy Now has an incredible archive on this subject too. Right now it starts here:

http://www.democracynow.org/topics/nsa/7

Specifically, they've done some great interviews with previous whistleblowers:

http://www.democracynow.org/appearances/william_binney http://www.democracynow.org/appearances/russell_tice http://www.democracynow.org/appearances/thomas_drake http://www.democracynow.org/appearances/jesselyn_radack

Other interesting guests:

http://www.democracynow.org/appearances/jacob_appelbaum http://www.democracynow.org/appearances/laura_poitras http://www.democracynow.org/appearances/james_bamford

freshhawk2• 13 years ago

Is it self-aggrandizing? I suppose I'm one of those people.

I was shocked by having this laid out as well but I really did just assume this was probably going on. It was technically possible, it was politically possible and it was financially possible. If I shared the worldview of the people doing this and been in the position to do this, I would have been itching to start this level of collection and data mining.

I will admit to part of it being satisfaction at no longer getting the "oh put your tinfoil hat away, no one would do that" response whenever it came up, which was always based solely on the old "I don't like the implications of this being true therefore it can't be" argument. It's also relief that there is finally a discussion about a subject that was previously only seriously discussed by a small number of people.

I take your point that the I-told-you-so gloating isn't helpful and doesn't reflect well on those who do it but I disagree that that was ever meant to discourage discussion, if anything it was anger at the fact this discussion has taken so long to occur.

sixothree3• 13 years ago

I'm betting tonight on the news, this story will be eerily missing.

qq66• 13 years ago

Room 641A?

phaus2• 13 years ago

I've been hearing about the NSA's massive data center in Utah for well over a year, from public news sources. They have always suspected that it's main purpose was the warehousing of American's private communications.

hackinthebochs2• 13 years ago

I'm one of those "none of this is new" types. The fact is, we ALL very much should have known. Do the words "Echelon" and "Total Information Awareness" ring any bells? These were terms being used pre-9/11. There is no excuse for someone technological and with a small inkling of understanding of human nature to not have seen all of this coming. There really isn't.

If you're waiting for someone like Snowden to come along and spoon-feed you all the ways the government can screw you, you're doing things completely wrong. Oversight requires foresight.

Joeri• 13 years ago

Steve gibson has been covering how they can get at this stuff and what alternatives you have on the security now podcast.

https://www.grc.com/securitynow.htm

Briefly summarized, the only way to do secure mail is pgp, the only way to do secure chat is to avoid all the main chat networks. And microsoft actively designs their systems to be easier to access for the NSA (far beyond their legal obligation) so you may assume that any microsoft product is a direct line to the NSA.

pasbesoin• 13 years ago

Keep in mind also timeframes. Facebook HTTPS use -- and more so use by default -- is more recent. Remember the whole "sheep" debacle?

Even Gmail HTTPS use is somewhat recent and not original to the product.

Further, one might combine this with reporting about initiatives to gain company SSL/TLS private keys, account passwords, and the like, in some interesting speculation -- if speculation it remains.

Amongst all the rest, I would point readers towards browser fingerprinting. It's difficult for me to imagine they are not using it.

If the public is going to have some degree of counter-measures, this will include browser and other client software becoming more pro-active about anonymizing its own profile / usage profile. For one thing, stop sending highly unique fingerprint data such as font listings to every Tom, Dick, and Harry. Just one thing amongst many...

northwest10• 13 years ago

but how are they getting all Facebook private messages and Gmail?

It was reported earlier that the NSA has installed hardware at their "partner" companies. As you certainly remember from the slides, they are: Facebook, Google, Microsoft/Skype, AOL, Paltalk, ...

jasonkolb• 13 years ago

What's really sickening is that you can tell that programmers or very technical people were involved at some level to design these systems which help people construct rubber-stamp plausible deniability. Whoever these people were knew full well that they were architecting systems that skirt the letter of the law if not outright flaunt it.

Somewhere there is an architectural diagram of these systems that describes how to make people check checkboxes before releasing information. CYA-oriented programming that has clearly driven the entire design of this thing.

raldi• 13 years ago

Users of PGP/encryption products being singled out is terrifying.

I didn't see that in the article. Do you have a citation?

osbertlancaster• 13 years ago

It is in the slides the article links to: http://www.theguardian.com/world/interactive/2013/jul/31/nsa...

Slide 16: "Show me all PGP usage in Iran"

smegel• 13 years ago

Your faith in Facebook and Google not giving the NSA open access...is a bit strange.

w_t_payne17• 13 years ago

Once again, whilst the shrill cries of protest claim that the government has gone too far in it's intrusive surveillance, the pragmatic amongst us are forced to admit that this is a capability that the state simply will not give up, even in the face of massive public protest and discontent.

Moreover, the technological trend is clear; and the avenues for sharing intimate personal information proliferate and multiply with every passing month. The debate therefore needs to shift. The question cannot be over whether the state should have access to this information. We are powerless to push on that point.

The question has to be this: Given that our state (and others) will necessarily know the most intimate details of our lives, how do we want it to behave? How do we want this information to be used? What do we want the newly intimate relationship between individual and state to look and feel like? It may well be that we come to a startling different conclusion than our initial starting points might presuppose.

There are tremendous social benefits to be had by using this treasure-trove of information wisely, just as there are tremendous dangers to be risked by using this trove with carelessness or malicious intent. However, we need to think very carefully about how we manage the relationship between individual and state; how we manage the relationship between individual and peer; and how we manage the relationship between individual and technology.

I feel strongly that this is the most important debate of our generation; perhaps the most important debate to be had in this new millennium.

jstalin9• 13 years ago

Slide 6 of the presentation clearly shows that pretty much every government is in on the program, with heavy concentration in western Europe.

One question, how did the dot in China get there?

http://www.theguardian.com/world/interactive/2013/jul/31/nsa...

jstalin9• 13 years ago

And look at slide 17:

"Show me all VPN startups in country X, and give me data so I can decrypt and discover users."

Holy crap. Is all encryption broken?

fabian2k• 13 years ago

I find the "Show me all exploitable machines in country X" on page 24 pretty scary as well

MichaelGG4• 13 years ago

There's no evidence that any encryption is broken (other than people misusing it (edit: or broken protocols like PPTP)). Anyone could do this kinda thing given enough motivation and money. Determining a VPN's users? Just monitor all inbound connections to the VPN service. Now you have the IPs of the users. The IP alone might be enough to know the user or a search on that IP might show them logging into other services that reveal their ID. Pretty simple.

vabmit6• 13 years ago

I think so. Yes. IIRC, the first time I saw this leaked was a combo leak by a Navy Seal and a member of the Executive. The Seal leaked that they powered down Bin Laden's computers to take his hard drives after they shot him. The Executive member said that the drives were encrypted and it would take a few days to get the data. Jihadis are known to use a custom version of PGP with 2048bit RSA keys. They either used that, a COTS drive encryption program (unlikely), or reviewed and adapted an open source drive encryption program. In either of the likely scenarios they would have been using 2048bit RSA. Therefor, it is highly likely (due to the NSA having target motivation even if the drives weren't well encrypted) that the smooth barrier does not exist and the NSA can factor 2048bit RSA in a hours to days scale time frame.

Also, it was leaked that NSA TAO had a 70%+ success rate compromising Chinese systems. Even with the tech companies giving them secret zero days for an extended period of time, anyone that has been a blackhat knows they're not getting to a 70% success rate through exploits. Therefor, it's highly likely they can decrypt VPN/SSH (TLS) traffic encrypted with AES256/RC4-128/3DES and/or the RSA/EC public cryptography used. As you noted the leaked slide seems to indicate that.

jstalin9• 13 years ago

I just noticed that the presentation is from 2008. Imagine what capabilities they've added in the last five years.

haakon2• 13 years ago

And what about all the dots along Antarctica?

jstalin9• 13 years ago

The only thing I can think of is satellites?

lifeformed• 13 years ago

Not to mention all those Antarcticans!

aschreyer• 13 years ago

Could be an embassy.

runn1ng4• 13 years ago

From the slides http://www.theguardian.com/world/interactive/2013/jul/31/nsa...

"Show me all the VPN startups in country X, and give me the data so I can decrypt and discover the users"

Does this mean using VPN is not very safe from dragnet?

znowi3• 13 years ago

If Google, Facebook, et al provide direct access to users' data, I'm fairly certain such critical infrastructure as VPN is also under NSA control.

patrickk• 13 years ago

I thought Google, Facebook etc. only had to provide access as they are compelled to under US law? If the VPN provider was non-US (and did not/claimed they did not keep logs of user activity), would this help? Or do you reckon the NSA has the ability to get at the data without the cooperation of the VPN provider? It would be great to have a couple of slides revealed on this area.

kintamanimatt9• 13 years ago

You probably don't need to break the encryption because eventually all traffic has to exit the VPN's company's endpoint, and at that point it can be captured. Meta data such as the browser's fingerprint can be used to tie traffic to an individual, for example, if you see them log in to a regular HTTP site with an email or a username, this information could probably be used to figure out who they are. Armed with this information, all other traffic originating from that endpoint (or elsewhere) with the same browser fingerprint.[-1] can be monitored. Weak keys can also cause the encrypted tunnel to be compromised. Also, PPTP is considered a very insecure tunneling protocol[0] but still used.

You could also break into the VPN company's servers and do interesting things too. There's also the possibility of timing attacks to determine the real IP address of the VPN user, although that's fairly a sophisticated method and quite difficult to do.

Bear in mind that this presentation dates back to 2008, which is a long time in tech years. Who knows what they're capable of now. All that's known is that they're not capable of less.

VPNs are useful for three things: protecting yourself against relatively unsophisticated bad guys sniffing traffic on a local network (for example, an unsecured wireless network), bypassing geographic content restrictions (e.g. using Pandora in Sweden), and circumventing ISP traffic shaping (often they'll not shape VPN traffic because it's used for businesses, and businesses can be whale customers).

[-1] http://en.wikipedia.org/wiki/Device_fingerprint

[0] http://blog.calyptix.com/2012/08/pptp-is-so-insecure-it-shou...

vidarh4• 13 years ago

all traffic has to exit the VPN's company's endpoint, and at that point it can be captured.

If the only thing they're dealing with is VPN's used as a private proxy for access to the public internet, you're right, and if so it's not so troubling (well, as in it is "only" just as troubling as having them access everyones web traffic).

But arguably most VPN traffic is exiting inside private networks and are intended for machines within those private networks. If they are capable of breaking or circumventing the crypto of those, then that's troubling at a whole different level because it potentially means massive unknown weaknesses in either specific crypto products, or in algorithms that have been assumed to still be reasonably safe.

northwest10• 13 years ago

You often hear "What, I should go darknet? Are you crazy?!".

But that's actually the one thing that makes the most sense now:

https://en.wikipedia.org/wiki/Darknet_%28file_sharing%29

nikcub4• 13 years ago

There are not only correlation attacks as kintamanimatt describes, but "VPN" is a broad term that encapsulates a lot of old and broken protocols. They could be talking about breaking CHAP + PPTP.

znowi3• 13 years ago

Holy shit... Apparently, the only way to ensure privacy is to go Stallman. Funny how yesterday's "conspiracy crackpot" became today's visionary.

jacquesm5• 13 years ago

Stallman never was a conspiracy crackpot, he always was a visionary. The only thing that changed is some people's judgment of him.

griffordson8• 13 years ago

Exactly. It has been obvious and widely understood for years to anyone who has ever used a network analyzer that systems like this could be built. The question was always would they be built. Stallman, and others, bet correctly based on their better reading of history and human nature.

astrodust12• 13 years ago

You can be completely correct and still be a crackpot.

What we need is strict limitations on what can and should be collected, and how it's used, plus better methods of securing what's being exchanged. For example, sending email as plain-text, leaving it on the server as plain-text, maybe that's a bad idea.

The NSA isn't necessarily the only reason you'd do this. Foreign governments are going to take an interest in this, too, and it's only a matter of time before someone gets access the data the NSA is hoarding. No program of this scale is ever 100% secure.

peterwwillis3• 13 years ago

How does browsing the web via e-mails and cron jobs make for more privacy?

runn1ng4• 13 years ago

No javascript tracking. Very strange behaviour (therefore, less behaviour tracking). That's what I can think of.

jaekwon4• 13 years ago

He only runs code that he can vet. You can't do that when you use Javascript, or use a proprietary OS, for example.

kristofferR• 13 years ago

This is by far the best guide I've found to do exactly that:

http://crunchbang.org/forums/viewtopic.php?id=24722

kudu4• 13 years ago

I haven't read all of it, but I'd recommend using LUKS instead of TrueCrypt.

shirro2• 13 years ago

You have to admit these guys are working on some cool problems. If you don't have a problem with the legality of it or potential for misuse it looks like a really interesting place to work.

stef253• 13 years ago

Yeah and it looks like intelligence & skills doesn't always equate with having morals.

jacquesm5• 13 years ago

That's exactly how they get people to work on it in the first place. If you have no conscience there are lots of places where you can work on 'cool problems'.

astrodust12• 13 years ago

Weaponizing a nuclear reaction? Cool, right?

aroman2• 13 years ago

I'm going to give you the benefit of the doubt and assume you're not condoning their actions, but in case you need a more clear example of where this paradigm breaks down:

IBM & Nazi Germany, Nazi Germany in general, etc. Ironically, sillicon valley came into existence building military SIGINT/ELINT systems for the cold war.[0]

[0] www.youtube.com/watch?v=hFSPHfZQpIQ

shirro2• 13 years ago

I think there is a developing consensus that the emerging US police state (and that of its allies) aren't appropriate for constitutional democracies. Comparing them to Nazi experiments or nuclear weapons is still a bit premature. The potential is there for it to turn really ugly which is why the time for political action is now. But national security is important and worthwhile within constitutional limits and under democratic oversight and I suspect a scaled back NSA would be no more evil or less interesting than Bletchley Park or the development of Radar.

alan_cx• 13 years ago

As were certain medical facilities in WW2.

impendia• 13 years ago

Another data point on the relationship between government and terrorism:

I live in Columbia, South Carolina. A mile from my house there is a prominent statue of Ben Tillman. Tillman was an explicit advocate of terrorism, and indeed personally engaged in it [1], which drove his popularity and ensured his election to the governorship and the United States Senate.

Government programs such as the NSA's exist to protect the interests of the powerful. Same as it ever was.

http://en.wikipedia.org/wiki/Benjamin_Tillman

jneal2• 13 years ago

Noticed one of the screenshots have a URL. It's a little blurry, but I suppose it's an intranet URL since the TLD looks like .nsa

URL looks like: https://gamut-wakefield.ein.nsa/utt/UTT/do/FRNewSelector#sel...

jneal2• 13 years ago

Interesting addition. I wondered what some of these abbreviations in the URL stood for. Upon searching, I found this:

http://www.techcareers.com/job.asp?id=64332188&aff=C014D02C-...

Job posting, requiring top-secret clearance, looking for people that have experience using certain tools including "GAMUT/UTT" - notice the URL from the NSA doc has "gamut" and "UTT". So i further looked into GAMUT/UTT and found this:

http://williamaarkin.wordpress.com/2012/03/13/nsa-code-names...

giulianob• 13 years ago

Holy shit.. they did really write a GUI in Visual Basic to track the killer's IP

griffordson8• 13 years ago

Thanks, there isn't much funny about any of this, but that got a legitimate laugh out of me. For anyone who doesn't know the reference, a clip from CSI:

http://www.youtube.com/watch?v=hkDD03yeLnU

mattbarrie5• 13 years ago

I think the guardian got trolled hard. This is too ridiculous to be true.

falk3• 13 years ago

I think you're in denial.

peterwwillis3• 13 years ago

Here's an interesting legal question:

If a non-US resident or NSA target posts a thread on HN, and a US person replies to the thread, is the US person now open to unlimited data collection?

Alternately, if you Facebook-like the same thing an NSA target has, are you then subject to unlimited data collection?

gcv• 13 years ago

I don't know about forums, but I suspect that being Facebook friends with a non-US citizen is enough to subject a US citizen to unlimited collection.

fchollet• 13 years ago

The information we have already shows that the US person/non-US person distinction is purely cosmetic, meant to allow them to pretend that they're at least trying to respect US laws to some tiny extent. It's rhetorical.

In reality you are always a valid target, US citizen or not.

emhart• 13 years ago

Fitting/sinister that the top of their stored data pyramid is titled "TrafficThief"

forgotAgain• 13 years ago

Aw, Fuck.

coenhyde2• 13 years ago

This is the most terrifying thing i've ever seen. I'm not exaggerating in the slightest.

arca_vorago4• 13 years ago

Lurkers first post here.

I always said saying "I told you so." when stuff like this started getting revealed would feel like a hollow phrase. Some of us have spent quite a bit of time talking about these issues, and were mostly rejected as crackpot "conspiracy theorists". While there are plenty of those around, maybe I could use this slight moment of pseudo-clarity to propose something.

I could tell you where this is going (removal of ex post facto, and eventually algorithmic based pre-crime), and who is largely behind it, but once again most of you would probably perform the standard knee-jerk reaction against "conspiracy theory", only to wait around and repeat the same kind of stuff you are saying now, whenever the next steps are put into action.

We curious geeks have been too cocky, always thinking we could use our superior knowledge of technology to beat "the man". Well boys, the man is learning our tricks, and he's starting to get better at them than us...

The NSA is but a cog in a greater machine, and until we all realize that and start conversing on what/who that machine is, we will continue to spin our wheels uselessly.

goombastic• 13 years ago

Happened also because some of the geeks rented themselves out to the NSA.

arca_vorago4• 13 years ago

Just the MICC in general. I remember talking to Mudge on the phone about two years ago, asking him how Darpa was treating him. He said he loved it... guess where he is now? Google.

krelian• 13 years ago

The problem with conspiracy theorists is that they believe their theories to be 99% certain instead of the most probable 1%. "I told you so" goes both ways except that conspiracy theorists never accept and admit it, there will always be another conspiracy to cover the current one.

In the overall game it simply makes much more mathematical sense to treat conspiracy theorists as crackpots since they will so often get it wrong. Even a broken clock is correct twice a day.

arca_vorago4• 13 years ago

Negative. The problem with conspiracy theorists, or any group for that matter, is that they are so easily put into that group and then dismissed as a whole. Personally I think at the core of the matter is an enormous imbalance of importance placed on the authority or source of an argument, to the detriment of argumentative logos. Possibly some form of the genetic fallacy.

For example, you deride an entire "group" of people for believing their postulations to be 99% probable and assert that instead they are more likely 1% probable. I would say you just fucking pulled those statistics out of your ass, the very act for which you are deriding the group for! Such irony in a single sentence has been seldom found.

Sure, I agree with you that there are a plethora of quite frankly ridiculous conspiracy theories (not to be confused with the theorists themselves, who can often be just as ridiculous). That being said though, you are in fact completely wrong when you say "In the overall game it simply makes much more mathematical sense to treat conspiracy theorists as crackpots since they will so often get it wrong." and here is why.

Conspiracies are a driving factor throughout history. To ignore them in the past and in the present is to dismiss the very core their historicity. To dismiss them based on some flawed, arbitrary assertions about mathematical calculations that have no academic backing is simply daft.

Also, as I said before, there are indeed plenty of logically absurd theories... but why exactly are they absurd? One, they defy logic, but more importantly, the more absurd ones are often based on no evidence whatsoever. If you have such a difficult time sifting through the completely absurd to find the logically probable/possible in order to more finely tune your engagement, that is a deficiency on your part, and no one else.

I'm trying to think of an analogy to this... It's very much like the general reaction to homeless people. Many people have a general reaction to homeless people in which they assume they are either scammers, drug addicts, or lazy. They may have had a bad experience with a homeless person, and now seem to think almost all homeless people are not worth the time/effort. I understand how they can come to that conclusion, but the statistical facts do not support their argument, and, similarly, the statistical facts do you not support yours either, because you have essentially taken the same approach to "conspiracy theorists".

w_t_payne17• 13 years ago

OK, I will take the bait. What do you think the greater machine is?

Here is my take:

You do not need to posit an organized Illuminati-like conspiracy to have cause for concern. We can find plenty to worry about even if we limit ourselves to properties of the system that are either emergent or driven by natural human behavioural traits.

For example, a lot of people in positions of authority got there because they have authoritarian instincts, and seek self-validation not only by dominating and controlling others, but by ensuring that their position of authority and dominance is recognized by others.

This is very human, and very instinctive, and operates at an unconscious, almost sexual level. The alpha male will seek to dominate the pack and to remind competing males of his superior status. You do not have to consciously be aware that you are seeking power, money and sex, but you are, nonetheless.

This instinct can operate both consciously and unconsciously. Those who make decisions to concentrate power and authority, to separate and elevate themselves from the general population - they do not have to be consciously aware of what they are doing. They can and will rationalize their beliefs and actions to make it fit in with the dominant culture of their peers. This process is called confabulation (http://en.wikipedia.org/wiki/Confabulation) and everybody does it all the time - it is the only way that we can make sense of our lives and live in a human body without going insane with the sheer irrationality of it all.

These instincts manifest themselves in lots of small, individually inconsequential decisions. Normally, this is OK, because our social and bureaucratic technologies are (were) too ineffective for too much harm to be done. The ongoing march of modern information technology, however, looks likely to change that, meaning that the unconsciously malicious instincts of humans in positions of authority can become amplified and magnified.

I would be particularly worried if this resulted in a feedback loop - so that increased power and increased power-seeking behaviour mutually reinforce one another in a runaway process. I cannot readily identify such a loop in operation though -- can anybody else?

arca_vorago4• 13 years ago

I hope I am not responding to too many people and seeming spammy, but here goes.

I agree completely that we do not necessarily need to posit and organized "Illuminati-like conspiracy" to have cause for concern. There are plenty of studies showing increasing likely-hood of sociopaths rising to the top of power structures, and is often just due to how to system as an autonomous entity functions.

What I do posit though, is that, in fact, there is, borrowing your own term for lack of a better one, an "illuminati-like conspiracy". I have been considering an attempt at scholarly paper on the matter for some time now, but let me try to be terse and possibly just point you in the right direction, because I don't think I'm quite prepared to defend the full assertion in public yet.

I will start with your question about power feedback loops. Here is a paper regarding the global network of corporate control that anyone interested in the global power structure should read. http://arxiv.org/pdf/1107.5728v2.pdf

I even contacted one of the researchers (Glattfelder) during the Libor scandal, wondering if we could use some of the new information to analyze the scandal better. He said it would be extremely difficult due to how good the companies are at obfuscating their dealings.

Now, as far as the conspiracy, I would like to point out one thing. I do not claim that there is but a single conspiracy (a trap assertion many fall into making), and instead would say there there are but a small number of very powerful ones operating at any one time, sometimes in competition and sometimes cooperatively. Regarding the "illuminati-like conspiracy" itself, I have one primary reading source for you, if you are genuinely interested in the subject. It should be enough to get you started on the more serious analysis of what I am talking about. http://www.amazon.com/Anglo-American-Establishment-Quigley-C...

MarkHarmon• 13 years ago

One reason I love HN is due to the contributors having better than average critical thinking skills (IMO). Unfortunately that isn't enough when it comes to asking someone to believe that all governments might be controlled by a secret organization, or something equally as sinister and unbelievable.

Since we know that our (USA) entire financial system is backed by (Federal Reserve) bankers that loan money without moral guidelines, it could be conceived that a family like the Rothschild are at the top of the world power hierarchy. What do they always say in the detective movies, "follow the money".

ferdo2• 13 years ago

The NSA is but a cog in a greater machine

Indeed. I always follow The Money because that's who owns the government.

America, including the NSA, is owned by the same small cartel that have the monopoly on the issuing of our currency and credit. Coincidentally, they're the same cartel we're "indebted" to.

Most people have never heard of the four largest banks in the world:

http://www.northerntrust.com/documents/white-papers/asset-ma...

chewxy2• 13 years ago

I don't think I have been more conflicted about this. I've just been talking to my cofounders about the technical feasibilities of XKeyScore, and honestly, our back-of-napkin engineering configurations indicate this is really an awesome project to be working on.

On the other hand, this is categorically 'evil' by my and my cofounders' ethical standards, and really, no one is safe. And that bugs the hell out of me.

On the one hand: really fucking cool. On the other, I really do not like the idea that I am being spied on.

I'm not sure how to process this information.

blackaspen• 13 years ago

Welcome to working in the modern advertising industry.

chewxy2• 13 years ago

I AM currently in the advertising industry

gtaylor• 13 years ago

On the one hand: really fucking cool.

I just don't see how this could be considered "cool". There are plenty of other marvels of modern computing that aren't so sinister.

I think a better word for this is "scary", due to the level of cooperation from corporations and the level of secrecy it was running under for so long.

hammerzeit• 13 years ago

Reading these slides, I'm trying to parse what these slides do or do not say. I'd like to leave aside the speculation about what the NSA is probably doing.

First of all, XKeyscore seems to be primarily about the frontend query interface rather than the backend data storage, at least as far as I can tell. It looks like you can basically query their database by email address and get a set of records (email, chat, http logs) back. It looks like there are separate tools for viewing specific records as well. I assume they're joining records on some combination of email address, IP address, timestamp, etc -- not unlike a modern ad server.

A few practical thoughts: * It's worth noting what's not shown in these slides. Specifically, I don't see any ability to query the full text of emails. The more I see about this, the more I'm convinced the NSA is not collecting email body texts directly from corporate servers. Facebook messages I'm less sure of. * How are they collecting HTTP data? I assume intercepting at network hubs? * Given that it appears that individual records are HTTP requests, I'm shocked at how few requests are in the database. 41 billion seems an order of magnitude smaller than I'd expect. Could it be a record is something else? * Interesting to note the "Miranda number" and "Foreign Factor" fields that look like ways of saying "yes, I have permission to do this." Might explain why a sysadmin could bypass these things but your everyday NSA analyst could not.

vidarh4• 13 years ago

Note regarding the amount of items, that the presentation is from 2008, and they claim to only be able to store 3 days worth of full data capture.

Regarding ability to query the full text of emails, this program does not seem to indicate that it would collect the data directly from the services servers in anyway. But consider that they do indicate the ability to monitor web traffic at the protocol level. Capturing e-mail is no harder, so it'd be surprising if they're not.

cm2012• 13 years ago

It doesn't show reading full emails in the screenshots, but the sentence right underneath reads: "The analyst then selects which of those returned emails they want to read by opening them in NSA reading software."

stordoff3• 13 years ago

One of the slides [1] has the full message text for a Facebook message. If they have it for Facebook, I'd be surprised if they also don't have it for email.

[1] https://image.guim.co.uk/sys-images/Guardian/Pix/audio/video...

dictum• 13 years ago

Foreignness factor

I know NSA's mandate is to spy on foreigners, but it's still very jingoistic and xenophobic that not being American makes it OK to spy on you.

kintamanimatt9• 13 years ago

One could assume that Americans are spied on by foreign governments and the data is just exchanged. The US spies on Brits, the UK spies on US persons, and the both compare notes.

lukifer3• 13 years ago

This is exactly why "every country spies on foreigners" is unacceptable. There's no reason to assume that this "note-sharing isn't already taking place.

foobarqux3• 13 years ago

It's less of an issue since many communication and internet services are located in the US.

s_q_b5• 13 years ago

Actually this is exactly what occurs. Intelligence exchange among America and its allies under Echelon, ANZUS, and UKUSA have been used in this exact way to end-run around anti-domestic surveillance laws.

qwertzlcoatl2• 13 years ago

Livestream to senate hearings covering all this can be found here: http://www.judiciary.senate.gov/hearings/hearing.cfm?id=0d93...

As of this moment it's all about FISA. Wonder if this new allegation will be talked about.

BgSpnnrs• 13 years ago

The senate hearings are with regard to phone metadata. This is a totally different show.

mrt0mat06• 13 years ago

they did mention it offhandedly, the woman(i was only listening) started making a list of things that the nsa should release yearly including how many crimes 702 and whatever helped prevent, and the main guy asked her to add a view, in the light of the news posted today, but said they would investigate that further at a later time. sorry it's poorly detailed but just wanted you to know it was mentioned.

naithemilkman• 13 years ago

Surely the question on everyone's mind is: how good is incognito mode???

kintamanimatt9• 13 years ago

It's not. It's only just a way to visit sites without a record being stored on your computer. Read the warning. For example, Firefox's private browsing mode states:

While this computer won't have a record of your browsing history, your internet service provider or employer can still track the pages you visit.

They should include government spooks in that warning!

shawabawa3• 13 years ago

Chrome does!

Going Incognito doesn't affect the behaviour of other people, servers or software. Be wary of:
...
Surveillance by secret agents

clarky07• 13 years ago

Chromes warning does include govt iirc

LoganCale8• 13 years ago

The only thing incognito mode does is keep your activity and cookies while in that mode from being saved into your local browser history. It doesn't do anything at all externally.

rosem• 13 years ago

I think it's insane that so many people are pointing the finger at Snowden, yet no one is pointing the finger back at the NSA / US Government.

venomsnake• 13 years ago

Credit where credit is due - NSA made useful and usable email search. Please give it to gmail and outlook.com ... I want to be able to search trough my mail as good as you guys can do.

kilian• 13 years ago

A 'fun' bit of weasel-wording by the chairman of the House intelligence committee: "He's lying. It's impossible for him to do what he was saying he could do." They seem to be denying it, but all they're really denying is that Snowden had access to the system personally.

wittyphrasehere• 13 years ago

It's impossible for him to do it, he's in Russia!

dlitz• 13 years ago

They mean he didn't have the "capability" to "collect" that data, which in NSA newspeak means he had the technical means, but not the legal authorization to do so.

aspensmonster2• 13 years ago

How do I find a strong-selector for a known target?
How do I find a cell of terrorists that has no known connection to strong-selectors?
Answer: Look for anomalous events
E.g. Someone whose language is out of place for the region they are in
Someone who is using encryption
Someone searching the web for suspicious stuff

Lovely. Suspicious stuff and encryption. But wait! There's more!

Show me all the VPN startups in country X, and give me the data so I can decrypt and discover the users.

Wait... what? I really hope that doesn't mean what it sounds like it means.

Slide 22 [regarding determining who authored a Jihadist document] redacted.

Well that's interesting.

Show me all the exploitable machines in country X.

That's cool. I'm guessing this is what Snowden meant by weak endpoint security.

Over 300 terrorists captured using intelligence generated from X-KEYSCORE
Slides 29 and 30 regarding this redacted.

What a shame.

talmand3• 13 years ago

I'd like to know where they are keeping these "over 300 terrorists" that have been captured due to this program.

How is one labeled a terrorist by this program I wonder?

o0-0o• 13 years ago

By captured, I'm sure they mean captured via droning.

fnordfnordfnord3• 13 years ago

I'd like to know how they check for/track false positives.

bowlofpetunias• 13 years ago

This. This is what worries me more than anything. Where have these 300 people gone? There should be records, trials, something. 300 suspected terrorists going on trial over a period of 5 years should have resulted in huge wave of almost 24/7 publicity.

That the NSA is in the business of total surveillance is bad enough. But there is the faint hint that the NSA is in the business of making people disappear.

katbyte2• 13 years ago

And these days it seems you can be labeled a terrorist just for protesting..

fragsworth• 13 years ago

Well, we know what happens when they're "caught" - they're put in Guantanamo forever.

Pretty fucking scary if you ask me.

weinzierl• 13 years ago

Someone whose language is out of place for the region they are in

Check (German browsing in English)

Someone who is using encryption

Check (my PGP Key is on my website)

Someone searching the web for suspicious stuff

Check (I'm a curious person, what did you expect?)

I guess I'm considered a terrorist nowadays.

northwest10• 13 years ago

> Show me all the exploitable machines in country X.
That's cool. I'm guessing this is what Snowden meant by weak endpoint security.

That, plus things like Microsoft and Apple operating systems. Don't forget: it's proven they work with the NSA, so backdoors certainly are guaranteed (plus, with Microsoft, we also know they hand 0-day exploits over to the NSA before they're fixed, plus you benefit from all the viruses, trojans, etc.). Again, if you missed it, start migrating now: https://prism-break.org/

LoganCale8• 13 years ago

There was a document released by the Guardian a few weeks to a month back showing how they also monitor open source issue trackers to find exploitable flaws.

yen223• 13 years ago

Has anyone verified the accuracy of the slides? How do we know they even came from the NSA?

netrus2• 13 years ago

Let's see if NSA dares to dispute their authentity.

LoganCale8• 13 years ago

They came from Snowden. And the NSA is quoted in the article acknowledging the program by name.

GI• 13 years ago

I am going to be obliterated for this comment! Does the fact they've caught 300 terrorists in anyway justify what they're doing? I am not saying it does, I just wondered what people's thoughts were (although I can guess!). It's interesting that it was included in the article in an attempt to give it some 'balance'..

skore5• 13 years ago

Sir Thomas More decides that he would rather die than lie or betray his faith. And one moment he is arguing with the particularly vicious which hunting prosecutor. A servant of the king and a hungry and ambitious man.

And More says: “You’d break the law to punish the devil, wouldn’t you?”

The prosecutor says: “break it? I’d cut down every law in England if that would take it to catch him”.

“Yes you would, wouldn’t you?” And then “When you would have cornered the devil and the devil would turn around to meet you, where would you run for protection, all the laws of England having been cut down and flattened? Who would protect you then?”

Every time you violate – or propose the violate – the right to free speech of someone else, you in potentia you’re making a rod for your own back. Because (…), to who do you reward the right to decide which speech is harmful, or who is the harmful speaker? Or to determine in advance what are the harmful consequences going to be, that we know enough about in advance to prevent? To whom would you give this job? To whom you’re going to award the task of being the censor?

http://howtoplayalone.wordpress.com/hitchens-on-free-speech/

sbi• 13 years ago

These are quotes from the Bolt drama "A Man for All Seasons," not Sir Thomas More himself.

homeomorphic• 13 years ago

In my opinion, no, but you do raise a question that legitimately is in the realm of politics: how much free reign should the secret agencies have, and how much security (measured in kilochildren?) should we get in return?

A huge problem with the NSA's activities is that they are, or at least have been, so secret that it is impossible to even discuss in a meaningful way the political question you posed. This should be unacceptable even to those who think catching the 300 terrorists made the surveillance ok.

mrchucklepants• 13 years ago

no.

hackinthebochs2• 13 years ago

300 "terrorists" caught for a few billion people's privacy completely destroyed at a price tag of 100 Billion (pulled out of my ass obviously). Absurdity at its finest.

w_t_payne17• 13 years ago

How many of these "terrorists" were just young men suffering typical teenage angst, and happened to be of the wrong religion?

w_t_payne17• 13 years ago

It depends. Not only does it depend on how much of a threat those 300 really were, and how much damage they would have done had this system not been in place, it also depends on how much this system gets abused, and how much the world suffers because the most influential country on the international stage that was actively pushing for greater transparency and openness has just had the moral high ground ripped right out from underneath it's feet, emboldening all the genuinely corrupt and malicious actors in Russia and China.

josephlord• 13 years ago

I don't think so but even if it is on the table "caught" and "terrorist" need to be very carefully defined.

If "caught" means convicted and "terrorist" means bomb maker or airline pilot with intent to crash a plane into skyscrapers it may be a discussion worth having. If "caught" means arrested or "terrorist" means gave $5 to an Islamic Medical Charity that turns out to have been shady it isn't even close to worth it on a practical level without even considering the principles and general human rights aspects.

My guess is somewhere in between but with people with as weak grasp of the language as the NSA seem to have I would be very careful.

nsns• 13 years ago

Pointing to any benefits of this program is the equivalent of an ad hominem attack. Undoubtedly there are many efficient yet illegal ways to counter terrorism. No one claims this program can not be beneficial, but that the cost is too high (abandoning the rule of law, obliterating personal privacy). It's like committing suicide to foil a murder attack.

typon3• 13 years ago

Yea...no. The problem with being a habitual liar is that people can't trust you even if you tell the truth. That 300 number sounds as much bullshit to me as all their other claims.

"We don't gather any data."

"We only gather metadata."

"Ok fine, we gather everything."

mattbarrie5• 13 years ago

Um... surely this has to be a spoof. "Select foreignness factor"?? really?

The user interface and way this is done just seems to amateur hour to believe this is actually true

mattbarrie5• 13 years ago

The numbers seem way off and too keystone cop to be true. 20 terabytes is not large for the NSA.

It can search BCC?? Only the sender has them. so everything would have to be collected at each ISP (which isn't impossible).. but I think the guardian has been trolled.

mattbarrie5• 13 years ago

I'm calling this a troll.

ihsw• 13 years ago

If you're using Gmail then Google sends your email, so Google knows your BCC lines.

el_zorro• 13 years ago

Keep in mind that this was from 2008.

eterm• 13 years ago

That's 20TB per site. Who knows how many collection sites they have?

w_t_payne17• 13 years ago

Umm ... hello? Defence contractors anybody?

In one sense, it is amateur hour ... but if you buy enough hours from enough amateurs ...

mladenkovacevic• 13 years ago

It seems this was meant to be declassified in 2032.. I guess by then they were hoping this would be so institutionalized and pervasive as to be the norm.

Also I wonder to what extent this is really used to hunt terrorists down and how much of it is used to gain political or economic advantages over other countries.

madaxe5• 13 years ago

Secrecy is like copyright - keeps on getting extended, or the documents just get "mislaid" long before the declassification date.

vasilipupkin• 13 years ago

I am curious: Suppose this is true and NSA analysts have the technical capability to access enormous amount of information with no authorization. But if they do do that, agains the law and the rules and their actions are recorded in the system, they could face penalties no? I mean I could kill someone with a hammer technically, that doesn't make hammer bad per se, does it ?

ryanmolden• 13 years ago

It says can be audited. I suspect the number of times that has been done is vanishingly small, perhaps only on Snowden, after his revelations :) It smells to me like plausible deniability. Claim you have set up a system where things can be audited, don't ever answer how frequently it is done or what the penalty is for abuse of the system. Top secret and all. This kind of system really should freak people out. I imagine it would be a great blackmail info system if people start speaking against the govt or "causing problems". I can imagine public figures don't want to be publicly shamed for their online activities, gambling, porn, affairs, etc...

antimatter15• 13 years ago

I wonder how Sencha (http://www.sencha.com/) feels about how the NSA is clearly using their ExtJS framework given the screenshots.

I guess this kind of puts different perspective to the whole debate that came from JSMin's "The Software shall be used for Good, not Evil." clause (http://wonko.com/post/jsmin-isnt-welcome-on-google-code) given that conceivably your open source framework might be a significant part of something like this.

nemothekid• 13 years ago

Its also very likely the machines are being run on Linux boxes. Should Linus being losing sleep knowing he aided the NSA in this? Even the very database this system runs on may be an offshoot of the Google BigTable paper. Should Google have never opens sourced the software that eventually became HBase/Hadoop/Cassandra because of the NSA?

A tool is a tool. I don't think Henry Ford should feel guilty for enabling people to kidnap children with greater speed.

foobarqux3• 13 years ago

No it doesn't. This issue was raised and thought out long before JSMin, for example in the case of the license that forbade use by the South African military. It's a bad idea.

ssafejava• 13 years ago

ExtJS is widely used on government systems and has been for years. If you want to deploy a rich web application that can handle large data tables with infinite scroll, filtering, sorting, etc., and run it all on IE >= 6 ExtJS is your only feasible choice.

aspensmonster2• 13 years ago

Slide 13 just got slightly redacted.

http://imgur.com/a/SVerP#0

madaxe5• 13 years ago

Because it has someone's personal data in it, and the Guardian respects privacy - unlike the NSA, who use it in presentations. That's the only reason they've redacted anything, if you look at where the slides sit in context.

HNaTTY• 13 years ago

The 64. IP address on that slide is to Google (obviously), here's the other end:

GeoIP City Edition, Rev 1: PK, 08, Islamabad, N/A, 33.700001, 73.166702, 0, 0

5.157.65.58.in-addr.arpa domain name pointer mbl-65-157-5.dsl.net.pk.

23674 | 58.65.157.0/24 | MBL-AS | PK | NAYATEL.PK | MICRONET BROADBAND (PVT) LTD.

akmiller• 13 years ago

Honest, maybe naive question, but what types of programmers actively help build and maintain systems like this? I turned down a job for a company that is less than a mile from my house because I viewed their business as immoral.

Hard for me to fathom anyone taking a job, helping to build systems like this. I get that many of the components of a system like this could be seen as harmless. However, a system of this complexity must have some talented engineers bringing it all together and making it work. How can they feel good about what they are doing?

k-mcgrady• 13 years ago

I guess some people are in it for the money. Some probably like the technical challenge. But I'm sure there are also quite a few that believe this system really is preventing terrorism, following the law, and not doing anything wrong.

general_failure3• 13 years ago

There's something more important than money and technical challenge.

It's called Power.

I can now look into any person's email and whatever legally. You and I may get a squishy feeling in the stomach about this but I can tell you that 1 out of 2 people will be OK with this if they aren't called out. (like watching porn, most people won't admit in front of kids and their parents but they will do it because they think they won't get caught)

nicholassmith3• 13 years ago

They're probably not thinking of it as immoral. Most likely they actually do believe they're improving national security by doing this.

Or they just like the paycheck, that's a big possibility. I imagine the NSA is probably happy to pay a lot with a large amount of bonuses to keep people in roles.

clicks4• 13 years ago

This sentiment was also echoed by Snowden in the original video interview. He supposedly had greater insight as to how civil rights at large were being violated, only because as a sysdmin he (and not others) had a broader picture of what was going on, and at what scale it was going on.

So, it's easy to imagine NSA recruiters coaxing potential hires with convincing lines like "we have internal courts, procedures, and checks and balances systems to prevent abuse of citizens' privacy".

g8oz• 13 years ago

In a profile on Palantir (which sells tools that help with this sort of thing), one employee was quoted as saying "this really is about saving the Shire". Had to laugh.

smackfu• 13 years ago

The NSA sells the job to naive college students on patriotism and generic descriptions of working with cutting edge cool stuff. Once you're in, it's easy to rationalize that all this will be used only for good. Especially if the alternative is quitting your stable job that supports your family.

nulldevnull3• 13 years ago

Many of the low-level analysts are very smart enlisted personnel with ranks of (E1 to E5) from among the various services. Many of them use the GI Bill to pay for college. Some stay in the IC, while others move one to different careers.

skore5• 13 years ago

How can they feel good about what they are doing?

You'd be amazed.

https://twitter.com/_nothingtohide

rohit89• 13 years ago

Seems to me that people aren't too concerned of nameless, faceless people looking through their stuff so long as people they know (parents, spouse, friends..) don't see it.

mcphilip• 13 years ago

One of my first jobs was at a company that sold and serviced surveillance equipment to school districts. It's a slippery slope where you can first focus on the good aspects -- murder cases solved -- but eventually encounter information that you have to make an ethical judgement on wether you are willing to play a role in implementing such technology. For instance, testing began on systems that could capture both audio and video and I became uncomfortable because kids say stupid things that can be taken out of context. I make no claims that audio surveillance was installed, but the casual nature of doing R&D on such systems without considering the implications became too uncomfortable for me to take part of.

aravenel• 13 years ago

The opportunity to work on a cutting-edge technical problem at scale most would never get to touch. That's a very tempting proposal. The secrecy itself probably also appeals to some folks.

I'd also imagine that a lot of folks only work on one small part of this, that in and of itself, is not objectionable. Where it becomes scary is when you expand the scope to cover every US citizen regardless of wrongdoing.

motters• 13 years ago

Times are hard and if it's a choice between your personal morals and feeding your kids or ensuring that your home doesn't get repossessed then there are likely to be plenty of programmers willing to make the compromise.

anon268• 13 years ago

I have "friends" (read: we mostly argue about this) who fit your description.

They will tell you that there are rules in place to prevent spying on Americans (and if you take a look at the Foreignness Factor screenshots, there is a sense in which this is true)

They will also tell you that the benefits outweigh the cost. Here we have a system that has allegedly caught 300 terrorists, and they would tell you that spying on foreign people to catch 300 terrorists is a good trade.

I disagree with them on both counts, but you asked what they would say.

bobbydavid2• 13 years ago

if you take as given that the NSA has prevented 300 9/11 attacks from occurring, you may feel it is justified.

surely you don't believe the premise, but as a thought experiment pretend it were true. would the NSA actions be justified?

MattyRad• 13 years ago

I have asked this same question myself. It's stunning that only one sys admin has come forth with this information. Further, politicians and higher business officials in companies usually don't know what computers are capable of, which leads me to believe that it's actually programmers who are suggesting that such horrifying systems be built (for example, no politician is going to say "let's make a copy of every piece of internet traffic everywhere." The programmer says "We can copy everything," and the politician says "Ok, do it"). The only excuse I could imagine that a programmer would tell themselves is "This is just my job, and I'm just building the software, but it's the government that's going to be using it, not me, so I've done nothing wrong."

general_failure3• 13 years ago

You could say the same for many jobs: being in the military ("i am just following orders") making bombs making atom bomb human cloning

bottom line is anything can be justified if you want to justify it.

burntsushi3• 13 years ago

How can they feel good about what they are doing?

Didn't you hear? The system is used to help track down bad guys who want to hurt America.

It sounds trite, but people really and truly believe that the issue is just that simple.

cookiecaper• 13 years ago

The internet is a public space. Unless you're using full client-side cryptography, one should not assume their activities are private, just as one doesn't assume their physical activities are private if they're done in the open in the public square. Using client-side crypto is the equivalent of entering a private residence; you can't just sit out in the open and expect privacy, you have to take special effort to keep your activities private.

e12e• 13 years ago

You might want to have a look at the HOPE9 keynote with one of the first NSA leakers:

http://www.youtube.com/watch?v=FOFtQ6n3WR4

Clearly a very smart guy, that went very far in the NSA -- and for a long time felt he was doing "the right thing" -- but eventually quit because of what the NSA were doing.

edit: He also touches on how compartmentalization leads to people not knowing what they're actually working on/how it will be used in some cases.

antitrust• 13 years ago

I realize that it isn't morally right, but I think such data storage is inevitable. With the rise of instant communications, the amounts of data people generate are massive, and old school law enforcement can't keep up. Thus with the increase in technology, there's going to be an increase in counter-technology.

I guess what we need to ask ourselves now is whether we want any secrets at all. A true Panopticon -- a society where everyone could see what everyone else was doing -- might bring a "freedom" from certain types of subterfuge, and attack.

Then again, I don't want to live in it.

That leads us to the question of how we handle the flood of data when looking for hostile activity, because governments are certain to use available technology to trap, parse and search that flood.

kyro3• 13 years ago

Is this information the NSA could've attained left to their own devices, or is this sort of stuff only accessible with help from service providers, eg Facebook and Google?

What I'm trying to ask is: with all the hullabaloo Google, Facebook, Yahoo, Microsoft, etc have made about individual, manual reviews of information requests, are we still being lied to? I suspect that we obviously haven't been told the whole story by these companies, and that they are a lot more implicit in this than they let on, but this article seems almost like definitive proof that they did indeed allow unlimited access to user information.

If this is saying what I think it's saying, then I feel seriously back stabbed by the startup darlings -- Zuckerberg, Brin and Page, etc -- that so many people here love and idolize. They should absolutely be held accountable.

jgrahamc2• 13 years ago

If you can tap large Internet connections then you can siphon off all HTTP and SMTP traffic and from there it's trivial to reconstruct sessions and from there get application level stuff.

I was doing similar things in the mid-1990s on shared Ethernet. It's really only a question of speed and scale and then of writing code that recognizes particular traffic (such as "this HTTP connection is a Facebook chat session").

kyro3• 13 years ago

Interesting. So then this can be done with zero assistance from service providers? Could providers have taken any steps to render that stream of information inaccessible? And if so, is it a costly effort?

mrt0mat06• 13 years ago

My question is, if they did this, why did each company have it's own join date in the prism timeline? wouldn't it just all happen at once? if they were tapping pipelines then the data would be collected regardless of the system, yet they all had very distinct dates that they were onboarded.

randac• 13 years ago

What makes you think they're even allowed to tell you the truth?

jcromartie• 13 years ago

are we still being lied to?

The answer will always be "yes".

northwest10• 13 years ago

"The NSA documents assert that by 2008, 300 terrorists had been captured using intelligence from XKeyscore."

So, even IF this number is not just another lie, XKeyscore has been made worthless, with something ridiculously small as the 2 prison breaks of the recent days.

That means: What remains is a police state that is not even "secure".

Good job, governments/lobbyists/"defense" corporations.

jstalin9• 13 years ago

I don't think it's a leap to assume that ALL TLS certificates are compromised and the NSA can monitor ALL Internet traffic.

nathan_long2• 13 years ago

No, that's a huge leap, actually. You could go make a certificate right now for your web site and keep the private key private. I could visit your site and verify that the signature is the expected one. If you and I are both diligent, we can know that our TLS session is safe.

rdl• 13 years ago

At this point the only thing protecting the US (and the world) from the worst tyranny imaginable is that USG's essentially unlimited power is wielded by individuals (rank and file workers, career bureaucrats, political appointees, and politicians) with a fairly reasonable sense of morality and belief that they are constrained by both the constitution and morality.

I'm don't think that is a stable long term system. Either some effective limitations (technical and political) are put in in the next several years, or a few decades of "us vs them" and self justifying security crises will produce a horrible result.

northwest10• 13 years ago

Absolutely.

Now, the thing that prevents that from happening is money.

So much money is concentrated on so few people that it protects itself and the owner becomes invulnerable. Add to that that too much money very often corrupts its owner. The predictable result for society seems pretty obvious.

jaekwon4• 13 years ago

You're spot on about money. I saw Keith Alexander's talk at the recent BlackHat conference, and was miffed at people applauding the guy, but then realized that it's a Black Hat conference, the epicenter of the monetization of exploits.

The other factor is that the NSA's reputation is irreparably tarnished, and they will continue to attract the wrong kind of people. I think we need to prepare for some dark times ahead.

totalforge• 13 years ago

Just a friendly reminder that if you looked at the slides, you have read a classified document, and therefore are guilty of a Federal felony. Cheers!

w_t_payne17• 13 years ago

And the NSA is passing your details on to the FBI riiiight about now.

Enjoy your federally-funded vacation!

northwest10• 13 years ago

At least you can be sure that dinner will always be served.

russellsprouts• 13 years ago

I don't think it's a felony unless you have signed a government agreement, like working as an NSA contractor.

rdouble• 13 years ago

It seems like working for the NSA is more like working at Inintech than it is like Minority Report. I've been more embarrassed by how Office Space retarded this seems than I have about the privacy abuses. I'm skeptical any of these supposed systems work, or even exist. It reminds me of Iran's pretend fighter jet.

w_t_payne17• 13 years ago

Imagine what Nixon would have got up to with this capability. He would probably still be in power!

stef253• 13 years ago

A couple years ago there was an AMA on Reddit from someone saying he was very deeply involved in spying on the general public's online lives, "at a level you can't imagine". Many technical questions were asked, all answered properly. I could never get it out of my head and now that Snowden has emerged I can't stop thinking he was the OP. Wish I could find this AMA again.

cracell• 13 years ago

Do you happen to have a link to this AMA? Not sure what to search for to find it.

stef253• 13 years ago

I can't find it, can't remember anything to search by. Very frustrating.

dmix• 13 years ago

Greenwald said in the comments, there is a lot more to come:

That House vote was about one specific topic - bulk collection of phone records - that this newest article has nothing to do with. That House vote isn't the be all and end all: it's just one small battle in what I can assure you will be a sustained and ongoing discussion/controversy.
There is a lot more to report still. Accuracy is the number one priority. That takes time.

swalsh• 13 years ago

The thing that blows my mind, is you hear over and over again about Billions of dollars being spent on large software projects for the government that seem fairly simplistic that ultimately fail.

The NSA is accomplishing some pretty impressive things, what are they doing differently?

fnordfnordfnord3• 13 years ago

Spending tons of our money while being accountable only to themselves.

mpyne3• 13 years ago

Spending tons of money didn't help when DoD was trying to field DIHMRS. The military still runs on an ancient COBOL-era payroll system because they can't successfully develop and field a replacement, despite some billions in wasted $$$.

superuser22• 13 years ago

Hiring mathematicians and computer scientists instead of MBAs and public policy people.

Probably also not having to follow government contracting rules (lowest bidder, preferring minorities and veterans) because who would have the authority to review their purchases?

brewdad• 13 years ago

Siphoning money away from "legitimate" projects leading to their ultimate "failure". It's just shell game accounting.

rtf1• 13 years ago

And it's not even limited by the internet. NSA collects every piece of information they can get their hands on, whether it's data on the internet or any other network, or spectrum signals, or simple imagery. And they do that by every means possible. James Bamford wrote years ago a number of books on the subject. And even before him, David Kahn painted a pretty clear picture. Why is everybody so excited so suddenly? Is there somebody on this forum who believes for a split second that Mr Obama and/or the US House/Senate are prepared to lay off way over 300,000 intelligence community workers, contractors and what have you??? -RTF

thejosh• 13 years ago

The funniest part is the ancient version of IE they are using in the screenshots.

MarcScott• 13 years ago

The scale, depth and technical sophistication of everything I've heard and read so far has made me change my mind on whether or not there is a technical solution to NSA and GCHQ surveillance. I'm now convinced that the only way to solve this is through politics. We need representatives that will enforce our rights to privacy, not clever hacks.

kazagistar5• 13 years ago

That said, we need to do the technical solutions as well. Don't use the difficulty as an excuse to give up on one partial solution, when the other solution is partial as well.

mrt0mat06• 13 years ago

I asked this in a deeper thread, but i would like to reask anyone that can explain. If the NSA is tapping pipelines as it seems they are, wouldn't the sources such as facebook and google all come online at the same time? if they were in fact referring to the pipeline access as their way into facebook and company, why did they all have different onboarding times? wouldn't they have all come on at the same time: the time when they tapped the pipelines? Maybe i misunderstand the process. I get that maybe they had to write some interface that interpreted the packets and sorted them as such, but that wouldn't take years.

stordoff3• 13 years ago

My interpretation is that the NSA basically have two main forms of collection: data directly from fibre intercepts, and data obtained (via voluntary agreement, court order, or otherwise) from private companies. This slide [1] would certainly suggest such an arrangement.

The fibre intercepts would fairly easily give access to HTTP traffic, and Facebook/Google/etc. would probably 'come online' at about the same time (there will likely be some differences as it appears there is a need to code a plug-in/processing engine for each major source to pull out usernames etc.[2])

What exactly the dates in the PRISM slide mean is somewhat unclear without more information. It could be, for example the date that the first court order is made, or the date when the company provides to the NSA a more automated way to query the data. I doubt that those dates are related to the fibre intercepts though.

[1] https://image.guim.co.uk/sys-images/Guardian/Pix/pictures/20...

[2] https://image.guim.co.uk/sys-images/Guardian/Pix/audio/video...

mrt0mat06• 13 years ago

so if that's the case, then it would seem that facebook, google, etc. are still lying? or could this be more gag order stuff?

buggedplan• 13 years ago

Germany used NSA's XKeyScore spying software: magazine report http://www.globalpost.com/dispatch/news/regions/europe/germa...

mikecarroll• 13 years ago

I wrote an e-mail to Congressman Mike Rogers about his misleading quote in this article. I encourage others to reuse my template and also ask him to justify his misleading remarks about Snowden's statement: https://news.ycombinator.com/item?id=6134672

We should start hold our public servant to task for lying to the American people about these programs.

adelpozo• 13 years ago

What makes this even scarier is to see IE in the screenshots. :)

VerilyForsooth2• 13 years ago

Why is the commentry on this topic always braindead?

The article states that there is a query interface using the email address as the key. But Where does it say that every single email/webpage from every single person is being collected? Such a task would be technically impossible. It seems far more likely that it's querying a database of pretargeted people.

There is so much hysterical nonsense regarding this topic. The cancer of conspiracy theory spreads.

computer• 13 years ago

The raw slides are included in the article. You should consider looking at them.

obs4711• 13 years ago

Yeah, I've read them. Where do they say that every single email/webpage from every single person is being collected?

pjmorris• 13 years ago

Such a task would be technically impossible.

Arthur C. Clarke's first law of technology: 'When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.'

mkhalil2• 13 years ago

Declassify on January 08, 2032. I wonder what kind of reaction the people of 2032 ( we who are still living that time ) would have had if they found out. Would they care? Worse reaction? Probably be used to gov. spying? It's a scary world.

_sabe_• 13 years ago

xkeyscore.com leads to Google maps.......just saying o.0

diggan• 13 years ago

No. It shows Google Maps in a iframe and was created 13 Jul 2013. https://gist.github.com/VictorBjelkholm/6127343

obelos• 13 years ago

How weird is it that cnn.com is included as a reason for being interested in HTTP?

LoganCale8• 13 years ago

Another example is shown where they are searching for everyone who read a particular article on the BBC website.

mkhalil2• 13 years ago

People aren't going to care about this until they understand what consequences this may have to THEIR personal life. Live with it. Not changing unless the knowledgeable/wise starts educating the general population on how they effects them.

Most people might speak against it (include people here) but at the end, they have the "I'm not doing anything wrong, who cares, not worth the effort" mentality.

fideloper• 13 years ago

Are the major news networks ignoring this story? Briefly checking, I only see Fox News reporting related stories, naturally blaming the Obama Administration (perhaps fairly in this case).

epoxyhockey• 13 years ago

In the first slide, there are 25 red dots in a row on the bottom.. I wonder what those are supposed to represent?

junto8• 13 years ago

Digital finger-printing, so that you can tell who leaked what document.

I dont want to detract from Snowden's very noble act, but I hazard a guess that Snowden knew that the documents he leaked could be traced back to him, or at a minimum a small team that he worked with.

llamataboot2• 13 years ago

Can someone explain this to me like a 5 year old and how it interfaces with PRISM?

noir_lord3• 13 years ago

This is considerably more invasive than PRISM.

This system logs HTTP metadata and data (think the address on the envelope and the contents of the envelope), the metadata for 30 days the contents for 3 days.

This http data is essentially everything that goes over the wire all of which is then shovelled into a database with a fairly sophisticated (if not pretty) front-end that allows really invasive searches.

You can search for stuff like "all emails that contain the words sex doll" or "nudes" and contain jpegs...of course the users would only use this system for legitimate operations covered by warrants.../s.

This is the first of these releases that have really made me stop and go "whoa" mostly because this is "better" (bigger, more complex and capable) than anything I expected them to have now (and this was in 2008).

llamataboot2• 13 years ago

I guess I am mostly confused about whether they have any email content capabilities...and if so, how?

cnlwsu• 13 years ago

Ok, so ignoring all moral/ethical issues with this. Wouldn't it just be awesome to work on a project like this? Unheard of funds, tons of data, interesting CS problems all around. I am sure they did everything possible to make it miserable on the developers but nonetheless... sounds fun from a completely detached CS perspective.

ratsbane• 13 years ago

Yes, I was thinking the same. Sort of the software engineer equivalent of flying the best fighter jets and recognizing that you're going to be using them to bomb people in mud huts instead of dogfighting equally-classed opponents.

w_t_payne17• 13 years ago

I just read the actual XKeyscore slide deck. Unlike a lot of leaks, these slides are totally worth a look-see: http://www.theguardian.com/world/interactive/2013/jul/31/nsa...

imrank1• 13 years ago

interesting that is not even on CNN or MSNBC yet. Has anyone else seen coverage on US news?

api2• 13 years ago

I'm getting seriously irritated at the "I have nothing to hide" crowd. For starters, here are a few ways this can go horribly wrong:

* Industrial espionage -- it's big business, and I'm sure it pays better than being an NSA analyst.

* Foreign espionage -- since this gives unlimited querying power to every agent, a single "turned" agent could inflict massive damage on U.S. government and industry interests on behalf of a foreign power. The potential for double agents is huge.

* False positives and guilt by association -- being flagged as a "person of interest" and then essentially persecuted because you have fringe ideological interests, are looking up a lot of info on terrorism for a book project, have a friend who knows radical Muslims, etc.

* Corrupt use in political campaigns by incumbent politicians with access -- obvious.

* Blackmail and other corruption.

* Use by government agencies with access to spy on other agencies.

... I'm sure creative people can think of more.

antimatter• 13 years ago

This. I can't seem to make people around me understand this point.

legutierr• 13 years ago

Do you list these negatives when having that conversation? Do they not see the possibility of something like this being used against them or family members when you lay out the possibilities specifically (blackmail and industrial espionage could hit anybody)?

The "nothing to hide" trope seems to me to be entirely based on a false dichotomy that contrasts "nothing to hide" with "unpatriotic/criminal". I think this is primarily because people lack the imagination to consider the other seedier and more lucrative uses of surveillance.

If they were confronted with these other possibilities, would your acquaintances change their thinking? Or do these other risks--for example, the risk of having an employer targeted by competitors unfairly (potentially leading to layoffs), or the risk of having a representative vote against the interests of his or her district because of blackmail (potentially leading to a loss of government services and investment)--simply not resonate?

mncolinlee• 13 years ago

"Industrial espionage--"

According to a New Zealand whistleblower back in the 90s, this was one of the main purposes of the Echelon network. Imagine what happens when your larger competitor gets in bed with the NSA. According to whistleblower Russ Tice, the Bush NSA was able to request intercepts on Senator Obama, so there certainly could be enough corruption for back room deals to occur for your startup's private information.

This is yet another reason to encrypt your git traffic.

area51org• 13 years ago

Also, how about: just because I have nothing to hide doesn't mean that I don't mind being watched.

jwoah12• 13 years ago

I completely agree. I got very frustrated talking to two intelligent people (one a lawyer) about this the other day. I think everyone should be educated to make sure they know about these points so that they can intelligently explain them to the "I've got nothing to hide" crowd. The 4th Amendment and the right to privacy just doesn't seem to carry much weight with the average person any longer (if it ever did).

dil8• 13 years ago

I think that the 'nothing to hide' crowd don't understand that the concept of things to hide could grow and grow unless the surveillance power are kept in check. Today, only terrorists have 'something to hide' tomorrow it could be activists, then journalists and so on, until they 'nothing to hide' people become the victims. http://rt.com/usa/justice-department-admits-spying-228/

People need to look at this long term and realise that abuses of power will continue to intrude larger and larger sets of the population unless they are stopped now.

diydsp2• 13 years ago

Thank you for making that post. I've been repeating these points on broken-record for the last few months.

I would also like to add /family/ as a huge pressure point, akin to your guilt by association.

IOW, one may have nothing to hide, but their family member does and one can be controlled by threats to that family member. It's disgusting and insidious, but that's how bad, scared and dangerous people operate. We are not using our intelligence if we allow ourselves to be vulnerable in this way.

a3n• 13 years ago

Your points are excellent. And even if we all do have nothing to hide, an agency of the government that decides on its own to break the 4th and 1st amendments, and reinterpret the law practically out of existence, in secret, and lies about it to Congress in its oversight capacity, is A Bad Thing(tm).

superuser22• 13 years ago

Anytime an adversary (opponent in a lawsuit, police investigator, promotion candidate, political candidate, etc) is owed a favor by a member of the intelligence community, you are going to lose.

The whole of your life may be innocent, but a single interaction (searched for porn? vented about someone in a private conversation?) taken out of context can almost certainly destroy you.

BecauseWeCan• 13 years ago

Api, please make a post expanding on these ideas. Then post it here and send some copies to international news sites, like the guardian.

We need to wake up, your post would help a lot.

throow2• 13 years ago

I'm having the same problem as some other commenters here in that people don't seem to care about these hypothetical situations. They mostly trust the gov't to only track the "bad guys". Are there any good cases of the above-mentioned actually happening? Any other suggestions for responding to the nothing-to-hide argument?

lukifer3• 13 years ago

My best counterargument for "nothing to hide": What do you think Nixon would have done with access to a limitless database of (effectively) every human on earth?

Privacy is important. But vastly more important is unaccountable power.

rodgerd• 13 years ago

It seems to be oft-overlooked that Manning and Snowden have managed enourmous breaches of what is supposed to be secured information (yours and mine, and probably everyone elses!). Staggering breaches. Any they're boh essentially nobodies - Snowden was an army washout who became a securtiy guard who has penetrated the US security apparatus in a way that would have been considered the greatest KGB coup ever if it had been a cold war operation.

There's a staggering lack of basic competance around protecting this stuff. The CIA director who lost his job over Aimes must be wondering what the modern mob have to do to get fired.

scrrr• 13 years ago

"Mr. <webservice-ceo> Does your company offer a backdoor for the government? It looks like it, even though you have declared that there was no backdoor just a few weeks ago!" - "Uhm. Not wittingly!"

MichaelGG4• 13 years ago

That's like accusing AT&T of having a backdoor because your apartment has thin walls and someone could overhear. Or that your ISP has a backdoor because your WiFi isn't encrypted.

The FBI was doing this decades ago with Carnivore. Why is it at all surprising that such a program continues to collect unencrypted information you sent over the Internet?

bobbydavid2• 13 years ago

access.

any citizen can go out and listen to public conversations, but we are not all invited to put our e-stethoscopes up to the internet backbone.

Buzaga2• 13 years ago

Yep I'm done, I'll just go about my life from now on...

It's total power, I think it's unlikely that they'll want to give up on this kind of power, they'll probably keep signing governments and 'the tech' will eventually be exported and in the hands of governments everywhere, they'll keep building this and they'll create tons of algorythms of course because it's just too much data, any resistance can be crushed... and it's so much power eventually some dark times will begin... I'm done with the topic.

general_failure3• 13 years ago

Totally agree. They are not going to give this up... and worse, we are not going to fight this. We are speaking up only because this is the internet. We just want to get on with our lives.

Kinda surprising why all the people who are 'overwhelmed' and 'terrified' in the parent thread don't come out and protest. Oh wait, there's kids to feed. My bad, sorry.

(no snark)

overgryphon• 13 years ago

For 99.9% percent of people, this is exactly what will happen. A few shocked moments, quickly followed by returning to all of the important things in their life. No one has time to protest all of the hundreds of wrong things we supposedly should be protesting and there aren't any clear or easy to implement steps to avoid NSA spying. There is simply no reasonable actionable item here for casual news readers.

The real consequences of this news will be seen in the actions of companies. As 'cloud' (oh I hate buzzwords) technology becomes increasingly more efficient and cheaper, as Amazon, Microsoft, Openstack and VMware duke it out over cloud customers, will those customers trust them with their data? Will companies invest in private clouds for increased security, or will large public cloud service providers be able to win over and keep their trust? How much money have public cloud service providers lost since the leaks began? How many companies are now unwilling to use cloud services from US-based companies?

avdempsey• 13 years ago

There may be a protest on Sunday near you: http://1984day.com

Many hands make light work.

Buzaga2• 13 years ago

Note: I'm a foreigner.. :t

andy_ppp2• 13 years ago

Judgement day is inevitable.

G�#��$�5�%��V��5�F�'98u�x�)�w��[_Fa��6�1f�!��['��"��VGu~w� ��|�U��Z�hep��G��^7{��K�wq��h|ڛ�m=�$L ��t� _��pM<�q��;��Y��C �M]!C�6ҝnV[�c�ϾWa�?C�M�"X*��b]0�Aļ��Li3`�P�#�j�f�u��!wb�]t_�q�&EԞw��r�.<?K��{

northwest10• 13 years ago

General reflection:

As I recall it, our right to privacy is defined by the "reasonable expectation of privacy".

Currently, I see any such "reasonable" expectation to be almost zero.

Therefor, I have to conclude that we have already lost the right to privacy.

kepano• 13 years ago

Can someone with data center expertise extrapolate the physical scale of this operation? In terms of storage and computing power it must rival if not surpass what Google has built, no?

heyitsnick• 13 years ago

And, according to Greenwald, there's a lot more to come. From the comments:

"There are thousands upon thousands of documents and they take time to read, process, vet, and report. These are very complex matters..... there is a lot more to report still. Accuracy is the number one priority. That takes time."

badclient• 13 years ago

As one slide indicates, the ability to search HTTP activity by keyword permits the analyst access to what the NSA calls "nearly everything a typical user does on the internet".

It seems like everyone's been attacking the wrong folks. From this article it appears that bulk of the data is being tapped at the data center level and then parsed. This begs the question how it would be able to make sense of https traffic.

curbrusiasm2• 13 years ago

This has been up here for 5 hours and on the Guardian's website for nearly 6 hours. How is it possible that not the NYTimes, FOX, NPR, the Washington Post, or CNN have picked this up? These organizations are an embarrassment to the profession of journalism.

chrisstanchak2• 13 years ago

https://news.ycombinator.com/item?id=6136016

0003• 13 years ago

Here is the newsdiff of today's NYT article on the disclosure of the declassified docs. [1] Talk about burying...

http://newsdiffs.org/diff/290704/290768/www.nytimes.com/2013...

curbrusiasm2• 13 years ago

So rather than making a headline, update an article on another topic, specifically update an article about a PR move by the White House intended to stop talk on this issue. Assholes.

typon3• 13 years ago

I don't think its so much "an embarrassment to the profession of journalism" in terms of incompetency but rather willful ignorance.

chrisstanchak2• 13 years ago

Summary of how this story is being covered around the world vs. in the US.

http://imgur.com/a/tS7h4

mrt0mat06• 13 years ago

i was shocked by this too. about an hour ago i went to CNN.com and searched the keyword xkeyscore and got 0 results. Not a single article?

amckinlay• 13 years ago

Wait, does the collection require FISA warrants? Is the collection still limited to foreign nationals?

junto8• 13 years ago

The map on page 6 is interesting. Server locations of note:

  Moscow, Russia

  Caracas, Venezuela

  Tripoli, Libya

  Hubei Province, China

  Burma

  Lagos, Nigeria

  Saudi Arabia

  Iran (and geograhically surrounding Iran)

  Ukraine

Based on page 13, I wonder if Google have any servers at these locations?

Oh, what a surprise: http://royal.pingdom.com/2008/04/11/map-of-all-google-data-c...

Does that look familiar?

jnbiche• 13 years ago

I'm very suspicious of Google's role in all this, but there's not much overlap in the list above and the map you refer to, outside of the normal population and business centers in Europe and the US. I didn't see any in Google data centers in Libya, Burma, Nigeria, Ukraine, Saudi Arabia, Iran, or even Venezuela (the only one in S. American looks like it's in Brazil). Based on a quick glance, the only overlap between the list above and Google's server looks like Moscow, a major global business center.

Were we looking at the same map?

junto8• 13 years ago

Sorry, I wasn't suggesting Google were complicit, but the NSA need to suck up search queries, so it makes sense to locate these NSA servers right next to local Google installations.

Again, page 13; a local Google Pakistan search query.

andy_ppp2• 13 years ago

I just censored an email I was about to write in case it is used to discredit me in the future. Jesus. The thought police are here :-(

sinak• 13 years ago

Hey folks,

Just wanted to add a note and say that if you're angry about this, the best thing that you can do is to get out into the streets and protest everything that's been going on. Check out the Restore the Fourth rallies happening this weekend, share them on social media, and sign up for your local event.

http://1984Day.com

Getting out into the streets is the single most significant thing you can do - even more effective than calling your legislators. The events on Sunday need to be bigger than the events July 4th for this to really be a success.

LeeLorean2• 13 years ago

It is interesting on slide 17 that the NSA can decrypt all VPN traffic.

Does this indicate that they have broken HTTPS, or simply that they own VPN companies like Private Internet Access?

CPAhem• 13 years ago

Slide 23: "Show me all the Microsoft Excel spreadsheets containing MAC addresses coming out of Iraq so I can perform network mapping"

Does MS Excel store your MAC address in the xlsx file?

budman• 13 years ago

Wow. Just Wow.

For years all of this was in the back of my mind as being capable but my not wanting to think like a conspiracy crackpot just dismissed the thought as it couldn't be possible. A conspiracy takes a lot of co-operation from within large corporations who must also remain it a secret. Surely someone would have a conscious and leak it? Or one of companies we all look up to as a modern example of do-good company would say "Hell NO" to the attempt and then let the world know what was attempted. Guess that was eventually proven true with Snowden (a real hero imo), just shocked they were able to operate to the scale they did for so long before a Snowden came along.

In my mind, this is not so much a shock to me regarding the NSA as well as the current evil government we have had in place. Doesn't take a genius to realize the president lies to our face on TV about trivial issues/promises, so expected for top secret stuff.

What is the BIG stomach churning shock to me is the very companies that we have come to know that are multi-billion dollar conglomerates providing service/products for millions for every day use has been a part of it. A part of this secret web while all the while proclaiming privacy for it users. I guess at end of day profits still rule the roost. "Just do this for us, turn a blind eye, and you get to go on making your billions". I wonder how many CEO's knew of all this. Gates? Zuckerburg? Etc etc.

I feel like I have no outs now. There are no alternatives to current establishment of companies that make our lives easier. Should we all wipe our PC's and use Linux, sell our phones and use Ubuntu Phone, not pay for SSL certs anymore (another mafia), etc?

nullc• 13 years ago

It's been being weirdly suppressed on reddit: http://www.anonmgur.com/up/17832a6eafb09376d012090ff1b06dbe....

Every time a thread on this hits the top it gets mod-deleted.

osth• 13 years ago

http://s3.documentcloud.org/documents/743252/nsa-pdfs-redact...

Missing: How much did this cost? Did the government (taxpayers) overpay?

rehack• 13 years ago

Makes me Wonder, if the Internet in this widespread form, was allowed so that they can snoop (so easily)?

When I was a kid, my father, had told me a story that in Russia people are scared to speak their minds, for fear of being snooped via any hidden gadgets in the walls.

w_t_payne17• 13 years ago

I wonder if we should try to put together a programme to try to drain the NSA of technical talent ... offering jobs or other incentives to try to persuade developers currently working for the agencies and their various contractors to resign?

snambi• 13 years ago

Wow... Govt is into big data. I wonder what they use for analyzing all this data

XKeyscore: NSA program collects 'nearly everything a user does on the internet'

Related Stories