The Surveillance Engine: How the NSA Built Its Own Secret Google

https://firstlook.org/theintercept/article/2014/08/25/icreach-nsa-cia-secret-google-crisscross-proton

by aburan28 • 12 years ago

356 88 12 years ago

firstlook.org

us0r• 12 years ago

Somewhat OT but its amazing out of 3 "FTEs" 1 was for design ("GUI") and this is what they came up with. [1] pg 22

https://firstlook.org/theintercept/document/2014/08/25/shari...

seiji3• 12 years ago

Also notice the entire screen shot. Netscape on Windows XP in, best we can tell, sometime after March 2006.

Netscape. With an AIM launcher in the status bar. Accessing TOP SECRET//COMINT//NOFORN resources.

apsec112• 12 years ago

This looks like Netscape 7 (http://en.wikipedia.org/wiki/Netscape_(version_7)#mediaviewe...), which had its final release in August 2004, not the original Netscape.

Would it have been more secure if they used IE 6?

C7E0F338E42448• 12 years ago

Maybe all software released after this is infested with spyware and exploits, and we are just led to believe it's the other way around

Amezarak• 12 years ago

It would be on a totally isolated network. Classified networks don't touch unclassified networks.

MalcolmDiggs• 12 years ago

It's kind of remarkable that they kept the lid on this on long as they did. With so many local/loosely-related agencies involved, I'm surprised somebody didn't leak this much sooner.

danvayn• 12 years ago

My guess is a lack of understanding.

d1str0• 12 years ago

It clearly says only a few intelligence officers from the different agencies had access. So it's not like any ol' police officer of FBI agent had access to this at all times.

chiph• 12 years ago

Or maybe they did, and the search page had been rebranded so the officers didn't know it was operated by the NSA. Perhaps "Fusion Center Search 9000" or similar.

Jimbotron• 12 years ago

Yes, I find it VERY hard to believe that this went on and nobody mentioned it at all. Not one disgruntled employee, not anyone mentioning it to a friend at a bar? No way.

sentenza• 12 years ago

Uhm. Just because you didn't believe it doesn't mean that it wasn't known. The CCC was talking about this stuff years ago, some of it conjecture, some of it information from old sources/anonymous sources and the like.

Now we have proof. So that's that.

Also, given the strong "disincentivization" for whistleblowing that is going on at intelligence agencies, I am not at all surprised that it took a few years before somebody went public with it.

Hell, it used to take decades before this kind of stuff became public. Just this year we were made aware that the West German spy agencies opened _all_ cross-border inter-German mail. You know, the kind of thing that we in West Germany were told only the East Germans would do. That was kept secret for more than 20 years after it stopped being done, even though thousands of people were involved.

chadwickthebold• 12 years ago

When your livelihood is based entirely on your security clearance and your ability to keep working in the security industry you make it a very big point to not talk about this stuff outside of work. I had a friend who went to work for a 3-letter agency and started consistently avoiding bars/drinking because he was afraid he'd say something. This is exactly why whistleblowing is a problem in the intelligence community - they'll sue you to oblivion and/or imprison you for saying anything at all.

omouse• 12 years ago

I wouldn't be surprised if they talked about it but if it's rebranded it could be seen as "Yet Another Tool That IT Provides". People talk about their favourite consumer apps, but only ever complain about the shit they're forced to use at work. In this case, the tool is working fine so there's no need to sing its praises. If it was a shitty search engine, they'd be telling everyone "we collect all this data and for the life of me, we can't even search it! what's the point of violating the constitution at all?!"

eyeareque• 12 years ago

People always assumed they did this but now we have proof.

I thought it was amazing that the government spent so much time discussing the call records being logged.. when they are doing so much worse. Maybe that's how they keep people focusing on what the government wants to talk about? (aka look over here, nevermind that thing over there...)

This find is way worse than call detail records..

higherpurpose• 12 years ago

It was pretty obvious that's why they were doing it. The FISA Amendments Act is also much worse than the 215 section of the Patriot Act - or the Executive Order 12333, which is even less talked about.

That doesn't mean the call records issue wasn't important. After all, they pick their torture and drone assassination targets based on that alone, in many cases. But considering the Internet is such a big part of everyone's lives these days, I think NSA getting a firehose from the big carriers and cable providers is a bigger deal, yet I've heard very little talk about denying NSA access to the undersea cables.

knowaveragejoe2• 12 years ago

I think it's a combination of things. On the one hand, the phone records issue is the most easily understood by the general public, and so it already had traction there. It sort of fell into the government's lap that they could focus all of their press' attention on that issue in particular.

I agree doubly so however - the phone records are near the very bottom of the list when it comes to the severity of what Snowden's(and possibly other's) revelations have brought to light.

wyck• 12 years ago

Brought to you by meta data commodities and pattern recognition limited - don't be evil. just be data.

nowarninglabel• 12 years ago

I don't think it remained "under the radar" like some are commenting. There's mention of it in Snowden's disclosures and if you search around, lots of government recruiting related info, e.g., http://www.socnet.com/archive/index.php/t-108034.html

eli2• 12 years ago

I guess it's always cool to learn about top secret stuff that spies do, but I don't get why the specifics of how their search engine works is significant. Am I just missing something?

IBM• 12 years ago

Glenn Greenwald got a treasure trove and it's time to cash in.

yen223• 12 years ago

Glenn Greenwald has a lot of Powerpoint slides. Am I the only one who wants to see the source code for ICREACH, and all the other ALLCAPS projects?

pdkl952• 12 years ago

This, as with most of Greenwald's work on this subject, has been a slow effort to educate people about a very technical (for the average person) topic. Building on previous reports that showed the type of activity and the threat it can cause, this installment is about establishing the pattern of illegality.

If the NSA could justify the time and expense to make a search tool, there must be a lot of searching needs... which aren't legal for the NSA to send to the domestic-oriented agencies.

Also, more explicitly-stated details about how powerful "metadata" can be. Obvious to anybody who understands INNER JOIN, but this is still new for most people.

//Of course, I'm sure all this ignoring of warrants and the 4th Amendment, to quote a DOJ lawyer in the recent EFF mess regarding Jewel v NSA, is all for the "national security of this company". (emphasis mine)

motters• 12 years ago

I think that's the key insight. It may not be legal for NSA to look at records of US citizens, but build a search engine and there are other departments for whom it is legal to do that. NSA just becomes the facilitator or router of the data.

brianmwaters_hn• 12 years ago

Also, this article goes over "parallel construction" quite a lot. Although we've heard about it a lot in the past year, this leak shows how and what kind of data NSA is sharing with DEA, FBI, etc.

It's even more important for us to know what kinds of data about us domestic law enforcement has access to, than it is to know what foreign intelligence has access to.

mpyne6• 12 years ago

Also, more explicitly-stated details about how powerful "metadata" can be. Obvious to anybody who understands INNER JOIN, but this is still new for most people.

The power of metadata is implicit. No one at NSA is arguing that metadata isn't powerful, that's the whole reason they're collecting it.

What the NSA is arguing is against the meme that they're trodding roughshod over the Constitution and illegally scooping up data (as they have done in the past). On the contrary, this time they've been very careful to try to remain within the law, and one of those arguments is by pointing out that, according to the law, some types of information are less protected than others.

Greenwald can educate all he wants, but the problem isn't that Americans are stupid or whatever you all seem to think. Americans on the large have been assuming the NSA is doing far worse than what Snowden accused them of, because Americans (on the large) expect that the NSA (or some other branch of the government) is actually able to intercept and listen in on "those jihadis' phone calls and emails".

You're talking about a generation of Americans who grew up listening to PBS stories hounding the Federal government agencies of the Bush and Clinton Administrations for "not connecting the dots", and for having a "lack of imagination", and for "treating terrorism like a law enforcement" problem and for being hamstrung by a bunch of civil liberties protections.

This is by and large the same generation that decided to vote against privacy when they bought into to having their digital lives collectively managed by Google, Apple, Microsoft and Facebook. Explain the low-level bits and bytes to them all you want, but they've already voted on the big picture they want to see.

curiousDog• 12 years ago

Wonder how and where they recruit their top-talent. Pretty much every top scorer in my school went to Facebook/Google/Msft/Amzn. The mediocre ones went to Northrop Grumman/Raytheon/Rockwell and the like.

anoothrowaway• 12 years ago

My experience in school (and grad school) was similar, so I've been curious about the same thing. Perhaps, they only recruit folks from the industry after they've worked for a while and have specialized?

visarga• 12 years ago

Maybe they recruit people while they are employed at FB/GG/Msft/Amz, you know, to help with national security while also receiving a second wage.

seiji3• 12 years ago

The Company doesn't exactly encourage recruits to advertise their new position. You mostly see people just vanish for a while. Young recruitment is largely done through direct faculty recommendations at schools, so it's more of a "don't find us, we'll find you" proposition.

hiddencost• 12 years ago

Yup. I had a friend who spent some time there. Easily the smartest kid in my major. Hired due to a direct referral from faculty. It gets scary, once you see how many math professors have security clearances.

niels_olson2• 12 years ago

Yeah, my mom majored in math and minored in Russian. She got the NSA offer in her last semester. But she wanted to teach, and thought the whole thing was more than a little spooky. And indeed, it was a faculty referral that prompted the interview offer.

psaintla• 12 years ago

Exactly, a good friend of mine in college was recommended for a job at the NSA by one of my professors in early 2001. He went to having a heavy online presence and being a constant in our social circles to completely disappearing. His personal site went dormant, he stopped showing up on irc, abruptly dumped his long time girlfriend. My roommate bumped into the guy at a grocery store in Virginia but other than that he doesn't exist to anyone who knew him for years or online.

poof131• 12 years ago

I think the bridge between math and programming is relatively small, so if you have the most mathematicians in the world, you probably have some pretty good programmers too.[1]

But for contractor, there are job boards for people with clearances.[2] Although you get a pay boost of at least 10-15k, I'm sure the bureaucracy is awful.

I also wouldn't assume that all the top people go to biz. Some very talented people work in gov and academia. Not everyone is guided primarily by money & scores.

[1] http://www.math.uh.edu/~tomforde/Web/Jobs.html [2] http://www.clearedconnections.com/

PostOnce• 12 years ago

The best programmer I know works for Northrop, because you can make >$100K and live in a rural or semirural area (near an AFB), you can have $1000/mo rent, vs living in SF making $150K and having massively higher expenses and have to be far away from "home".

srj2• 12 years ago

Sure but isn't life a great deal more boring out there? It depends on a person's priorities.

ekm2• 12 years ago

I am guessing they get them young.Who sponsors MATHCOUNTS,AMC and other contests for highly gifted kids?Raytheon,Northrop Grumman.

lucasgw• 12 years ago

For the same reason some MDs go to work for Médecins Sans Frontières instead of a top-paying job at a private hospital or partnership. A sense of duty. I grew up in a small town dominated by military/industrial companies. A lot of top talent moves there because they believe working for the government is what they should do.

wil421• 12 years ago

These recruiters find you, you dont find them. I doubt that any of your classmates would be so dumb as to tell everyone they got hired by a 3 letter agency. They probably are looking for more seasoned devs than the typical college grad.

Also you have to be an American citizen with parents who live in America. At least I know this is the case for civilians who want high security clearances.

pdkl952• 12 years ago

What I want to know is if this search tool is the backend for the "parallel construction" application forms[1] from earlier this year?

[1] https://www.techdirt.com/articles/20140203/11143926078/paral...

atmosx3• 12 years ago

Possibly, among other things.

niels_olson2• 12 years ago

Is it just me, or is it tragicomical that I can't access firstlook from a government network?

Tangokat• 12 years ago

According to the Intercept, the U.S Military has banned employees from visiting First Look because there could be classified information on the site. [1] [2] (extra source since you can't see the real one..)

Maybe that is what you are experiencing?

[1] https://firstlook.org/theintercept/2014/08/20/u-s-military-b...

[2] https://thebulletwire.com/u-s-military-bans-glenn-greenwalds...

atmosx3• 12 years ago

That's similar to the wikileaks ban. I don't understand this policy: If something is accessible virtually from everywhere, what do you gain by not allowing your employees to access the info?!

The only thing I can think of, is internal issues at political level (who said what lies to whom to achieve more funding about his work, etc.) which even that won't take long before the interested party eventually finds out.

just1comment6773• 12 years ago

The U.S. DoD is banned from accessing firstlook.org (among others) from government-owned computers. In fact a memo was sent explicitly permitting DoD members from accessing these websites from personally-owned systems.

There is a memo from a few years ago that prohibited accessing Wikileaks from personally-owned computers, but the order was rescinded.

Twirrim• 12 years ago

I think I'm more surprised that this was a surprise to people.

Of course they built a search engine. Wouldn't you? Don't you have similar at your workplace? We use them all the time. Think about web interfaces built on top of ElasticSearch, for example. Is that not a 'search engine'?

dredmorbius• 12 years ago

It's not the building a search engine part.

It's the opening it up to LEO. Taking homeland security-justified warrantless searches and turning those over to criminal investigations is far beyond unconstitutional.

mpyne6• 12 years ago

Taking homeland security-justified warrantless searches and turning those over to criminal investigations is far beyond unconstitutional.

Except, that it's not.

It's all about how the government comes into possession of information that makes it the fruit (or not) of the "poisonous tree".

Once the government legally learns of some bit of information, there's no requirement by itself that they have to "pretend not to know" something is going on. That's why a third party voluntarily divulging information to the government doesn't cause that evidence to get thrown out of court: There may be a cause for civil or even criminal action against that third party, but that doesn't invalidate the evidence.

Now, Congress has passed statute laws limiting information sharing between intelligence agencies and law enforcement precisely because of the threat of having this all-seeing eye subvert democratic government, but the limits were never complete exclusions, even before 9/11. And the reason statute laws were needed is because there was no barrier to this activity from the judicial side alone.

Parallel construction came in not because evidence was illegally gathered, but because giving the (fully legal!) chain of custody for evidence derived from intelligence sources would have quite naturally have "burned" that source or method, so the NSA would require law enforcement agencies using that data to use alternate (again though, legal) means to make a case in court, in order to protect their intelligence source. It's the old apocryphal dilemma about whether to let Coventry be bombed to protect ULTRA, applied to the post-9/11 world.

cryptolect• 12 years ago

To me, this explains why the five eyes nations are pushing for (meta)data retention legislation. It's a condition of participating in the scheme.

DigitalSea• 12 years ago

Imagine if hackers were to somehow to find out where this tool is being hosted and then brute force their way in? Presumably such a tool is heavily secured and off the grid, but if it is somehow accessible from the Internet and an attacker were to find out where, it's only a matter of time now that the cat is out of the bag. Could you imagine hackers having access to troves of metadata and information like that? It's a scary thought.

I'm surprised such a thing took so long to be revealed. If you've got as much data as the NSA has, wouldn't you want a Google like search engine to be able to search through it? It makes so much sense which is why I am surprised some people are surprised about this.

oceanstone• 12 years ago

Furthermore, think of how profitable surveillance on certain individuals could be. Could a military contractor not have a thriving side business funneling out intelligence on innocent American targets? Could multiple people not be hired just to clear tracks and make justifiable excuses for accessing the records?

If Snowden, as one person, accomplished what he did with (the majority of Americans would say) good intentions, imagine what a team of people, who were just as smart if not smarter, could do if they didn't have morals guiding their actions. Imagine how well a team could cover their tracks, dot all the I's, and cross all the T's, compared to just one guy.

This is all common sense, but it bears repeating.

smokeyj• 12 years ago

I'd just play the market. Find dirt on CEOs, short their stock and pass the dirt to the media.

BorisMelnik• 12 years ago

I am just not surprised about this at all. They have more information than I could ever imagine, wouldn't any rational person assume there is a search engine to index, sort, parse, and return results?

seems like a lot until you consider how many indexed pages Google has: http://i.imgur.com/EqIJAoL.jpg

why not throw in grains of sand or atoms in the universe?

Perseids• 12 years ago

Nice comparison, though I hate it when graphics look like they were drawn to scale, but actually aren't. (The Google indexed pages bar would have to be more than 10 times larger.)

More on topic: Whether you can compare the index sizes or not mostly depends on their signal to noise ration. Given the number of Google search request per month and assuming and average of 5 viewed results, you can put a lower bound of 99% of pages that are never seen by anyone. As the search results should greatly overlap the actual number of relevant sites is probably orders of magnitude less than 1%.

fit2rule• 12 years ago

I believe its really telling that, rather than take the tactic of fully promoting open society, the inclination is towards more and more secrecy. Like, dire, utter secrecy. Kill someone-style, secrecy.

Imagine the other end of the scale - where in fact every detail about everyones lives is wide open and available for everyone and anyone to access. Willingly. Freely. A new order of celebrity: total telepathy.

Do you think we'd be dealing with terrorism, then? Would there be the idealist, killing souls, for a little private time?

Dolimiter2• 12 years ago

Is anyone else puzzled that stories about the NSA are accepted as gospel fact when posted on Hacker News?

The Reality Distortion Field appears, and people believe, because they want to believe.

It depresses me, the lack of intelligent discourse.

Most tech people I meet actually believe that the NSA records and stores all telephone calls. It's depressingly stupid, but I have given up arguing, logic and sense are not welcome when the NSA is the topic.

shiven• 12 years ago

Perhaps proof to the contrary would be more useful than the veracity or logic of an argument?

Unfortunately, with the NSA, independently verifiable proof of anything, especially doing the right thing, is impossible to attain.

mpyne6• 12 years ago

You're asking the parent commenter to prove a negative? Is this what logic has come to at HN?

glitchdout• 12 years ago

They do record and store (foreign and domestic) phone calls. Your comment proves you're simply an ignorant or a shill.

Proof:

* NSA Collects ‘Word for Word’ Every Domestic Communication, Says Former Analyst http://www.pbs.org/newshour/bb/government_programs-july-dec1...

* NSA collecting phone records of millions of Verizon customers daily http://www.theguardian.com/world/2013/jun/06/nsa-phone-recor...

* NSA surveillance program reaches ‘into the past’ to retrieve, replay phone calls http://www.washingtonpost.com/world/national-security/nsa-su...

* NSA spying flap extends to contents of U.S. phone calls http://www.cnet.com/news/nsa-spying-flap-extends-to-contents...

Dolimiter2• 12 years ago

I'm a "shill"? Really? This is the level of debate?

Also, none of your links in any way prove that the NSA stores all phone calls. None of them. 1 is hearsay from someone, another talks about metadata, and another says that the NSA can record any phone call (yes we know that). Nowhere is there any proof that all calls are being recorded.

This is what I am talking about. It doesn't seem possible to have a conversation about the NSA on Hacker News that is based on logic, reasoning or sense.

DanielBMarkham• 12 years ago

This has been released and is use by almost two dozen federal agencies and now is the first time we're hearing about it?

When folks tell us crazy things, like the government is tracking every place you go and your opinions through your cell phone and social networks, we're supposed to say something like "That's extraordinary. With extraordinary claims, we require extraordinary proof" Then, if they persist, we're supposed to say something like "Such a program would require far too many people to keep a secret. We couldn't even keep the atom bomb a secret. The government is terrible at keeping secrets. Such a claim is just too far-fetched."

These are the traditional things taught to people who are supposed to be clear-headed and rational. It's the way we engage crackpots without taking them too seriously.

These responses seem to have failed us miserably in the current circumstances. As it turns out, yes, that's what they were doing, and yes, it was extraordinary and required lots of people to keep incredible secrets. But it still happened.

These things keep happening in the realm of automated surveillance, both by the government and corporations (and worse, when corps do it and the govt scoops it all up later) that would have been considered completely whacked just ten years ago. The stuff of paranoid fantasies.

Our tools of rational inquiry have failed us.

knowaveragejoe2• 12 years ago

Just curious, where is the bit about keeping track of your opinions? Surely it's been proposed(so has everything and the kitchen sink), but that doesn't mean it's effective or useful in any way.

mikegioia• 12 years ago

You're up against a growing population of people who inherently think that "the government is tracking every place you go and your opinions through your cell phone and social networks" is NOT crazy.

Squarel• 12 years ago

Tracking of information has been known about since the initial NSA whistleblowers after 9/11

I have not heard a single conspiracy theorist suggest any of the things disclosed by Snowden, and I have yet to see evidence of RFID chips and the government monitoring us through our microwaves, or that Person of Interest is based on a true story.

Extraordinary claims still require extraordinary evidence, and that evidence usually comes from someone involved, not from a random blog/site on the internet, or a youtube channel.

The one thing everyone misses with this, which is my main concern, is less that the entire NSA/Insert Alphabet Soup here can track me, but that any one individual within these organisations who decides they don't like you for any reason can majorly screw you over. It is not the whole that concerns me, it is the parts.

cryoshon• 12 years ago

For people paying attention to the evidence, the writing has been on the wall for many years-- enough to build credible, defensible, strong suspicions, at least. Of course, these people have been slandered in perpetuity by the government, who stands to gain stability from the "rationalist" suspicions seeming ridiculous. The confirmatory value of these reveals cannot be understated: a subset of the people who have been much maligned as "conspiracy theorists" were in fact reasonable in their assumptions and correct in their hunches.

If these evidence-attenders and conspiracy theorists were maligned as being on the lunatic fringe up until recently and were fully exonerated, what else could they have been right about? This question has the government running scared, scrambling to reassure the public that the now-redeemed former crackpots are still off the rocker, and have nothing else that is relevant to say about things which the public doesn't yet believe.

The public failed these conspiracy theorists because they accepted the establishment friendly line that they were kooky-- the public accepted the false associations and intentional counfoundment made between people with legitimate but not mainstream ideas backed with evidence, and various cultish, indefensible fantasy theories like aliens controlling the government, and the moon landing being faked. Thankfully, the public received a gut shot when Snowden went public with the truth. Maybe next time people will listen a bit closer to voices that the government would love to suppress.

srj2• 12 years ago

Maybe there was an earlier release I missed but this report looks to be the first to show the extent of the call record data being collected. SIM, IMEI, Lat/Lng and more. Logged for every American, every day, all since 2007. And now shared outside of the military context it was gathered and authorized under to civilian agencies.

It's possible the program has been reigned in since these slides were authored but that seems unlikely.

The Surveillance Engine: How the NSA Built Its Own Secret Google

Related Stories