Xkeyscorerules100.txt

http://daserste.ndr.de/panorama/xkeyscorerules100.txt

by peterkelly • 12 years ago

224 79 12 years ago

daserste.ndr.de

v64• 12 years ago

Does anyone have any context or information as to what this is?

anateus• 12 years ago

http://en.wikipedia.org/wiki/XKeyscore

acqq7• 12 years ago

https://news.ycombinator.com/item?id=7983124

The article commented has the whole context.

For us programmers it's interesting, among other bigger issues, to see that the rules contain the pieces of code in C++.

GHFigs8• 12 years ago

The article commented has the whole context.

It really doesn't. In spite of repeatedly claiming things like that people searching for Tor are "monitored" or users are "tracked", it's completely vague about what that those terms actually mean and provides zero examples.

WestCoastJustin• 12 years ago

Additionally, /. has a pretty good summary of what this is [1]. -- "If you search the web for communications security information, or read online tech publications like Linux Journal or BoingBoing, you might be a terrorist. The German publication Das Erste disclosed a crumb of alleged XKeyScore configuration, with the vague suggestion of more source code to come, showing that Tor directory servers and their users, and as usual the interested and their neighbor's dogs due to overcapture, were flagged for closer monitoring. Linux Journal, whose domain is part of a listed selector, has a few choice words on their coveted award. Would it be irresponsible not to speculate further?"

[1] http://yro.slashdot.org/story/14/07/03/1846215/nsa-considers...

tlrobinson2• 12 years ago

Linux Journal seems out of place. They must have had an article about Tails?

aw3c2• 12 years ago

http://www.linuxjournal.com/content/nsa-linux-journal-extrem...

I saw it on the HN homepage some moments ago, I guess the mods hid it for some reason?

MatthiasP• 12 years ago

Yes they did[1], and a link to that article was published on an 'extremist' forum, triggering the inclusion of the domain in the filter.

[1]http://www.linuxjournal.com/content/linux-distro-tales-you-c...

frandroid• 12 years ago

Knowledge is power, and the NSA wants to track those with power.

bostik• 12 years ago

That is also implied in the boingboing.net article[1]. There is one paragraph in particular which should ring a familiar bell...

One expert suggested that the NSA's intention here was to separate the sheep from the goats -- to split the entire population of the Internet into "people who have the technical know-how to be private" and "people who don't" and then capture all the communications from the first group.

And why would that be familiar? First they came for the terrorists, but I said nothing for I was not a terrorist...

[1] http://boingboing.net/2014/07/03/if-you-read-boing-boing-the...

chrismsnz• 12 years ago

Considering that XKeyscore was designed to slurp all the things, all the time, this code actually just looks like a very small, tor-specific subset of simple tagging rules.

One would assume that those tags can be used to later analyse the data to pluck out things like "find users from syria who visited tor/tails websites and is related to xyz".

sp3323• 12 years ago

What language is this?

acqq7• 12 years ago

Obviously the custom language (which uses $ like Bash or Perl) but which allows the programmer to include the pieces of C++. Hints: static std::string and boost!

The Boost, for which Google writes:

http://google-styleguide.googlecode.com/svn/trunk/cppguide.x...

"Cons: Some Boost libraries encourage coding practices which can hamper readability, such as metaprogramming and other advanced template techniques, and an excessively "functional" style of programming.

Decision: In order to maintain a high level of readability for all contributors who might read and maintain code, we only allow an approved subset of Boost features."

Centigonal2• 12 years ago

/* These variables define terms and websites relating to the TAILs (The Amnesic Incognito Live System) software program, a comsec mechanism advocated by extremists on extremist forums. */

That's an interesting definition. It might cover legitimate extremists, but a quick look at Wikipedia tells me that TAILs has also been used by some pretty respectable Pulitzer Prize winners.

These comments provide an interesting and ultimately disheartening insight to how the people designing these surveillance systems view privacy software (and, by extension, privacy?).

opendais4• 12 years ago

I'm sure a legitimate extremist would want to use something like TAILs.

I'm more worried about the fact they are tracking people who access articles about tails by the media than I am about that particular component. If you read one article on linuxjournal containing 'tails' you are flagged:

$TAILS_terms=word('tails' or 'Amnesiac Incognito Live System') and word('linux' or ' USB ' or ' CD ' or 'secure desktop' or ' IRC ' or 'truecrypt' or ' tor '); $TAILS_websites=('tails.boum.org/') or ('linuxjournal.com/content/linux*');

So they aren't really "monitoring extremists" they are monitoring anyone with an interest in security [even if its just a hobby].

GHFigs8• 12 years ago

We have no context for what "flagged" or "monitoring" means, though. There's no indication that these rules are taken to mean anything in isolation beyond what is frankly common sense. The idea that just Googling Tails means you're monitored is a narrative leap.

If I write an email about Viagra, the recipient's spam filter is undoubtedly going to consider it to have a non-zero probability of being spam. That doesn't mean it will be blocked, in the end. Nor does it mean that the spam filter is wrong to take notice.

GHFigs8• 12 years ago

That doesn't make sense to me. It's a definition that relates to the people they're looking for. That it's explicitly more specific than "all Tails users" really doesn't seem disheartening to me.

That Tails is used by people other than extremists doesn't invalidate the comment, by someone interested in "comsec mechanisms used by extremists", that it is, in addition to anything else it may be, a "comsec mechanism advocated by extremists in extremist forums".

It's not like "legitimate extremists" have some totally parallel universe of software that's only used by them. It's fundamental to most of these tools that they'll be used by different people in different ways, and that some of those people will be by some standard or another "bad guys".

Centigonal2• 12 years ago

The description is not exclusive, I'm not arguing that. Your last point makes a lot of sense; most people/groups who do nasty things do it using off-the-shelf components. My issue is with the description painting TAILs in broad strokes as comsec for extremists. Yes, they've got National Security in the name, so they're looking at the software from a national security perspective. There is a wide gap, however, between describing software and its dangerous potential and describing software only in context of its dangerous potential.

If an analyst who hasn't heard of TAILs reads that description, it would sound to them like the program is something that's passed around extremist forums in much the same way malware toolkits are disseminated in warez forums, rather than what it is, which is a Debian fork that routes things through Tor. I say this because that was my first impression, which seemed off, leading me to google, then to wikipedia, and then back here in a huff.

Now, some examples (in order of ascending silliness) of why describing something in the context of one use case is harmful when many use cases exist:

* A lot of people use nmap to explore their home networks or as part of their jobs, potentially in the computer security industry. A lot of crackers also use nmap to case out potential targets. Calling nmap a "network scanning utility advocated by computer hackers" makes illegalizing nmap sound a lot more attractive than it actually would be, even if the statement is true.

* In the real world, certain products are systematically abused for less-than-kosher purposes. Still, we never refer to canned air as a household inhalant without mentioning its dusting use-case first. Potassium nitrate is fertilizer first, rocket fuel second, and only tangentially mentioned as an oxidizer for explosives. Other oxidizers, even the ones that are illegal for consumer sale, are written about the same way.

* Reductio ad absurdum: There's a lot of general purpose software everyone uses. I wouldn't be wrong if I said "Microsoft Word is a text management tool used by terrorist groups to hatch evil plots" or "SMS is a communications technology used by insurgents to detonate bombs" or, extending the idiom, "The Quran is a book used by militant Islamist groups to justify killing and brutalizing civilians." These descriptions are all, however, deeply misleading.

api• 12 years ago

In other news: the NSA developed a DSL with embedded C++. Is this the most horrific revelation yet?

acqq7• 12 years ago

Not compared to the possibly in-house developed object-oriented event-driven C framework for async communication:

http://www.securityweek.com/kaspersky-lab-duqu-framework-lik...

"“It is possible that its authors used an in-house framework to generate intermediary C code, or they used another completely different programming language,” Soumenkov explained.

For reference, Stuxnet was written entirely in Microsoft Visual C++.

The Kaspersky researchers say certain “slices” of code in the Payload DLL may have been initially compiled in separate object files before being linked in a single DLL, but the slice in question is different. “This slice is different from others, because it was not compiled from C++ sources. It contains no references to any standard or user-written C++ functions.”

But there a few things the researchers do know about the mystery code: It’s object-oriented and event driven, and performs its own set of related activities ideal for network applications.

The highly event driven architecture points to code which was designed to be used in variety of conditions, including asynchronous commutations."

For the reference, New York Times, June 2012:

http://www.nytimes.com/2012/06/01/world/middleeast/obama-ord...

"Eventually the beacon would have to “phone home” — literally send a message back to the headquarters of the National Security Agency"

Phlarp• 12 years ago

Would I be wrong to gather that they also built an in house map reduce implementation? What year is this code from? Most of the other documents have been from 2007-2009, when did Google first implement map reduce?

acqq7• 12 years ago

Google's paper from 2004:

http://static.googleusercontent.com/media/research.google.co...

"We wrote the first version of the MapReduce library in February of 2003"

ogijaoijfawje2• 12 years ago

I believe this "source code" is made up, invented for the masses. In fact, the more and more I see of these surveillance reports and reveals, the more I believe this is all purposeful deception, and that while it may be true they're doing all these things, they aren't leaks, but announcements.

On a board where most of us should be familiar with the concept of not trusting user input, I think we should all take a step backward and treat these "leaks" as just that: input from an untrusted source. This could all be a fabrication we're buying into.

mschuster912• 12 years ago

Poor enough you use a throwaway, but I see what you're trying to tell.

I personally believe the NDR/Spiegel to be reputable, trustworthy media. It may be that we have another Hitler's diary scandal there, but the other Snowden/NSA material they have published so far has held up examinations, the NSA iirc even acknowledging some of the docs as actually valid.

ogijaoijfawje2• 12 years ago

I'm not disputing that the NSA stuff is real - of course they're doing it, it's what the agency is expected to do.

I'm disputing whether these are leaks or announcements. The media might not even know they're part of the plan. Is it that farfetched for the NSA to say "Hey, we want to make this information public for some reason, and we're going to do so by using a whistleblower. You're going to release all these documents to a bunch of media sources and live abroad for a while and be hailed as a hero"? There's a long history of leaking information, factual or otherwise, for a number of purposes. There's potentially no end to the rabbit hole. Maybe Snowden doesn't even know he's part of the plan.

It's my opinion that the NSA, some group, or some individual, is letting us know they're watching closely. Maybe they're acclimating us to the idea of being spied on, or worse, distracting us from or preparing us for something else. I think the big picture has yet to be revealed.

err4nt• 12 years ago

A magician (conjurer) uses misdirection of the viewers attention and a strong narrative as his primary tools for deception.

chippy2• 12 years ago

If it is an intentional leak then we should ask ourselves what benefit or what effect does this information have?

For example - would it reduce the number of people using TAILS, or increase? Would it make technologist laugh at incompetence and make the NSA less scary, or would it lead to more intense scrutiny and wariness?

That is what we should be asking, not disregarding the information. We should look at the effects. The nature of the comments in this thread is one of those.

frisco2• 12 years ago

I don't have a good reason to believe that this is real, but if it is the most surprising part to me is the "mapreduce" rule definition in there. As far as I know the only group with a C++ mapreduce implementation called "mapreduce" that also uses protocol buffers (the "proto:" block is protocol buffers) is Google. This seems to say to me that the NSA is using a Google implementation of map reduce. That can't possibly be right, can it?

jmillikin• 12 years ago

They're not using the Google implementation; it looks like they rolled their own.

frisco2• 12 years ago

Your profile says you work at Google... are you saying that the code snippet here is inconsistent with Google's mapreduce?

drunkcatsdgaf• 12 years ago

Interesting video about them using openstack https://www.youtube.com/watch?v=NgahKksMZis

Zaephyr• 12 years ago

While mapreduce isn't my area of expertise, my guess is it's probably easier for them to use mapreduce/ and Hadoop than invent their own wheel in some cases.

georgemcbay• 12 years ago

Well, if the NSA were among the group of organizations with a C++ mapreduce implementation (developed in-house) the code probably wouldn't be on github or otherwise divulged to the public... seems like a rather large leap of logic to assume they are using Google's code.

BSBG• 12 years ago

That's because they're using a derivative of Accumulo (which was developed internally at the Agency).

nickporter• 12 years ago

From the filename, I thought this was some kind of X window config file for key remapping.

jevinskie• 12 years ago

    // START_DEFINITION
    /*
    The fingerprint identifies sessions visiting the Tor Project website from
    non-fvey countries.
    */
    fingerprint('anonymizer/tor/torpoject_visit')=http_host('www.torproject.org')
    and not(xff_cc('US' OR 'GB' OR 'CA' OR 'AU' OR 'NZ'));
    // END_DEFINITION

I was surprised to see that they actually tried to exclude Five Eyes countries. The cynic in me wonders if there is "bug" that neuters the restriction.

sp3323• 12 years ago

I'm wondering if "FVEY" meant something before it was rendered as "five eyes"?

Mandatum• 12 years ago

The "Five Eyes" term has been in use since 2006. Strangely enough the original Wikipedia document archived on the Wayback Machine seems to be unavailable, however we do know it pointed to "USAUK_Community" whose relationship was extended to include other allies which is now known as the "Five Eyes".

Before 2007 the amount of searches for "FVEY" was near zero. The relationship between these five nations wasn't named publicly until 2006 or so, before that the relationship was there - just under another name.

However there are documents dated to 2001 available on mors.org which have FVEY references: http://www.mors.org/UserFiles/file/82nd-Symposium/Form%20712...

These are public so you can assume internal naming was used well before then.

dsl• 12 years ago

FVEY is the accepted shorthand for AUSCANNZUKUS classification, and has been used since the 50's when UKUSA was expanded.

revelation• 12 years ago

Well, we have already learned that their idea of origin country is IP ranges, even if the data clearly indicates otherwise.

That said, they have a variable for directory servers in FVEY countries and one for non-FVEY countries. So I guess they collect everything and the distinction is only here because theres different legal boilerplate for the analysts to follow.

NamTaf• 12 years ago

Of course not; we're just all in bed together so hard that they get direct access from our respective agencies anyway.

Why spy through clandestine means when your buddies hand you their own info anyway?

pkinsky• 12 years ago

What's interesting is that torproject is misspelled in anonymizer/tor/torpoject_visit. I suppose bugs are a universal constant.

mvanveen• 12 years ago

how about merely 50% confidence on whether or not an individual is considered "foreign?"

makomk• 12 years ago

There's a nice bit in here that automatically collates a list of Tor bridge nodes from snooped e-mails. The full list of bridge nodes isn't public, and one of the ways the Tor project attempted to prevent someone from building a complete list was by requiring people to use a valid GMail address to request them, effectively piggy-backing on Google's account verification to stop people from using a swathe of fake accounts to request nodes. Unfortunately, that failed to take account of the fact that the NSA had completely compromised Google's internal network.

sp3323• 12 years ago

Actually it was Britain's GCHQ that tapped Google's datacenter links and shared the data with the NSA. I only mention this to remind peeople that it's more than just one country and agency that's doing this.

mschuster912• 12 years ago

The problem I have with this selector source code is that it is incredibly complex to execute.

How is the NSA able to do this in realtime for all their interception points?!

AlyssaRowan• 12 years ago

XKeyScore runs nearline, not in-line - and it's distributed.

dthunt• 12 years ago

It probably would have been a lot smarter to post this somewhere inside the continental US.

coderholic• 12 years ago

Here are details for all of the IPs in that doc:

  $ curl -s http://daserste.ndr.de/panorama/xkeyscorerules100.txt | grep -Eo "([0-9]+\.?){4}" | xargs -I% curl -s http://ipinfo.io/%
  {
    "ip": "193.23.244.244",
    "hostname": "No Hostname",
    "city": null,
    "region": null,
    "country": "DE",
    "loc": "51.0000,9.0000",
    "org": "AS50472 Chaos Computer Club e.V."
  }{
    "ip": "194.109.206.212",
    "hostname": "tor.dizum.com",
    "city": null,
    "region": null,
    "country": "NL",
    "loc": "52.5000,5.7500",
    "org": "AS3265 XS4ALL Internet BV"
  }{
    "ip": "86.59.21.38",
    "hostname": "No Hostname",
    "city": null,
    "region": null,
    "country": "AT",
    "loc": "47.3333,13.3333",
    "org": "AS3248 Tele2 Telecommunication GmbH"
  }{
    "ip": "213.115.239.118",
    "hostname": "No Hostname",
    "city": null,
    "region": null,
    "country": "SE",
    "loc": "62.0000,15.0000",
    "org": "AS2119 Telenor Norge AS"
  }{
    "ip": "212.112.245.170",
    "hostname": "No Hostname",
    "city": null,
    "region": null,
    "country": "DE",
    "loc": "51.0000,9.0000",
    "org": "AS24900 QSC AG"
  }{
    "ip": "128.31.0.39",
    "hostname": "belegost.csail.mit.edu",
    "city": "Cambridge",
    "region": "Massachusetts",
    "country": "US",
    "loc": "42.3646,-71.1028",
    "org": "AS3 Massachusetts Institute of Technology",
    "postal": "02139"
  }{
    "ip": "216.224.124.114",
    "hostname": "No Hostname",
    "city": "Aptos",
    "region": "California",
    "country": "US",
    "loc": "37.0082,-121.8777",
    "org": "AS40231 Ethr.Net LLC",
    "postal": "95003"
  }{
    "ip": "208.83.223.34",
    "hostname": "No Hostname",
    "city": "San Francisco",
    "region": "California",
    "country": "US",
    "loc": "37.7749,-122.4194",
    "org": "AS40475 Applied Operations, LLC",
    "postal": "94159"
  }{
    "ip": "128.31.0.34",
    "hostname": "moria.csail.mit.edu",
    "city": "Cambridge",
    "region": "Massachusetts",
    "country": "US",
    "loc": "42.3646,-71.1028",
    "org": "AS3 Massachusetts Institute of Technology",
    "postal": "02139"
  }

maximumoverload2• 12 years ago

OK, I am going to play a devil's advocate.

- The job of three-letters agencies is to find out all various enemies of the state. That includes terrorists, but also various gangsters, officials of sort-of-enemy states (like Russia). That's why they exist.

- Following US nation interests has higher priority than rights of citizens, especially in other countries.

- People, that US have a reason to spy on, probably will use Tor/Tails, or at least try to find something about it. It makes sense for NSA to filter those people and focus their spying on them specially.

- Not all Tor interested folks will be evil, but the percentage there will be much higher than in just random internet. So it makes sense to focus on them. Just like it makes sense for a local police to be in a neighbourhood, that's known for a higher criminality.

So I understand why NSA does this, and why do they single out Tor-interested folks.

chroem2• 12 years ago

The job of three-letters agencies is to find out all various enemies of the state.

Within the confines of the law and US constitution.

If all you care about is rooting out enemies of the state, then you're left with organizations like the SS or Stasi.

maximumoverload2• 12 years ago

Does the US constitution matter with respect to non-US entities though?

Why should US-based agency care about rights of German citizens, for example?

chippy2• 12 years ago

The job of a government is towards it's country. A country is also made up of multinational companies and corporations.

Modern say spying is commercial and industrial in nature also... it's not just used to catch bad guys, it's used to get an advantage.

Whilst industrial spying does not trigger emotions as much and infringe their own citizens rights as much, it's arguably an even worse applications of our intelligence agencies.

philip1209• 12 years ago

So the score may not be IPv6 compatible?

xwintermutex• 12 years ago

The file is dissected here: http://blog.erratasec.com/2014/07/reading-xkeyscore-rules-so...

DarkIye• 12 years ago

lol, nobody cares. flails arms in hysteria

andy_ppp• 12 years ago

Presumably reading hacker news also puts you on one of their lists.

Just remember should there ever be a problem between us, we know everything about you.

Xkeyscorerules100.txt

Related Stories