NSA built a NoSQL database

http://wiki.apache.org/incubator/AccumuloProposal

by rschildmeijer • 15 years ago

139 55 15 years ago

wiki.apache.org

nirvana• 15 years ago

It seems that the tags for cells seems to be an important feature of this database, and they also mention it is appropriate for places where "privacy is important". Can someone explain the connection between these two? If I'm understanding right, the labeling makes it easy to address individual cells, but I'm not sure how that enhances privacy.

aaronharnly• 15 years ago

They seem to be access tags, not just arbitrary tags. I interpret it as item-level ACLs, something like row-level security in SQL databases.

cookiecaper• 15 years ago

I would imagine that this is similar to other ACL products in which the NSA has previously expressed interest, like SELinux. The "labeling" probably means setting permission levels.

mjs• 15 years ago

"There is a risk that Accumulo will be criticized for not providing adequate security. The access labels in Accumulo do not in themselves provide a complete security solution, but are a mechanism for labeling each piece of data with the authorizations that are necessary to see it."

jimueller• 15 years ago

The labeling likely refers to Mandatory Access Control (MAC) where objects (data, cells) are assigned classification labels (e.g. Top Secret, Confidential) and subjects (users, processes) can only access objects that match the subject's assigned classification level.

emehrkay• 15 years ago

    svn co https://svn.apache.org/repos/asf/incubator/accumulo

doesn't seem to work

zokier• 15 years ago

It's just a proposal atm. The svn-repo is one of the items they are requesting from ASF

jimbokun• 15 years ago

" The core codebase consists of 200,000 lines of code (mainly Java) and 100s of pages of documentation."

100s of pages of documentation is a promising start for any open source project.

jrockway• 15 years ago

It depends on what the documentation is. If it's 100 pages of "AbstractClassFactoryClassFactoryFactory is a class that builds AbstractClassFactoryClassFactory objects", then that's useless.

Also explains why it's 200,000 lines of code, for something that should be an order of magnitude smaller.

contol-m• 15 years ago

Why do you say that has to be order of magnitude smaller? Other BigTable clones like HBase are atleast 100K lines of code, if not more.

Garbage• 15 years ago

Another Java hater? :s

brainlounge• 15 years ago

200k lines ... of which 85k lines are generated from Thrift and 10% from other projects, according to related ML discussions.

mypov2• 15 years ago

NSA uses Java. Great to know. Apparently the professionals at the NSA are not swayed by fashion.

jackowayed2• 15 years ago

They use more "fashionable" languages too. Just not for distributed databases. Most popular open source databases are also written in C/C++/Java.

vegai2• 15 years ago

That's a ridiculous defense: they were obviously swayed by fashion at some point. It just happened several years ago.

spullara• 15 years ago

Government & consulting projects generally require the code and the use of the code to be fully documented. I'm actually looking forward to that part of it more than the software if it is any good.

vegai2• 15 years ago

200k. Why do Java programs always balloon up?

gaius• 15 years ago

Zawinski's Law.

va_coder• 15 years ago

It sounds like Oracle's Label security applied to Hadoop.

It'll be a challenge because Label security slows things down quite a bit.

zedshaw7• 15 years ago

Prediction: There will be a backdoor.

dotBen• 15 years ago

C'mon Zed, really?

a) The code will be open source - the community can verify the code for anything untoward

b) Given the nature of the product, most implementations are going to be behind a firewall anyway, with the storage layer talking to business logic. Even if there was a backdoor, and I'm sure there isn't, not sure how NSA could get in.

Do you think there's a backdoor in NSA's open-source algorithm for SHA-1 too?

I applaud the government for putting tax dollars back into open source. My only gripe is the lack of transparency as to what this is primarily used for within the NSA (to be expected I guess). I generally like to know what I'm helping commit code to go do - although granted you have no idea what other open source projects are used for regardless of whether the lead sponsor is government or private company.

seabee• 15 years ago

If there are plenty of good uses for the code, I'd still want to improve it, even if I find out it's used by the Kitten Krusher 3000.

Unless a "please don't use this code for evil" license is legally binding, that's just the nature of open source.

pipeline_tux2• 15 years ago

I don't necessarily think there will be one, but I wouldn't be surprised either.

Security flaws can be extremely subtle and 200,000 lines of code is a lot to review... Given that there's plausible deniability (we didn't do it intentionally, it was a genuine bug!), if you were them, wouldn't it at least cross your mind to try it?

Also, at some point, if it becomes popular, some sysadmin at a large foreign government agency or company will forget to firewall off a box running it (ignoring that they could also be connecting back directly - automatic updates anyone?)

adnam• 15 years ago

I think it was just a joke, chill :)

taftster• 15 years ago

"My only gripe is the lack of transparency as to what this is primarily used for within the NSA (to be expected I guess)."

It's likely just used exactly how you think it would be; to hold massive amounts of key/value data. No doubt, the NSA likely has tons of data to work with. A NoSQL approach would be seemingly beneficial for this use case.

bane• 15 years ago

Warning Zed! This is a humor free zone.

zedshaw7• 15 years ago

Damn, you'd think with 200k lines of awesome Java that needs to be documented with a manual that's hundreds of pages long that uses 3 other massive Java projects and released by a government agency that's done backdoors in everything from crypto systems, operating systems, to even backdoors themselves, that there'd be at least a plausibility of them putting one in.

popcornchicken• 15 years ago

Awesome NSA...

So NoSQL approach makes all those skiddies SQLi attacks moot.

Still 200k lines of code = ~2000 bugs...

So, opening it to the public will expose (some) of those, and fixes will be created. and Now, when are you going to show off that really kool advanced A.I. you guys are sitting on!

samstave• 15 years ago

Some time ago I met some people from the .gov cyber security, NSA and other offices. The head of the .gov office on cyber security was really nice and invited me to go to dinner with them.

The guy from the NSA was hands down the biggest evil piece of shit I have ever experienced in my life. The way he talked, what he said, and the fact that he was given free reign to commit crimes in his training, which he openly bragged about, made me want to murder the guy right then and there.

I lost any and all respect for what the government and the NSA do.

awj• 15 years ago

...because of one guy, who may have been lying through his teeth in an attempt to impress the other people at dinner?

I'm not trying to defend this guy, certainly it's a big problem if he did what he said, but you can't fix something when you paint the whole fucking thing with a brush you picked up looking at one of the uglier parts.

aedocw• 15 years ago

Though I'm not defending the NSA, you should be careful not to make broad assumptions based on a single data point.

http://lesswrong.com/lw/dr/generalizing_from_one_example/

PLejeck• 15 years ago

Now we know how they store the data gleaned from wiretaps!

Create2• 15 years ago

Given that the charter of given agency is certainly not to produce FLOSS, and most certainly not for the pleasure of a foundation which has its worst adversaries as founders (hint: Ben Laurie).

It would be most plausible to have direct access to the build infrastructure, which in turn would give access to ... without the hoops of going through Oracle and IBM or whatever corporate projects.

And if you read the spiegel article (which has to do) with Ben's past-present, it is clear, that the USA is on the "offensive". The surest way to discredit any anonymity provider for whistle-blowers is to discredit the providers. Which has just happened in the last few days (note, that the contents of the 7z itself was already past 0-day, and therefore valueless, as a USA Official noted in the article).

NSA built a NoSQL database

Related Stories