NoSQL Meets Bitcoin and Brings Down Two Exchanges

http://hackingdistributed.com/2014/04/06/another-one-bites-the-dust-flexcoin/

by silenteh • 12 years ago

103 68 12 years ago

bobx11• 12 years ago

This article was only written to promote something called HyperDex which he is in some way involved.

adambard• 12 years ago

TLDR: Exactly what you might have guessed: the exchanges backed their financial operations with a datastore without transactions, tragedy ensues.

This can't be viewed as an indictment of "NoSQL" though; there are plenty of databases featuring ACID transactions that qualify as "NoSQL" (FoundationDB and OrientDB come to mind). That said, none of the popular NoSQL contenders have them, and the new guys with transactions tend to be commercial, so if you need transactions in an free and open-source datastore, just use PostgreSQL.

nemothekid2• 12 years ago

While I do recognize this is a marketing piece for HyperDex, I'm disappointed HyperDex only chooses to compare itself with MongoDB and Cassandra.

It would be different if HyperDex was fully open source and free, but its not. The "transaction" feature is only available with a license, which in most cases will price you out of any developer who is looking to start a bitcoin exchange. So unfortunately it comes across as "If only you had spent $x/mo, buying our product, then you wouldn't have had this problem!"

Now that isn't bad, and its the definition of marketing, but then it goes on to compare itself against Cassandra (which has a widely different use case in the wild) and Mongo (which is already the whipping boy of the database community). However it doesn't help the use cases are also extremely well served by something like Postgres, which doesn't need a Warp addon, and has a huge developer community.

What I would be more interested in, is how does HyperDex compare with the new "NoSQL" databases (FoundationDB, OrientDB, Spanner-like Databases) or even the traditional commercial SQL like Oracle.

In short, it doesn't help to test and compare unsuccessful and unpopular use cases with these technologies. Most competent people, the same people who might pay for HyperDex, already know Cassandra is bad at transactions. What I would like to see is tell me why someone like Netflix, or Ooyala (tried, tested and popular use cases) would move from Cassandra to HyperDex.

threeseed4• 12 years ago

Cassandra's transaction support has been improving pretty rapidly.

http://www.datastax.com/dev/blog/lightweight-transactions-in...

grizzles3• 12 years ago

For what I can only guess are strategic reasons, Datastax is moving away from the schema-less design that made Cassandra popular with early adopters. There are workarounds to this deprecation such as using the old Thrift API, but it doesn't really inspire confidence in the product unless the new schema oriented way of doing things appeals to you. PS those aren't transactions.

JohnTHaller• 12 years ago

It would be different if HyperDex was fully open source and free, but its not. The "transaction" feature is only available with a license, which in most cases will price you out of any developer who is looking to start a bitcoin exchange. So unfortunately it comes across as "If only you had spent $x/mo, buying our product, then you wouldn't have had this problem!"

Just use PostreSQL. Open source. Free. ACID compliant.

teawithcarl• 12 years ago

Cornell CS professor, superb work.

"What may or may not have happened at Mt. Gox"

patio11• 12 years ago

That's because banks employ systems that guard against this kind of elementary error. It's called transactions with ACID guarantees.

Bank systems are often not, in fact, ACID. Think of how many billions of transactions are processed in end-of-the-day batch runs, for example. Overdrafting is not only not impossible, it is a ludicrously profitable feature of the system.

How is this secure? Velocity limits, sophisticated anti-fraud systems, acceptable losses, and, as a bit of a by-the-way, the fact that people who make a habit of defrauding banks generally have a few weeks prior to being woken up by very serious men with guns.

rdtsc2• 12 years ago

It is interesting because the transaction log inconsistencies can often be resolved technically and because of the regulatory framework. Overdrafts can be "solved" in business sense.

Now imagine something like an bitcoin exchange and bitcoin transaction log. Transaction errors cannot be fixed easily. So overdrafting an account means never getting the money back. There a single account would need to have ACID properties in way.

It seems ability to deal with and tolerate data conflicts and inconsistencies is different for each business. There was some interest in CRTDs (data types that can automatically converge after a conflict occurs, like imagine a max() function or a set union operation) but still means shortly there could be an inconsistency in the system and sometimes that can be exploited and is just not an option.

Karellen• 12 years ago

Bank systems are ACID. They're often not immediate, using batch runs as you say, but the databases the batch runs work on are definitely ACID.

The problem is not necessarily with overdrafts, as you suggest, but - as the article says - with race conditions between multiple readers/writers causing two near-simultaneous withdrawls to only make a single debit to the source account, while crediting both destination accounts.

Note that this could lead to unexpected overdrafts being noticed later, if the exchange also wrote a consistent log of all transactions, and attempted to reconcile the log with the real-time balances periodically (e.g. on a daily basis), but it's not clear if they even did that. (Although I might be missing something)

mbreese• 12 years ago

That's why banks have an available balance and a posted balance. One is the best available estimate of how much money you can spend. The other is the reconciled balance after transactions have been posted.

It would have been possible for the exchanges to have used a NoSQL db with this strategy, but they didn't. They should have been keeping transaction logs and reconciling balances, but they didn't.

While, I'm no fan of MongoDB, it isn't really their fault. These exchanges were designed by people who didn't know any better.

pron• 12 years ago

If I'm not mistaken, an eventually consistent database must also guarantee, well, eventual consistency.

mriou• 12 years ago

Banks are made of a mess of different systems. Some are locally ACID, but overall reconciliation and compensation are what keeps it all together.

Regardless, this has little to do with banking. The systems are too widely different to be compared productively. The only conclusion you can get from this (marketing) story is that when you have an eventually consistent system, you will need occasional distributed locking. Cassandra often comes in pair with ZooKeeper.

The other conclusion is that no matter how trivial they sound, distributed transactional systems are far from trivial.

jeffdavis• 12 years ago

ACID would have solved this problem though, right?

ACID is simpler to reason about. If there is a strong justification for abandoning it, and enough expertise to still make the system work, then fine. But I doubt they had either the justification (I assume their transaction volume could be easily handled with a SQL system) or the expertise.

I'm bothered that MongoDB strives to make people feel comfortable abandoning ACID who clearly don't understand what they are doing.

aardvark179• 12 years ago

"Historically, Bitcoin exchanges that suffered significant losses turned into fractional reserve banks, only to fold later."

That is not what fractional reserve banking is and I wish bitcoin articles would stop misusing the term.

mikeash• 12 years ago

The continued misuse of that term seems telling. If they really think this is how normal banks work, it helps explain why they set up these bitcoin systems in such crazy ways.

rdtsc2• 12 years ago

Please don't be MongoDB, please don't be MongoDB

...

Sigh

a failure of distributed systems academics to educate developers and to equip them with clear-thinking frameworks.

Here is the thing, distributed systems are not standard academic courses. Just like networking they are often optional. That is really sad and it needs to change.

The CAP theorem is not something fringe and exotic anymore it should be standard taught to everyone.

MongoDB has really messed up. They promoted and marketed a shitty product wrapped in nice marketing. It created 2 things -- low barrier to entry and a trap. Both wrapped in one nice shiny package. "You should have read the docs to enable actually writing data disk when the user asks them to" doesn't jive with me. A really shitty default setup for something calling itself a database.

Oh well, they reaped the benefits of their strategy, both the short term (success) and long term (failure) - it became the butt of jokes and ridicule.

threeseed4• 12 years ago

Have you actually used the database since if you did you would know that the write concern was set to FSYNC_SAFE by every single driver (before it was the default) and given the lack of configurability should have been known by every developer.

It never ceases to amaze me how few technical people actually "get it" even for products designed for themselves. Many, many people like things to be simple and easy. MongoDB fits this criteria perfectly and no matter how many times you call it a shitty product isn't going to dampen its popularity.

zorlem• 12 years ago

I, too, don't like the way MongoDB works and is marketed, but I would not bet too strongly on them failing. Only 10 years or so ago MySQL did pretty much the same, but they managed to succeed, despite there being better alternatives at the time.

Even today, after all the development effort put into MySQL and its derivatives, there are still better and more capable databases, but that doesn't prevent MySQL from being one of the top three RDBMS-es (and we can bet it is the most popular FOSS RDBMS).

ChuckMcM• 12 years ago

Its unfortunate to use the ATM example because just such an exploit was used by carders [1] to get more money out of accounts than the accounts actually had.

[1] http://time.com/48344/hackers-target-atms-for-unlimited-with...

edwintorok• 12 years ago

ATMs aren't as smart as the article says either: "the real code will check to see if there are sufficient funds".

I noticed that (some?) ATMs around here tend to implement this flawed algorithm, paraphrasing the article:

  0. database.begin()
  1. mybalance = database.read("account-number")
  2. newbalance = mybalance - amount
  3. database.write("account-number", newbalance)
  4. database.commit()
  5. dispense_cash(amount)

At first sight this might seem ACID, but its not. Consider what happens if the ATM is out of cash: step#5 fails, but the money was already withdrawn from your account!

It doesn't automatically revert the transaction either: you have to go to a branch office and ask for the transaction to be reverted. So much for atomic transactions at ATMs.

dnmmnd2• 12 years ago

Whilst that maybe the way your bank has chosen to operate, it is most certainly possible for ATMs to report a step#5 failure to the banks host which will reverse the transaction.

ams6110• 12 years ago

Wow I've never seen this happen. What happened to step -1:

  if (cash_in_machine < amount) abort

jasomill• 12 years ago

Interesting. In one particularly ADHD-inspired case, I've also seen

  5. Dispense cash
  6. Wait a minute or so
  7. Retract cash

where 7 did not lead to a credit on my account until several days after I reported the problem to my bank. That's probably by design, however, since otherwise the machine would require automated defenses against surreptitious replacement of dispensed bills with counterfeits.

Your example could be a similar failsafe against cases where the machine fails to properly dispense cash in a manner that nevertheless allows the cash to be retrieved. Otherwise you have to carefully separate errors that can't possibly lead to cash being dispensed (e.g., hopper empty) vs. those that might (e.g., bills jammed somewhere in the dispenser mechanism).

hibikir• 12 years ago

The first problem to be had when considering distributed systems is the fact that there is no such thing as a general purpose distributed system.

Some problems require doing 50x more reads than writes. Others have more writes than reads. Sometimes losing a few inserts is not a big deal, but a small delay is terrible. Other systems just can't tolerate data loss, and would pay any price to keep it that way. You might have a lot of small pieces of data, searchable in very simple ways, or you might have multi GB documents that require so much indexing than Solr and Elastic seem inadequate.

In the old days, you just bought a bigger box for the RDBMS, and you just were covered by knowing one tech. Now we need to go through a wide variety of tools that will rarely do everything you want, and have to write a bunch of code to compensate for the limitations of the tools. And then your DB of choice decides that they will only support their self hosted product, then they get bought by IBM, and you wonder if you'll be forced to retool your entire backend (Hello Cloudant!)

Until there is some clarity in the market, and we get to see distributed stacks that are general purpose, we'll see issues like this popping up all the time, as there is a lack of people that both have good knowledge of all the available options and the skill to put them together into something that will solve your specific problem.

camus2• 12 years ago

Question!

Do transactional NoSQL(others than HyperDex) databases exist? or do transactions make horizontal scaling difficult?

nacs• 12 years ago

Sounds like the author of this 'article' just wanted to make a thinly veiled plug for the "HyperDex" product.

Blaming all of this on MongoDB is silly. You can do the kind of idiocy displayed by these exchanges even on generic DBs by doing things like not using transactions and/or not waiting for write confirmation.

The root problem is in the (lack of) design and programming of these systems not their data store.

tlarkworthy• 12 years ago

you can put ACID like transactions on top of NoSQL databases, I did it here, but it took a static analyser to help get the logic right:

https://github.com/tomlarkworthy/firesafe/wiki/Send-Item

rubyfan• 12 years ago

This article lacks a fact base on what actually happened and seems to totally rely on conjecture in order to compare Hyperdex against leading NoSQL solutions.

dhendo• 12 years ago

Mongo does have document-level atomic update operators, such as $inc: http://docs.mongodb.org/manual/reference/operator/update/inc...

thom_nic• 12 years ago

Yes this was completely ignored by the article. This along with a 'where' clause can guarantee the operation will only succeed iff there is sufficient funds. I don't know if other NoSQL DBs offer these sort of atomic operators.

Which makes this statement seem particularly unfair: "The problem here stemmed from the broken-by-design interface and semantics offered by MongoDB." He seems to call out Mongo by name a couple times but doesn't say that the exchange was actually using Mongo.

danbruc• 12 years ago

It's the same story over and over again. The young wild guys arrive, full of energy and ambition to make the world a better place. They throw all the ancient stuff overboard because it seems ugly and useless. Then they painfully rediscover why the ugly stuff is there and not useless at all. And after a lot of pain they finally arrive at a slightly improved version of what they threw overboard long ago, at least if they did not surrender before.

PhantomGremlin• 12 years ago

The young wild guys arrive

That's exactly the problem. The 20-something hipsters who start these companies don't yet have the life experience to realize how ignorant they are. So of course they don't pay any attention to the lessons of history. And so they, very painfully, re-learn them.

If the hipsters had some humility they would have also hired a few grizzled, experienced old fogies.

Fortunately not all startups assume that anyone over the age of 30 isn't worth hiring. For a non-technical example, look at how Google hired Eric Schmidt to provide adult supervision.

jokoon• 12 years ago

What are the advantages of nosql btw ?

threeseed4• 12 years ago

Basically developers came for the scalability and stayed because they are so easy to use/manage.

nikatwork• 12 years ago

NoSQL is fantastic for backing item-orientated webapps, it removes the transactional and relational overheads. Conversely I would not use NoSQL for anything concerning FIRE or complex inter-related data - those safeguards become important.

Scalability is something of a red herring with NoSQL. Yes it's easier to scale, but my primary criteria for choosing it would be the nature of the data, not the intended scale.

Consultant32452• 12 years ago

A more "use-case" example than your other responses.. I believe Amazon uses a NoSQL database for their product data. It's tons of reads and relatively few writes. They don't give a crap about transaction quality for product descriptions, comments, etc. However, when you get to the checkout page and they start doing things like taking your money and updating their inventory... that's and ACID compliant database. When you're dealing with money you want transactions, rollbacks, etc.

late2part• 12 years ago

But, it is webscale!

bkirwi• 12 years ago

A bit irritated at how the author complains about other databases astroturfing on Hacker News, then writes a glowing post about HyperDex without mentioning that he's one of the authors.

That said, I agree with the thesis that current-generation distributed systems provide a terrible API for users who actually want to get concurrency right. I'm optimistic about projects like Summingbird[0] and Bloom[1], which run very high-level declarative programs with simple consistency guarantees.

[0] https://github.com/twitter/summingbird [1] http://www.bloom-lang.net/

atmosx2• 12 years ago

It points to a social failure: a failure of distributed systems academics to educate developers and to equip them with clear-thinking frameworks.

So, will the OP do an exchange and show us how it's done using his database of and approach of choice. Because writing articles on other's failures is kinda of... Easy.

I'm a little sick of this attitude. Nothing personal with - what I thought was Sinan Eren at first, confusing the names after so many years - the OP here, but given the fact that he is deliberately calling out amateurs and poor designers people who (apparently) deserved it, he might very well create a secure bitcoin exchange in the real world, using his technologies.. There is a huge market for that, today more than ever.

sillysaurus3• 12 years ago

The attitude is warranted when they're losing other people's money.

atmosx2• 12 years ago

Totally get that, but his approach seems a little snooty to me and runs throughout the article that's why I had this reaction/comment.

DigitalSea• 12 years ago

As soon as I saw the term NoSQL and "Brings Down" in the title of the submission, I knew it was going to be MongoDB. Firstly, I don't understand the hatred towards MongoDB. Like most NoSQL datastores, it doesn't offer things like transactions and ACID compliance: this is well-known thing. There are some things that MongoDB does very well, currently I am using it to store clicks for an internal application which then displays those clicks on a page overlay, it handles the task very well. I would never use it for

MongoDB is not to blame for bringing down the exchanges that decided to use it, the stupidity of the developers behind said exchanges are the sole reason. You can open up a can of tinned pineapples with an axe, but that doesn't make it the right tool for the job.

This post was one big marketing piece for HyperDex and while it is touted as open source, but that isn't entirely true. If you want support for transactions, you have to by a licence. So basically what this post was saying is, "If you spent X amount of money on a HyperDex licence, you wouldn't have had these issues" most people that start these exchanges start them from nothing and probably can't afford or wouldn't warrant spending money on a licence.

If you replaced MongoDB with pretty much every other NoSQL database out there, the result would have been the same. Mongo just has a better marketing team behind them in the form of 10Gen getting people to buy into the hype. However, as far as I am aware of, 10Gen aren't telling people to use MongoDB for exchange platforms and anything involving transactions (I could be wrong though). This is a classic case of seeing people talking about how NoSQL is the future and others getting caught up in the hype.

If you want a database that you can rely on, especially when dealing with transactions: use something like Postgresql, not MongoDB and or insert over-hyped NoSQL database here

hackerboos• 12 years ago

it doesn't offer things like transactions and ACID compliance: this is well-known thing

Evidently not. Which is why people are still using it for financial based systems.

kalleboo• 12 years ago

the stupidity of the developers behind said exchanges are the sole reason

Exactly. The fact that they picked MongoDB for this application in the first place tells me that even if they used an ACID-compliant database, they still wouldn't have known to use transactions, and would be vulnerable to the same attack.

kylemaxwell• 12 years ago

Not quite understanding the focus on NoSQL for this use case set. I love MongoDB in the right place and Redis in others. But when you have a bunch of requirements that lend themselves to traditional DBs, choosing MongoDB or HyperDex feels like choosing a tech because it's cool rather than because it is the proper engineering choice.

cranklin• 12 years ago

"They did what anyone would do after reading one too many astroturf articles on Hacker News. Sure, their system failed, but in a sense, the overall system failed them."

The technology didn't fail them. On the contrary, the technology did exactly what it was supposed to do. The devs could have (should have) considered these race condition scenarios.

ibyon• 12 years ago

I don't think that MongoDB or any nosql is right choice for financial application. You can build for yelp but not for banking.

platform• 12 years ago

Marklogic supports ACID transactions. being that it is also a horizontally scalable system, it allows those transactions not just to spawn multiple objects, but also across multiple hosts. http://www.marklogic.com/blog/can-you-pass-the-acid-test/

They also support data tiering into HDFS, and now most recently RDF with SPARQL

They have been doing ACID for a number of years, not sure for how long.

They are not free (unless your data set is small)

kofejnik• 12 years ago

Why wouldn't they use a plain old ACID database? Was their load something that Oracle couldn't handle?

NoSQL Meets Bitcoin and Brings Down Two Exchanges

Related Stories