Dear NoSQL: "SQL-isn't-scalable" is a lie

http://www.yafla.com/dforbes/Getting_Real_about_NoSQL_and_the_SQL_Isnt_Scalable_Lie/

by portman • 16 years ago

138 102 16 years ago

yafla.com

jbellis4• 16 years ago

tl;dr: "sql does so scale! if you throw $millions of hardware at it." yawn.

mmt3• 16 years ago

The article points out that it has already scaled with hundres of kilobucks or even millions.

However, it also points out much lower-end hardware solutions that cost under $10k but perform much better than the largest EC2 instance, for I/O.

ETA: This is why I tend to roll my eyes at the notion of "commodity" hardware. The article's low-end array is 400MB/s, but rolling ones own can yield over twice that for the same or lower price tag. All this well before reaching the unscalable cliff of enterprise pricing.

illumin82• 16 years ago

Yes, but there's a big difference between a 400MB/sec. supported storage array for < $10K, and a garage built "roll your own" storage array that can do 800MB/sec. for half the price.

The main difference is that, even in startup companies, you have a supported solution and can call someone at 2 am when your storage dies and expect to find replacement parts and support. Good luck trying to drive to Fry's and buy replacement hard drives for a server that was built 2 years ago by someone that no longer works there.

kennu• 16 years ago

I don't think NoSQL people usually claim that SQL isn't scalable, just that it's unnecessarily complicated to scale.

You generally have to partition your data horizontally and thus give up many of the features that SQL has to offer: ACID transactions, unique keys, auto-increment primary keys, etc.

Then you have to come up with your own solutions to replace those features: eventual consistency, UUID keys, map/reduce, etc. And these happen to be exactly the kind of features that many NoSQL databases can give you out-of-the-box.

wmf10• 16 years ago

You generally have to partition your data horizontally and thus give up many of the features that SQL has to offer

There are plenty of databases that will partition data without giving up any SQL features, but they cost money.

barrkel2• 16 years ago

Consistency, Availability, Partition tolerance. NoSQL stores usually sacrifice consistency, and instead settle for eventual consistency. SQL (i.e. RDBMS) stores, with their usual emphasis on transactions, must necessarily sacrifice something else. SQL that doesn't hew to a hard line on consistency and transactions doesn't really have all the features of SQL. This is the distinction that matters most, IMHO, in the NoSQL strand of thinking.

freetard• 16 years ago

Except for postgresql.

jbellis4• 16 years ago

> There are plenty of databases that will partition data without giving up any SQL features, but they cost money.

They also either rely on a single huge SAN for storage (single point of failure + expensive as hell) like Oracle RAC, or they require specialized gear like infiniband to reduce intra-node latency like Exadata (starting price: seven figures) or they're analytics databases that are designed for huge queries with latencies to match like Vertica, ParAccel, etc. (Think minutes between data being loaded and being available to query.)

I'll take NoSQL, thanks.

tpz• 16 years ago

I'm not even so sure that they are unnecessarily complicated to scale, or that you should need or have to replace the features you list. I am sure, however, that everything you list ends up needing to happen if the solution chosen doesn't apply well to the problem at hand.

When you really dig deep down into each and every article on this subject, whether for or against NOSQL, the most important (and yet unstated) fact is this:

It isn't that RDBMS systems scale or don't scale, or that NOSQL systems scale or don't scale, it's that any solution which prioritizes (just for example) consistency and availability is not going to scale effectively for a problem that instead prioritizes availability and partition tolerance.

I am willing to bet that any time a problem and a given solution don't align on the two attributes they've respectively prioritized from CAP, there will be a claim that the solution "doesn't scale". The reality is just that the solution wasn't applicable to the problem at hand. If instead one evaluates solutions which match the problem's CAP priorities, the solutions will scale effectively (modulo their individual pros and cons relative to the other options within the evaluated CAP-priority-matching set of possible solutions, of course.)

viraptor2• 16 years ago

Reading this article was a bit annoying. Right from the start: "I work in the financial industry [...] I worked in the insurance, telecommunication, and power generation industries." I was thinking only - you're not even supposed to look at nosql from that perspective - there's nothing for you there... just go away.

It doesn't support transactions and acid most of the time. There's no "we pay $xxxM for the support and blame you for everything" company in nosql products. It's not the same workload as you'd expect from kv/document-store used as a webpage backend.

One of the few serious "nosql" databases for enterprises like that is Berkeley DB - it's got what they need. I'm not sure why did he write that blog post... it just stated the obvious, but in form of a rant.

The funny thing though is - Berkeley DB is exactly what NoSQL is about... and it is used for local reliable storage in many big enterprises. Replication, logging, transactions, etc. - and it's just a kv-store really.

wvenable4• 16 years ago

I'm not sure why, these days, anyone would choose Berkeley DB over something like sqlite.

viraptor2• 16 years ago

BDB has proper transactions, provides the same type-safety as sqlite (i.e. none), doesn't have to go through abstractions like JDBC (overkill for sqlite), is backed by Oracle, has pure java implementation, has replication.

Sqlite has columns and can search based on them. It can also save you a couple of lines on manual joins. Does it provide something more?

(although tbh, I'd take Tokyo Tyrant over both of them - has columns, writer lock + server-side scripts instead of transactions, same model of replication as BDB)

evgen7• 16 years ago

My favorite part of the rant is the suggestion that the nosql alternative to spending millions on a ginormous RDBMS was throwing away throughput by using Amazon AWS. I guess any argument becomes easier to make when you define the other side by its incompetence.

The only valid point made seems to be that vertical scaling of a RDBMS can be a multi-million dollar exercise.

wmf10• 16 years ago

Unfortunately, what you call incompetence a lot of other people call best practice.

evgen7• 16 years ago

If throughput and transaction/analysis speed is a requirement then AWS is not the answer -- anyone who suggests otherwise has never used it for anything larger than a toy dataset. I am currently migrating a large data analysis system (20+ TB running over about 500 EC2 hadoop workers) to a dedicated cluster because the internal EC2 latency reached a tipping point in our analysis runs. If you have the dough to spend on a big vertical system you can spend half as much on a dedicated cluster running a NoSQL solution and probably meet the required spec. The original article was comparing a Porsche Cayenne to a bicycle; the bicycle works for some problems and in certain cases it (or a fleet of them) can solve the problem better, but it was a dishonest strawman comparison for the subject at hand.

JunkDNA• 16 years ago

This is a very reasoned, well-written article. While I like the idea that the NoSQL movement is questioning some core database assumptions, I have often been uncomfortable with the meme that "SQL doesn't scale". Scalability is always a function of your circumstance. This article makes a good point that the group of people for which SQL doesn't scale need another option (internet startups with little cash and massive server load). While that case gets a lot of attention around here, it's important that people not try to extrapolate that experience to other areas of IT.

evgen7• 16 years ago

What the NoSQL "movement" is doing is questioning some basic axioms of architecture and operations that may no longer hold in the modern era. There are many aspects to data, how it is used, how it should be stored & accessed, and what is or it not important within the set of available data. The RDBMS cabal set a particular set of standards (ACID) that were a good match for a lot of the early "big data" systems and the computing infrastructure of the time, but times change. As more people are bringing new data sets, new uses and access patterns for this data, and the computing infrastructure is changing from large servers to swarms of small systems the options for storing and accessing this data are also changing. I don't know many in the NoSQL crowd who are actually opposed to a RDBMS as a solution, but it is not necessarily the first choice for any problem; most of the people in this group seem to start by considering the data and the application and then picking and choosing the characteristics of the data system that will be used to meet these needs. The massive and highly distributed systems get a lot of attention (and they are almost always NoSQL systems) but that does not mean that alternative data systems do not have a place up and down the IT stack, nor does it mean that a RDBMS is not a good option for some situations.

seunosewa• 16 years ago

"The RDBMS cabal set a particular set of standards (ACID)" What's the problem really? ACID or SQL? They are mutually exclusive. You can have ACID without SQL (BDB) or SQL without ACID (MySQL 3.X). Which of the two is the problem?

WorkerBee• 16 years ago

The RDBMS cabal set a particular set of standards (ACID) that were a good match for a lot of the early "big data" systems and the computing infrastructure of the time, but times change

Times change, but in the sense of adding additional systems with new requirements.

The existing systems have not gone away, and are not going anywhere. In fact there are more than ever of them around. I certainly still want my bank to run a system that has ACID transactions. Likewise my mobile phone's billing system, utilities bill, etc.

rbranson3• 16 years ago

The real crux of the issue isn't that SQL won't scale, it's that it won't scale efficiently or effectively. SQL scales, but you need expensive, exotic hardware (InfiniBand switches, SAN storage) that requires niche engineering skills just to get up and going. By the time a startup-type company purchases all of this hardware and has figured out how to use it, the window of opportunity will have closed. In addition, you're then pretty much locked into support from the vendor for performance issues. With solutions like Cassandra, you've got the source and can solve your own problems, which, to a company rich in software engineering talent, is exactly what they want.

Even if you look past all that, RAC and ASE will run into limits as they're still relational, which is not a truly scalable approach to data modeling. There are real limits to their scalability as well. Oracle RAC will only scale to 100 nodes. That's a lot of dough for a 100 node limit.

So there it is. SQL doesn't scale (effectively or efficiently).

wavesplash• 16 years ago

"So there it is. SQL doesn't scale (effectively or efficiently)"

I think you just re-iterated the author's reason for writing perfectly.

Instead of saying 'SQL' (the query language) doesn't scale, perhaps you can say full ACID doesn't scale cheaply.

We spent a bunch of time looking at large scale distribution at Berkeley for the Mariposa project. The short version: if you want to break ACID then you have to take the semantics of the data into consideration. Transactional financial data needs full ACID, many other types of data have looser needs (status updates, etc) and those looser needs can be taken advantage of.

As it turns out, even common DB's like Oracle, Postgres and MySQL can be used differently to get much cheaper scale for certain types of data. See the FriendFeed article on MySQL as a docdb (also the Friendly ruby mapping on github).

To paraphrase the author:

"ACID scales just fine if you need it (and if you need it you most likely have budget for it). If you don't need full ACID, there are plenty of other cost effective solutions you can run on low bandwidth disks/memory limited CPUs. Mix and match to fit your needs.

Please, enough with the religion."

scythe2• 16 years ago

> So there it is. SQL doesn't scale (effectively or efficiently).

You might just want to say "cheaply".

Snark7• 16 years ago

This is simply not true. I work for a major internet company that has millions of users. We use SQL for all of our data (images and SWFs excepted, of course). We do all logging to SQL tables. We use key-value cache servers for hot, non-relational data, but we don't have a lot of that.

We built most of our data center with cheap hardware. It is quite possible that we splurged on a load balancer and some other funky networking gadgets; I don't know. We did, however build quite a lot of custom software to abstract away the fact that we have split our data among dozens of machines. This is essentially the work of two people (one hardware, one software). Finding the people who can build such a system is difficult, to put it mildly, but once you find them, you don't need deep pockets.

Reddit is another example of a site that runs on SQL. Part of the problem with the "SQL won't scale" argument is that essentially all of the live queries function like key-value lookups (or lookups against a very small number of rows). If you're tempted to do something insane like service a request with a join, then you're doing it wrong and you need to cache.

Lookups by index will have the same performance characteristics no matter what. It doesn't matter if you're using a Python dict, an index in MySQL, or a key-value store.

THEY ARE ALL O(1).

est• 16 years ago

maybe SQL scales, ACID probably don't

donw2• 16 years ago

I would wager that SQL, and ACID, will scale larger than many startups will ever actually need.

megaduck• 16 years ago

True, but my sincere hope is to outgrow Postgres someday.

lucifer3• 16 years ago

I think the problem is that in a startup, the leads are typical code slingers who are very good with programming, competent with SQL, and manage basic DBA (hand held by documentation).

Even if they are that rare startup that has heavy weights on both sides of the data/process divide, the second issue is that intelligent schema design (imo) does not fit the agile model quite that well.

Its really more of a economic issue than a technological one: it is quite possible that the poorly designed and implemented SQL - see above - solution can not scale. Obviously the problem is not SQL or RDBMs. But when you have a very smart, but young and not yet seasoned, tech team who hasn't seen it all, do you think they will blame themselves or lament that "SQL can't scale"?

wooster3• 16 years ago

"Such a platform can yield very satisfactory performance for tens or hundreds of thousands of active users"

There are 253 million Internet users in China alone. What happens when your site needs to scale from 0.001% of them using the site simultaneously to needing to scale to 1% of them using the site simultaneously? Within a month?

"Of course if you index poorly or create some horrendous joins"

Which in the Twitter and Facebook cases is exactly what they have to do on many of their requests. As I've personally found out, relying on a database to do a join across a social network graph is a recipe for disaster. One day you'll be woken up because your database's query planner decided to switch from using a hash join to doing a full table scan against 10s of millions of users on every request. Then, you'll be left either trying to tweak the query plan back to working order, or actually doing what you should've done in the first place: architect around message queues and custom daemons more suited to your query load.

"Even with billions upon billions of help tickets."

At 50 million tweets a day, Twitter would hit 18 billion tweets within a year. Good luck architecting a database system to handle that kind of load. That is, one in which the database system is serving all of the requests (including Twitter streams) and isn't just being used for data warehousing.

"Such a solution — even on a stodgy old RDBMS — is scalable far beyond any real world need"

The disconnect here is this guy's needs are not the needs of a lot of us who are actively looking at alternatives. He is simply not familiar with the problem domain.

stuhood• 16 years ago

This is a perfect response: thank you.

Also, the definition of scalability seems fairly clear: how many times can you square your capacity before you need to re-architect?

donw2• 16 years ago

I don't think the original poster was making the argument that Twitter should run fine on a SQL database; in fact, I think he seemed to indicate the opposite. Namely, that large, nominally non-relational datasets that can afford to lose a little data here and there, or at the very least just take awhile to save it, are really what you need for serving up a big, fresh pile o' Social Networking.

So, bringing up Twitter or Facebook really doesn't make a good case against RDBMSes as a good tool in the toolbox -- they've got a very unique set of needs that don't apply to a lot of rest of the world. So, of course, SQL isn't the best solution when you're dealing with trillions of rows of data, and don't really want to spend hundreds of millions of dollars on the infrastructure required to guarantee that you never go down, and never lose a tweet.

And keep in mind, RDMBS helped them get to point where they could enjoy these problems; Twitter probably wouldn't exist in all its current glory if they spent a year building it to be 'scalable' before launching.

I think the reason that a lot of people end up hating RDBMS and SQL is because of one-or-more of (a) their only experience is with MySQL, which really isn't that awesome; (b) they've been burned by bad schema design; or, (c) they don't really get relational algebra or set theory.

For an example of 'bad schema design', I once worked at a company that had indices on nearly every column of their DB, even though almost none of these ever got queried. There was one database table with five indices on three columns, and of course this was the table that logged every single HTTP request processed by the front end. Including API calls. Did I mention that this table was never queried by any part of the application?

It was a poor design decision, and sure enough, it completely torpedoed performance. But the problem wasn't the RDBMS, because it did exactly what it was told to do, no matter how asinine.

So, in short, RDBMS aren't the solution to all problems, but they do solve a lot of problems adequately. NoSQL databases also serve an important role in the toolbox, but are much more narrowly-focused.

wooster3• 16 years ago

You make some good points.

However, the original author's point basically boiled down to: if you define scalability as the problems you can scale an RDBMS to solve, RDBMS systems are scalable. I'm not big on arguing the finer points of someone's tautology.

The particulars of a situation determine the scalability of a solution. For a lot of us working at web scale or on interesting new problems, an RDBMS won't scale. Sometimes it won't scale within the constraints we have, but sometimes it won't scale because we won't be able to build the system we're trying to build. His example of a company-internal billing system really only served to highlight the disconnect between the crowd following along well-trod ground, and the people out front doing innovative work.

halostatue2• 16 years ago

I'd also suggest that even in a social networking space, there's multiple types of data and some of it should be stored in an ACID environment (which probably means SQL). You may not care if you can't save a few thousand (or million) status updates to your data store immediately, but you need to care a lot more about your customer profile data (e.g., account, password, etc.) and changes there should be as ACID as possible. Your advertising or subscriber billing models should probably be ACID-backed as well. In both of these cases, that probably means SQL.

wvenable4• 16 years ago

The disconnect here is that only a tiny percentage of sites on the Internet have the requirements of Twitter and Facebook. They are outliers. Using them as an example here is ridiculous.

jdminhbg• 16 years ago

You don't need to have Twitter's userbase to have Twitter's data problem. Look at someone like FlightCaster -- they're using a NoSQL database to handle a huge amount of data to be useful to even their first user.

With the advent of cheap, reliable, available commodity hardware and network access, previously difficult data problems are solvable. SQL doesn't make sense for all of those problems.

wooster3• 16 years ago

Those are the problems I'm dealing with at the moment, so they're the ones that came to mind.

keithwarren2• 16 years ago

I wonder how many people in the NoSQL and SQL doesnt scale crowd either have never met a truly competent, much less good DBA (trust me, they are very very rare) or decided it could not scale because they applied their programmatic and procedural logic to a tool that operates in a very different (SET based) paradigm.

callahad• 16 years ago

Then that's another win for many NoSQL databases -- they often operate in a way that's much more natural for developers.

lucifer3• 16 years ago

Lets wait a few years until all the bells and whistles are bolted (in ad hoc, product specific manner (until someone invents ... a formal query language ;) in layers and tiers that take a NoSQL data store to a full blown database which requires management.

(Didn't we go through this with the XML hype? Wasn't it the "simple" golden nail? Have you checked out the "Simple" landscape of SOAP and W[TF]S-*?)

We're not living in past; its more like back to the future. (Just wait for it, and "No-NoSql" ..)

waterlesscloud• 16 years ago

I've always thought good DBAs were the rarest thing in the industry. And the most valuable.

evgen7• 16 years ago

"Good" DBAs share digs with santa and the tooth fairy, but competent and opinionated DBAs are not as hard to find as you might think. Just whisper 'NoSQL' and they seem to crawl out of the woodwork :)

joe_the_user3• 16 years ago

DBAs don't scale!

If your tool requires people who don't exist, it might be otherwise great but you've got problems.

I'm not even a NoSqlista by any mean, it just seems ironic that you are making a good for nosql - you just need programmers, not DBAs.

coderdude• 16 years ago

I'm not sure where people are getting this idea from. DBA != "Guy who works with relational databases." If you have a sufficiently large amount of data and/or a sufficiently high amount of requests for that data, you will eventually need someone to manage your data(base| store) and the accompanying cluster.

Edit: Or, now the programmer is the DBA, and he still requires the skillset a DBA.

Nelson69• 16 years ago

The biggest SQL problems I've seen the last few years have been ORM using developers using the database as a data structure store, rather than a database. For some reason, the act of getting data in and out of a database was the "hard problem" that had to be solved.

I know this sounds parochial and if you're a close to no money start-up then it might as well be Greek but there is a reason for the business tier in a 3 tier architecture, it translates transactional data in to your data structures... Done well it can scale like a mutha.

I do sort of appreciate the approach of NoSQL though. Map/Reduce is a concept missing from RDBMS and it seems like it could be incredibly valuable for certain problems. There are probably other concepts that could be applied as well. At some point I see the two camps sort of merging back together. ACID is just too important.

rmorrison2• 16 years ago

I'm curious how services like Amazon's RDS will change this perception.

A SQL database may be difficult to scale, but it is something that can be largely encapsulated and outsourced. If Amazon RDS, or some other product, handles the hardware and software configuration, then the developers can just focus on the application portion of it.

This isn't to say that scalability is guaranteed, it's still important to optimize queries and the data structure. Also, there are problems where NoSQL is simply the better and/or cheaper option.

But if these services can encapsulate a lot of the difficult part of scaling SQL, it still makes SQL a very attractive and powerful option for most(?) problems.

jbellis4• 16 years ago

> I'm curious how services like Amazon's RDS will change this perception.

It won't. It's just hosted MySQL.

wmf10• 16 years ago

RDS doesn't scale at all. The performance of RDS won't exceed that of MySQL running on a "Quadruple Extra Large" instance (because that's what RDS is).

rmorrison2• 16 years ago

Ok, but how about a service taking whatever scalable SQL systems the financial or pharmaceutical industry use, even Oracle, Postgres, or Microsoft SQL server, and then selling it as a service similar to RDS?

My point is, I think there is an opportunity providing enterprise-level SQL scalability on a per-use basis. It won't replace NoSQL systems, because there are some problems where they're clearly better, but it could be done and provide relatively-affordable, scalable SQL access to startups.

steveklabnik• 16 years ago

I was really with this article, right up to this:

> If you lose a Status Update, or several thousand of them, it will likely go unnoticed.

What? If Facebook lost half of their photos, or of Twitter lost a few thousand tweets, there'd be riots in the streets. Okay, maybe not quite that much unrest, but still.

wvenable4• 16 years ago

I've run a few medium sized sites (with traffic most people here would drool over) and I would say that people are much more forgiving of slow pages than lost data. Losing a few forum posts would cause riots in the streets.

petercooper2• 16 years ago

I like the article, but the first comment left on it is telling:

I've been dealing with vague "we should use NoSQL" comments from a few of the less capable members of the team.

This appears to be a common attitude of development managers in the corporate world: that anyone who starts suggesting anything vaguely "new fangled" is surely a naïve novice.. rather than being good at picking up and investigating new technologies.

wvenable4• 16 years ago

Unfortunately, managers who aren't defensive about new technologies end up working with XML databases! Not all new technologies are better and certainly the benefits of NoSQL are up for debate.

petercooper2• 16 years ago

XML databases are exactly the sort of glossy new technology that stuck-in-the-mud managers did pick up on.. mostly because they respond to glossy vendor pamphlets, sales calls, and trade show pitches than the grassroots findings of their underlings.

tbrownaw• 16 years ago

They're dealing with taxes (ie, money) and the comments are vague, so it could just as well be that the comments are coming from ungrounded n00bs who get blown about by the winds of every passing fad. I'd expect someone good at actually figuring out new technologies to be able to come up with some rather non-vague reasons to use them.

warfangle• 16 years ago

While SQL can scale, I think this argument is a little spurious.

I truly dislike the NoSQL stance of "never SQL;" it has its place, and its place isn't necessarily at the twitters or the facebooks of today. SQL scales very well with datasets that make sense for RDBMS. CRUD style applications. Core business apps. Data that doesn't necessarily need to be mined furiously. Trying to shoehorn, say, a high-volume message system or intensely self-referential (graph) dataset into an RDBMS is a recipe for disaster, however. Many of the performance issues people see with RDBMS seem to stem from this, I believe.

If your app is hugely real-time datadriven and the datamining (if necessary) can be offloaded to cron jobs, a K/V store is great. And can scale very quickly and relatively cheaply.

If you're doing something hugely relational that, if loaded into a SQL server would require an immense number of self-joins (I'm looking at you, graph analysis) that must be done in real-time and not offloaded to a cron job, a graph database is probably the way to go. They're harder to scale to a huge amount of data, but certain datamining tasks are made much easier - and don't require distributed map-reduce execution. Scaling will get much easier once some systems come out that use k-means (or similar) to cluster and shard the data. That kind of smart scaling would be nigh impossible on either a KV store or a traditional RDBMS. Google gets away with it with BigTable because they can throw so much cheap iron at it - a truly brute force solution. The same solution that needs to be taken with RDBMS when you shoehorn datasets into it that don't make sense.

Emil Eifrem (Neo4J) said it best in his presentation ( http://nosql.mypopescu.com/post/342947902/presentation-graph... ): NoSQL doesn't mean Never SQL, it just means Not Only SQL.

pw0ncakes• 16 years ago

SQL has, by modern standards, a shitty API. The DSL is an ugly mess and it's extremely difficult to reason about performance, especially when joins are involved.

Ergo, some not-by-choice SQL users only scratch the surface and use common features: INSERT, UPDATE, DELETE, SELECT, also known as CRUD.

Ergo, it becomes easy (although wrong) to conclude that SQL isn't doing very complicated work behind the scenes and that it's just another overcomplicated dinosaur POS like the Windows operating system, one that remains popular only because it's a standard.

Thus, a lot of undue hate gets directed at SQL with little attention paid to the subtleties of what it does extremely well and where it behaves poorly.

I think "SQL isn't scalable" is in the same league as "Java isn't concurrent". It can be, if you have learned the necessary skills and are willing to deal with a bit of pain. Is Clojure astronomically better for concurrency? Sure, but people can and do scale with SQL databases, and they do this in large part because there are problems for which SQL is the appropriate solution.

derefr• 16 years ago

Why are SQL and RDBMSes conflated? Couldn't one database speak both SQL and something else if it wanted to? In the same way that Clojure and Java both end up as JVM bytecode, couldn't SQL just become another layer on the encoding stack, above something simpler?

richcollins• 16 years ago

One complaint that I have with SQL databases that you don't often hear elsewhere is that they are very complex. People often use them where a much simpler solution would suffice because they are what people are used to using. All else equal, we should choose the simple solution over the complex one, because it is less likely to fail and easier to extend.

nostrademons• 16 years ago

I thought his example really telling.

Say that his internal help ticket tracking system was built for IBM, one of the largest corporations out there with 300,000 employees. 300k users is tiny for a consumer app. We had more than that when I was volunteering for a Harry Potter fanfiction website. Even if he was working at the largest company on earth by employee count (Wal-Mart), he'd still have fewer users than we had for Harry Potter fanfiction. And usually employees don't submit more than one or two help tickets a day, while Harry Potter fans tend to view a forum thread every minute or so.

It really hits home how consumer data processing has changed the game for data management. When I was working in the financial industry, we dealt with about 50GB of data/day coming off the exchanges. I thought that was a lot. But at Google, there's terabytes per day - at least two orders of magnitude more - and the total volume of financial transactions is basically rounding error on the data we handle.

It makes sense that with this exponential explosion in data, we'd need different techniques to handle it. Quite likely, RDBMSes do scale for the scale he's talking about. But a bunch of industries have opened up within the last ten years that require several orders of magnitude more data, and it's naive to think that just because it works for a corporate help desk or POS system, it'll work for a system that logs every page view and every action for millions of users.

ergo98• 16 years ago

>300k users is tiny for a consumer app.

Active users, not the total number of users in the user table. People grossly overestimate the scale of most web properties, where the number of active users is far lower than you likely imagine.

>But a bunch of industries have opened up within the last ten years that require several orders of magnitude more data, and it's naive to think that just because it works for a corporate help desk or POS system, it'll work for a system that logs every page view and every action for millions of users.

Strangely it says nothing of the sort. Yet here, again, you've used Google as the example. How many Googles are there? How much does that apply to about 99.999% of people who deal with databases?

Yet it always appears as the example.

petewarden• 16 years ago

> How many Googles are there?

There's tens of thousands of alternative search engines out there, see this for a sample from '07:

http://www.altsearchengines.com/2007/10/29/the-top-10-list-o...

Most of them fail, but there's a lot of people trying to solve Big Data problems on a shoe string budget. That's part of the disconnect, us NoSQL folks are excited by having cheap solutions to problems we wouldn't be able to afford to tackle otherwise, whilst SQL folks are are shaking their heads at the mess we'll have to clean up if our prototypes do become successful.

Aleran• 16 years ago

Was anybody else dumbstruck by the article's first comment by this bright chap, "Jeff"?

Brilliant. I've forwarded this to my team.

We make a tax solution and I've been dealing with vague "we should use NoSQL" comments from a few of the less capable members of the team.

If his team members read all the way to the comments it's going to be very awkward tomorrow at work.

joe_the_user3• 16 years ago

Well if some of the dumbshits I work with at the White House read this I'll be in trouble too but I don't think that's going to happen.

Care to buy some classified material?

citatus• 16 years ago

I wish that people would understand that people who disagree with them are not 'liars' but are instead people who disagree with them.

It is possible to honestly disagree on a technical issue, you know. And it is good professional and personal practice to only accuse someone of lying when you are quite sure that they are deliberately telling you something they know to be false...

gsteph22• 16 years ago

The word "butthurt" comes to mind when thinking of this article. ;D

Dear NoSQL: "SQL-isn't-scalable" is a lie

Related Stories