Three years later, Mr. Moore is still letting us punt on database sharding

http://37signals.com/svn/posts/3089-three-years-later-mr-moore-is-still-letting-us-punt-on-database-sharding

by jlangenauer • 14 years ago

127 54 14 years ago

groggles• 14 years ago

Pedantry -- that isn't Moore's Law related, aside from the loosest interpretation that computers as a whole get faster.

This is so true, though. Much of the horizontally scalability need comes from the land of extremely underpowered VPS machines on platforms like AWS. Yet the mind-boggling scale of performance you can inexpensively* (*-term used relatively) acquire for database servers is astonishing. SSDs (and plug-in flash drives) and boundless memory have changed everything.

_delirium• 14 years ago

Given all the problems they've had with AWS, I wonder what the overall cost/benefit would be for someone like Reddit moving from the flexed-EC2-servers model to just one high-powered db server and some caching webservers in front of it.

adestefan• 14 years ago

Reddit's probably was using trying to use EBS as a backend for a transactional database. The issue is that EBS makes no guarantees on latency, only that they'll never lose your data.

foobarbazetc2• 14 years ago

Reddit could easily cut their spend on hosting by like 50-75% just by moving to a dedicated hosting provider. They'd also get bare metal speeds in the process.

Who knows why they've stuck with AWS, though. It's possible Amazon is giving them a discount so they can be used as an example.

masklinn• 14 years ago

> Pedantry -- that isn't Moore's Law related

Actually, RAM would be the perfect application of the exact meaning of Moore's law: more RAM is an almost direct function of more transistors, and Moore's law is:

> the number of transistors that can be placed inexpensively on an integrated circuit doubles approximately every two years

Moore's law precisely predicts you can double your amount of RAM every two years for the exact same price. Which is pretty much what TFA is about.

SSD storage size is the same thing.

ohashi2• 14 years ago

For some reason, I don't imagine BaseCamp's use is really exploding like some other startups might in terms of database usage.

It could very well be a case of their growth is scaling in-line with Moore's law, so it just happens to work out well.

moe• 14 years ago

I agree. Without any figures this is a pretty meaningless blog-post...

ams61102• 14 years ago

I think most "other startups" would LOVE to be in the ballpark with BaseCamp in terms of usage. No BaseCamp is not big like Facebook, but you've heard of them right? Most "ordinary" business people who do project planning work have probably heard of BaseCamp too. Most people never hear about most startups, fewer try their services, fewer than that become regular users.

The point of the post is that in many/most cases, it's still easier and cheaper to throw hardware at a performance problem than to devote scarce engineering effort to optimization. And it's only getting more and more so. If you are a startup and you can throw $10,000 of hardware at a problem you can then keep your $100K engineers working on things that hardware alone can't solve.

jonknee• 14 years ago

> Most "ordinary" business people who do project planning work have probably heard of BaseCamp too.

I don't think so. Maybe we just have a different opinion on ordinary, but I come across people all the time that would think Excel is the normal thing to use for this (and pretty much any other task...). Using web applications for day to day workflow is still alien to a lot of "ordinary" business types.

untog• 14 years ago

Most "ordinary" business people who do project planning work have probably heard of BaseCamp too.

I think this might be the tech bubble showing. I've never heard of any non-tech, non-startup companies using Basecamp. I'm not saying that they don't, of course, but I'd be interested to hear some case studies in its use outside of tech-savvy crowds.

patio11• 14 years ago

Even among startups, web scale data requirements are the exception, not the rule. Facebook and Google are ginormous. There are many, many very impressive applications whose database wouldn't tax a single commodity server. (Similarly, there are applications that make terrible businesses but which consume computing resources like losing a byte of information would doom humanity.)

I mean, go through a list of YC companies or other startups you respect, winnow it down to the ones that exited or otherwise achieved some level of success, and play guess-the-size. How many terabytes of storage do you think e.g. Airbnb needs?

j_baker3• 14 years ago

One thing I've found is that scaling rarely means solving difficult problems. Rather, it means putting more time into finding optimal solutions to problems that are trivial at smaller scale. For example, should your startup use Apache, nginx, or HAProxy as a load balancer? If you're just launching, the answer is "Who cares, just ship the fucking thing!". If you reach the point where you start measuring page views in the billions (and yes there are start ups that are at this point), it matters a great deal. Or should you use Postgres, MySQL, or some shiny NoSQL thing? Again, probably doesn't matter for small websites. But for larger services, it matters.

Also, don't underestimate how large log files can grow in a data-driven business (like AirBNB seems to be). I could easily believe that they have many terabytes of data just from logging actions their customers have taken.

ohashi2• 14 years ago

I don't disagree with anything you've stated. Explosive growth and requiring massive amounts of data storage are surely the exception not the rule.

That said, the blog post talks about enormous growth and it still fits inside Moore's Law's growth. I guess my gut is just saying it's not really that enormous in terms of startup scaling if it's still within those limits. Not to take anything away from 37Signal's success, but it feels like nothing of value was really added by this post. I present the post of a picture of 864GB of ram as supplementary evidence that is near the top of HN right now.

rbranson• 14 years ago

Perhaps for the applications of yesterday (like Basecamp) this is the case, but the real innovation taking place is around collecting massive amounts of data and processing it in interesting ways. These systems are used every day to make quantified business decisions rather than best guessing based on someone's hunch. 37signals builds questionably good UIs on-top of a database, something people have been doing for decades now. The future is in augmenting intelligence by gathering massive amounts of data and reducing it for human consumption.

InclinedPlane• 14 years ago

There are a ton of high-traffic websites out there that don't need an architecture any more complex than a standalone DB server + PHP + varnish (or the equivalent).

More so, if devs spent as much time tuning the performance of their apps as they did fantasizing about "web scale" architectural pivots they would typically be farther ahead. StackOverflow.com is a perfect example of this. They run on tiny handful of windows machines, support gobs and gobs of traffic, and have absolutely fantastic performance. And as much of that is due to paying attention to performance and making sure to find and remove the bottlenecks where they exist as it is to using cutting-edge architectures like database sharding, map+reduce, eventual consistency models, etc.

alexjipark• 14 years ago

hmmm

shanemhansen2• 14 years ago

A lot of startups engage in a sort of cargo-cult architecture. Their reasoning goes something like this:

1. Amazon/Facebook/Google have a lot of traffic.

2. Amazon/Facebook/Google use X to scale horizontally. ergo:

3. My little startup should use X and scale horizontally.

What they fail to realize is that most of these companies would be ecstatic if they could scale machines vertically, if they could focus on great user features instead of having to figure out how to shard in the application layer. You should never forget that Amazon, Facebook, and twitter all started out as pretty basic LAMP stacks and built the tools when it was obvious that no other tool would do. I think google's an exception because their MVP was in fact a web scale application. So by all means, vet your idea, get some customers, get traction, and scale the cheap way by buying more ram for as long as you possibly can.

marshray• 14 years ago

Yes. But let's look at the other side too.

Before Google came along and showed the business people the benefits of horizontal scaling, any software engineer would be automatically considered crazy if they suggested an architecture that wasn't built on a central RDBMS.

So you have to weigh it against the other cargo-cult. How many startups along the way have failed due the inability to scale horizontally?

How many have failed due to too much cost and complexity associated with re-engineering an architecture in which the assumption of fully ACID transactions permeates the entire codebase? (While the phone is ringing off the hook because production systems are falling over under load.)

j_baker3• 14 years ago

I realize this may not be a popular opinion on HN, but there's something to be said for planning ahead. I've seen this story before and already know how it ends: you wait until the last possible moment to switch to a more horizontally scalable system and next thing you know you're spending more time and money maintaining the "cheap" solution than it would have taken to switch to something like Cassandra beforehand. To make matters worse, your service is crashing and the short term fix takes a day and requires two or three people to do the replica switch shuffle. The long term fix will take a couple of weeks, if you have the time for it of course.

Long story short, I have to grant that you shouldn't worry about scaling up too soon or too quickly. But that don't go to the opposite extreme by putting it off until the last possible moment.

spudlyo• 14 years ago

Amazon did not start out as a LAMP stack, it was more like a "DNBC" stack.

D igital Unix

N etscape Commerce Server

B erkeley DB (NoSQL!)

C code (linked into the HTTP server)

cperciva2• 14 years ago

B erkeley DB (NoSQL!)

I'd call BDB "pre-SQL" rather than "NoSQL".

krupan3• 14 years ago

He mentions Fusion-io drives getting faster. What's interesting about iodrives is that you don't have to buy a new one to get more speed, just updating your CPU or RAM will speed up an iodrive:

"Because VSL leverages powerful host CPU cores and memory for block mapping, ioMemory performance automatically improves as organizations upgrade host CPUs or memory" (http://www.fusionio.com/overviews/vsl-technical-overview/)

VSL (Virtual Storage Layer) is the Fusion-io software that makes an iodrive look like a block device to the OS, among other things.

(Yes, I'm an engineer at Fusion)

bitops2• 14 years ago

What never seems to get mentioned is energy cost. Sure, you can keep throwing hardware at the problem, but eventually that will lead to a substantial power bill.

rorrr• 14 years ago

Sharding will lead to an even much higher power bill.

wildwood• 14 years ago

Agreed, sharding takes "throwing hardware at the problem" to a whole new level.

wes-exp• 14 years ago

For those with custom hardware setups like this, how much time do you estimate you've spent designing, building, and maintaining custom servers?

I can see how it would be less effort than sharding, but it seems like throwing hardware at the problem is not a completely "free lunch" when you get to the scale of having to custom design a server supercomputer and source specially made SSDs and terabytes of RAM.

sciurus• 14 years ago

You don't need to "custom design a server supercomputer". This stuff is as mainstream as Dell. You can go to dell.com and order a Poweredge R815 with 64 processor cores and 512GB of RAM for $16,000. If you want SSDs, you can purchase SATA or PCIe ones from Dell as well.

http://www.dell.com/us/enterprise/p/poweredge-r815/pd http://www.dell.com/us/business/p/fusion-io-drive/pd

kyledrake2• 14 years ago

I did a presentation on a game I made at Geoloqi that talks about the Fusion IO drives and why I think local hardware with SSD is the best solution for persistent data stores right now: http://www.slideshare.net/KyleDrake/building-mapattack

kyledrake2• 14 years ago

The one warning I want to provide though is that not all SSDs are created equal: Make sure you get one that writes its cache to the disk on power failure, or you're going to be in a world of hurt.

krupan3• 14 years ago

Can you explain more? You aren't talking about writing data that is cached in the SSD to a regular hard-drive, are you?

bwooce2• 14 years ago

Gah, stop abusing the word punt. The headline is nonsensical.

Punt does not mean avoidance. It has much closer associations to "attempt" (punter, plus rugby usage).

I totally agree with the rest of the article, why shard or otherwise distribute your database before being required? Although I think I would build my DB as shard 1 of 1 to allow for the future case.

ellyagg• 14 years ago

You're thinking of British English. In America, punt means avoid, and most people here probably think of an American football punt--which is the voluntary end of a failed offensive possession--so suggests you're giving up for now and will try again later. We probably bastardized it, sure, but that ship has sailed.

bwooce2• 14 years ago

Thanks. Such a great word, lost forever :)

So if I said "take a punt", would that be to take, or not take?

I should build a guide of "words not to use on presentations".

mhartl• 14 years ago

I think they mean "Dr. Moore". "Mr. Moore" is a hack documentarian. Dr. Moore, on the other hand, has a Ph.D. in Chemistry from Caltech.

dkhenry• 14 years ago

This is nothing more then trying to pay the interest on your technical debt instead of paying off the balance. The longer you wait the more it grows and the bigger the problem becomes down the road. Right now its 20K for a new system, And for your troubles you still get no redundancy. Yes ram is cheap and you can just buy a bigger system next year, but hopefully you will hit a limit (growth is like that) then you will need to scale your now large and complex system. Code will need to be reworked. Interfaces will need to be re architected. The guy who wrote that bit of code for you that runs your whole business he may not be with you any more, even if he is that hack he put in place because you had the horsepower to spare, he doesn't really remember why he did it at 3:00am that one morning when stuff got out of hand. Now you have downtime and huge development costs, because you ignored a problem and instead threw hardware at it.

wazoox• 14 years ago

One of the most common similar occurrence I meet regularly is people grossly underestimating how many bytes per second you can really get from basic hardware such as hard drives, gigabit ethernet cards, etc. People often still remember that in 1998 the fastest FC 10k rpm drives hardly topped 12 MB/s, and that you needed an 8 CPU Origin2000 to push 80 MB/s through a GigE link (given that you had a terrific RAID array at both ends).

Nowadays, even the most basic PC can saturate a GigE link (115 MB/s), and the slowest hard drives go 100 MB/s. Any SSD sustains several thousands IOPS, and so on.

One of the most unfortunate result is that people often buy hugely powerful hardware when very basic stuff would have done the job just fine. How many people have I seen running puny workloads on 100k bucks 3Par or EMC arrays, where a handful of SSDs in a server would have done as well or better? Pulling 40Gb fibre across the room when cat 6 cable would have sufficed?

Three years later, Mr. Moore is still letting us punt on database sharding

Related Stories