Scaling out Postgres, Part 1: Partitioning

http://petrohi.me/post/30848036722/scaling-out-postgres-partitioning

by craigkerstiens • 14 years ago

60 25 14 years ago

simple102• 14 years ago

Craig, thanks for sharing Peter's article. As a side note, we started on Heroku (on the cedar stack as well as actually working out of the Heroku office -- we're big fans). We're now hosting with Liquid Web on dedicated db boxes due to the large RAM and IO speed requirements of our app.

We also initially wrote the app to use a mix of Postgres and MongoDB but ended up moving everything into Postgres.

stoked2• 14 years ago

What were the reasons for moving away from MongoDB?

simple102• 14 years ago

Our app does a lot of writing. We were on Mongo 1.8 which has a global write lock. So concurrency was a big problem under load. We also have a lot of users with networks of 200k+ people – not even including followers. This meant that we had to have arrays in the User collection with 200k+ IDs to perform a lot of our queries. The RAM needed for indexes wasn't really worth the cost and performance started to become a big concern. In the end, our data makes more sense in SQL with a cache layer (Redis in our case) rather than NoSQL.

Our experience thus far in Mongo vs Postgres: go with what you know. We probably could have made Mongo work for us (especially Mongo 2.0+), but it was easier for us to work out the performance issues with Postgres. Plus, things made more sense in SQL.

If you're using Mongo and need a cloud provider, I can recommend the people at MongoLab. They really went above and beyond helping us get up and running and are really cool people.

jadoint• 14 years ago

As someone who's currently looking for a new hosting provider, I'm curious to know what made you go with Liquid Web. My experiences with them have been so-so and have been thinking of moving to SoftLayer for possibly more reliable, albeit more expensive, hardware.

taligent5• 14 years ago

Having to implement your own sharding in your application layer for a database with 3 tables is ridiculous.

No wonder people are moving to products like Cassandra, HBase, MongoDB, CouchDB etc which automatically and transparently shard across as many nodes as you like.

petrohi3• 14 years ago

It is not the number of tables that requires sharding, but the amount of data and associated workload.

While automatic sharding definitely has its place, we wanted to be very explicit in our consistency tradeoffs.

taligent5• 14 years ago

Sure. But let's be realistic here the amount of data you would be dealing with is pretty insignificant compared to the bigger users of Cassandra and HBase or even MongoDB.

But hey if you wanted to be explicit about consistency then I am sure you had your reasons. Just strikes me as all a bit odd. Then again I didn't realise PostgreSQL didn't have an automatic sharding feature.

rekoros2• 14 years ago

I don't have any experience with HBase and CouchDB, but Cassandra and MongoDB will only shard automatically and transparently if your application does not do much writing. :-P

Also, the number of tables has nothing to with the problem of sharding, it's the amount of data you're trying to store.

taligent5• 14 years ago

I suppose you would have an issue with MongoDB specifically if you were trying to rebalance from day one. But generally you wouldn't. I know Foursquare is using MongoDB's Auto Sharding in production and they would definitely have a lot of writes.

And the number of tables DOES matter for sharding if you are implementing it yourself.

monstrado• 14 years ago

HBase guy chiming in. Sharding in HBase is a bit different than the other NoSQL guys. HBase is super fast when doing random lookups and scans, so the developer is in complete control of his failure or success, depending on the row key design.

Good thing about HBase is that it's designed to hold a massive amount of data, I've seen people with hundreds of thousands of columns on a single row, it's pretty remarkable.

orthecreedence• 14 years ago

I used MongoDB in a medium-sized production environment (Aol's Editions backend, which I built with my brother). The sharding, for our needs, was far from automatic. In fact, the only way to get MongoDB to shard records evenly across servers (this is with 1.8) was to for all intents and purposes implement our own sharding on top of it. In other words, the global write lock coupled with worthless auto-sharding (at least given our purposes: high write, temporary but mission-critical data) made MongoDB a PITA. That, along with the fact that we were more or less told that MongoDB would be able to handle what we were planning by the MongoDB consultants Aol hired completely soured my view of 10gen. The only reason we needed sharding in the first place was because we were getting large amounts of queued writes on a 32GB server with 16 cores (with < 300 w/s). Pathetic.

My point is that if you want to spread out your writes on a database that supports rich queries and honors the "C" in CAP, you're going to have to do some work. The distributed DBs are nice and shiny if you can forego some of the nicer querying features of relational DBs (or MongoDB), but not everyone's requirements fit into document-based records, and to assume they do is just NoSQL fanboism.

Note: I am using MongoDB for my latest venture because we are realistic about it's abilities and also because our development time with MongoDB is no joke about 1.5x-2x faster than with MySQL (or equivalent). So MongoDB has its place and it definitely a good system (especially after v2) but NoSQL isn't for everybody, especially if you can't get past the hype and look at decisions very technically and objectively.

rbranson• 14 years ago

Maybe I'm not understanding something, but there seems to be an implicit assumption that the message broker (RabbitMQ) is more reliable than the database. This seems to be a common fallacy.

petrohi3• 14 years ago

Yes, I can see where you're coming from. Message queue in the article is more of an abstract service that provides certain guarantees and less of a specific RabbitMQ implementation. There are examples of message queues implementations on top of database--the resulting tradeoffs can be an interesting discussion.

bbromhead• 14 years ago

Glad to see you are having "good problems".

Didn't like parallel queries using pgpool2?

petrohi3• 14 years ago

We are planning to use pgpool2, but only for connection pooling. In our case the subset of postgres databases to execute a distributed query is dependent on the input parameters, current sharding configuration and cached routes. Cached routes are used to handle adding new shards without repartitioning--I will cover this in the second part.

I guess we could use pgpool2 parallel queries, but this would likely require some adjustments in how they are implemented.

Scaling out Postgres, Part 1: Partitioning

Related Stories