alexstaubo

> Ruby is palpably slower than everything.

Try 1.9. It's usable today, and should be fully stable by January. Performance-wise, it's on par with Python.

After going through Courier New, Profont, Consolas, Monaco etc., my favourite programming font ended up being Panic Sans, which comes with Coda, a Mac IDE that I don't use. For developers whose point size of preference hovers closer to 12 points than 6, it's perfect. Sample: http://purefiction.net/paste/panicsans.png

Skip lists are one of my favourite data structures. They're slower than balanced trees, but ridiculously simple to implement.

Of course. Thanks.

PostgreSQL table partitioning has nothing to do with shards. They don't let you split a database across multiple machines. You can use them to split a single table across multiple disks (using tablespaces), but that's not the same thing.

Sharding is icky and hard because it strokes relational databases against the grain. It also, incidentally, goes against the conventions of most web frameworks, including Rails. Since there's no database that does it for you, you really have to design for it from the beginning. Clay Shirky's scalability book, with anecdotes from Flickr's early development, is a must-read.

The Skype people do transparent sharding with PostgreSQL (the highscalability.com guy blogged about it here: http://highscalability.com/skype-plans-postgresql-scale-1-bi...). They accomplish this using their own PL/Proxy plugin (http://developer.skype.com/SkypeGarage/DbProjects/PlProxy). The only way they can do this is by _never_ doing SQL in the client. Instead, all SQL is wrapped in server-side functions, aka stored procedures (written in PL/Python). The magic in PL/Proxy is to enable the execution of these functions based on a hash. So for example, a function get_user_email(username) is implemented as:

create function get_user_email(username text) returns text as $$ cluster 'userdb'; run on hashtext(username); $$ language plproxy;

...which results in the real function being executed on some other server.

There are pros and cons to this approach. Could they not have implemented this in the application? Yeah. The additional latency of doing one extra remote call (even if it can be pipelined) can't be good.

Another way of automating sharding is to use a middle layer like PgPool, GridSQL or Continuent uni/cluster. The first two are open-source projects, the latter is commercial; PgPool is written in C, the last two are Java (although GridSQL has C bindings).

PgPool is a proxy that can also do replication, partitioning, load-balancing and parallelization -- and you can pretty much pick which ones you want. PgPool intercepts all SQL statements, inspects them, possibly rewrites them, and sends them to one of several backend PostgreSQL servers. For example, it can route all updates to multiple PostgreSQL servers, ensuring that they're identical (until one of them goes offline, at which point you have to pull it manually back into the pool). It can also partition data, by checking inserts and routing them to the right box, and you can write the partitioning functions in PL/pgSQL. And it can also route queries semi-intelligently, so that a query that is known to only touch a single partition will only go to that partition. And it can parallelize them, so that when you do "select * from foo", it'll run the query on all servers and then combine the results.

GridSQL covers the same ground as PgPool, but seems a little more advanced. It has a complete SQL parser that is supposed to be able to do query routing and rewriting more efficiently. From what I can see, they are emphasizing performance and parallel queries above anything else.

I haven't tried uni/cluster, but it's similar in scope and features.

The main problem with these products is that they themselves become a bottleneck. I don't think you can scale them horizontally by just piling on more proxies -- the proxies themselves are gatekeepers. So you're just trading one bottleneck for another.

After implementing a couple of large, popular, bottom-heavy, hard-to-scale Rails apps, I am now of the opinion that well-designed apps should never talk directly to a database; by doing this, you are making the database layer a bottleneck, which is particularly bad with relational databases, which cannot scale very far horizontally.

Instead, you should have the application serve an internal API that can be broken up and multiplied and moved. For example, consider the get_user_email function from the exampe above. Consider the Rails way:

User.find(params[:id]).email

You are already tied to a very specific code path -- find the object, read the email. To shard this thing, you have to override the find method, as well as any update methods, and you will end up a graceless patch on top of ActiveRecord. What if we made the use case -- getting the email address of a user -- explicit, as a real API?

User.get_email(params[:id])

Similarly, you would "sculpt" dedicated methods to handle more complicated queries. Consider:

User.get_latest_comments(params[:id] :in_forum => params[:forum_id])

Now we have isolated ourselves from the database, and we have an API that can actually be sharded. In fact, what we have is an abstraction. Abstractions are useful. Sure, the implementation can use ActiveRecord or whatever, but it's no longer dependent on anything except the input. It's not dependent on database model details like which table holds the email address -- the abstraction separates the concern of the application from the data API.

If we have a single app database and it doesn't scale, we can move all the users over to a different server with its own database and make the API be remote, based on a REST or dRb interface. If this user server becomes too slow, we can just add another user server -- "just add another box" is the core tenet of any scale-out stategy.

So if I was building a new app today, that's what I would do.

Seconding simonw's Yahoo recommendation. The API is great and the results are often as good as Google's. If you scrape Google searches, they will eventually block you.

You're welcome!

Part of the reason for there not being a "standard" setup is that there are tons of OODBs, and they are all implemented a little differently.

Some of the major ones are Objectivity/DB, Versant, ObjectStore, GemStone (which comes in Smalltalk and Java flavours, where the former is a more mature and flexible product) and InterSystems Caché. Caché is the odd one out; it's object database as well as an SQL database as well as a MUMPS app server, and while it has plenty of support for things like Java and .NET, it's fundamentally a MUMPS system. To most of today's programmers it smells a bit weird.

OODBs have traditionally been intimately tied to the language you write your application in because they usually attempt to seamlessly add persistence as an extension of the application's object model: that is, when you create normal language objects in RAM, they become persistent on disk. As a result, you tend to pick an OODB based on the language you use, and once you've picked the OODB, you tend to stick with that language.

OODBs started out with C++, but automatic persistence was quite awkward and necessitated inheriting your classes from the OODB's own class libraries and doing a lot of calls to the OODB runtime. The OODB firms quickly latched onto Java when it became a serious contender in the late 90s. The reason is simple: It's really easy to make persistence transparent and elegant in Java. OODBs typically provide post-processing tools that munge the Java bytecode -- disassembling your code and adding persistence calls.

With persistence in place, all you have to do is set up a schema (typically by letting the OODB tools "sniff" your classes) and then add transaction code to your app. For all the transparency, it's actually a bit invasive, and OODBs are typically very strict about accessing instances in a transactional context. You generally don't store long-lived references to objects, and there are special requirements for sending objects to other threads.

As for specific products, I was never happy with any of the OODBs I tried back in the late 1990s. Objectivity and GemStone seem like the most mature ones, but they never provided evaluation versions so I could not try them out; they've got the enterprise bug, so they're expensive and not very open in terms of documentation, code availability and so on. In fact, you will have to turn a few stones on the GemStone site to even find the GemStone OODB products. If you are interested in MagLev you might want to look at GemStone/S, since it's essentially the same thing.

For Smalltalk there is also the open-source Magma database, which sounds promising. For Java there is an interesting open-source project called Neo (http://www.neo4j.org/), a "netowrk database" which unlike the aforementioned OODBs don't actually attempt to integrate with your application's object space; instead, you create Node objects and attach attributes to them programmatically. If you're going down this route there's also CouchDB, a pure-data "document database".