So long AWS, and thanks for all the fish: FathomDB (W08) focusing on its new DB

http://blog.fathomdb.com/53681308

by justinsb • 15 years ago

28 27 15 years ago

blog.fathomdb.com

Joakal2• 15 years ago

This author blames Amazon for a poor SLA yet continues using their service?

What cloud provider offers a more reasonable SLA?

justinsb11• 15 years ago

I'm the author. I think you're conflating the SLA with the guarantees the cloud offers: one of the points I tried to make was that an SLA doesn't offer you any real guarantees; they only really benefit the service provider. If every SLA gave you a "50 times" penalty, as FathomDB is paying, then they would be interesting, because at that level the provider has to take uptime seriously.

In terms of who offers a better cloud, clouds based on OpenStack are 'the great hope'. IaaS is about commoditizing infrastructure, so open source seems necessary; rather than AWS's very closed, secretive approach.

Joakal2• 15 years ago

I should have rephrased better; "What provider offers a SLA that takes IaaS uptime more seriously than Amazon?"

mtrimpe• 15 years ago

Given that this is the first blog post and you still seem to be in private beta... how many customers do you actually have now? One? ;)

sunchild2• 15 years ago

I deal with SLAs in my job all the time. After over a decade negotiating uptime SLAs, I'm firmly convinced that the best protection against downtime is a huge, vocal userbase, and the best remedy for downtime is an unqualified termination right and refund of prepaid fees.

An SLA (even with severe penalties) is certainly setting up the right incentives on paper, but in practice it has little effect on the results. Why? Because enforcing a promise on a piece of paper is bothersome, expensive and sours the relationship.

Also, saying you'll pay 50x penalty is only half the story. What are the triggering events that entitle customers to payment? Is the SLA measured end-to-end (i.e., from the user's endpoint to yours)? Is the SLA penalty cumulative? Can your customers actually profit from your downtime (i.e., you would end up owing them more cash than they paid you)?

Without more details, your post suggests that each fraction of a second of downtime experienced by any single user of your service entitles them to a credit of 50x what they paid for that fraction of a second. Is that your policy?

ChuckMcM2• 15 years ago

This reads poorly.

FathomDB customers saw outages because the implementation Fathom did using MySQL on AWS went out when AWS went out.

So in conclusion Fathom is going to build something which Oracle, IBM, SyBase, and every other database company in the history of the planet has tried and failed to do, build a relational database on a distributed infrastucture that is more reliable than the underlying infrastructure? Uh, good luck with that. Seriously, its Turing Prize material if you succeed.

I was thinking the punch line would be "Gee Netflix uses AWS and they didn't go down so we're going to more what they did." I guess they are going to compete with MongoDB and Riak.

justinsb11• 15 years ago

Thanks for the good-luck wishes. Obviously it's a big undertaking, but we're seeing very promising results, so it's time to go all-in. Take that as you will :-)

Incidentally, Oracle and IBM both have clustered database products, though they can certainly be improved upon.

jacques_chester4• 15 years ago

Oracle and IBM do have clustered stuff and it's amazing, but it still tends to assume closely linked networks of high-reliability hardware and have trouble with very large, slowly linked networks of low-reliability hardware.

You might find Greenplum and VoltDB to be more interesting models to study.

The key question is: are you aiming at OLTP or OLAP? Because, as those boring mainframe-era guys have discovered, it's difficult to serve two very different purposes.

benatkin• 15 years ago

I agree that it reads poorly. If I was a tinier bit more motivated to finish my HN session more quickly, I would've assumed that FathomDB did lose some of their customers' data. That's what I thought when I read the headline, and only when I started reading the article did I see that they didn't lose any of their customers' data.

unshift2• 15 years ago

sorry, this is probably tangential, but how is the AWS outage any different from any other hardware outage, regardless of whether or not you actually own the hardware?

why does it seem that everyone expects 100% uptime from a VM just because it's in "the cloud"? shouldn't "the cloud" be used to make fault-tolerance even easier, because you have access to multiple geographic regions and multiple providers with little fuss?

they're still computers, and they're still bound to go down occasionally. i don't see how running leased VMs somehow absolves you of doing basic operations work and guarantees a bulletproof experience. regardless of where you host it, writing a nicely distributed and fault-tolerant system is and always has been difficult. the only part that's gotten easier is finding rack space.

justinsb11• 15 years ago

With physical hardware, the basic failure characteristics etc are well known. When the hardware is virtual or abstracted as it is on the cloud, you have no choice but to rely on the promises made by the provider. If you can't build a reliable system on the abstraction provided by the cloud, you can't reliably use the cloud. But with a good cloud, you can architect a reasonable solution based on the provider's promises.

However, if the provider doesn't keep those promises, then all your hard work and calculations go totally out the window.

At that point, you have to figure out a way to run a database on a system that effectively offers you no guarantees. That's what we're working on.

dialtone• 15 years ago

For your application though a failure is a failure in any scenario. EBS went down in one zone and was slow in a couple others for a few hours, it's a bad downtime but some services survived the problem relatively unscathed. This suggests that something could have been done to avoid any trouble at all.

If the datacenter you are in has a slight conditioning problem this summer and 10% of your drives breaks down due to excessive heat, how quickly will you be able to re-provision the data center?

About a week after AWS outage the italian ISP Aruba had a UPS failure due to a fire in the UPS room and the entire datacenter switched off automatically during the night. For the following 8 hours that datacenter was off for every customer. How would your new solution handle such a situation?

Designing a hot standby replication solution for MySQL/PostgreSQL that works across regions seems easier to me rather than implementing a database from scratch that should solve a very complex problem.

unshift2• 15 years ago

sorry but that seems really naive. an SLA is not a guarantee of service -- you can't force a computer or network to stay online because someone with a sheet of paper said so. it's a target for a best-effort guarantee and when that guarantee is broken then you get a bit of a refund.

it's like when you have a big building project and there's a provision that the contractor will refund $5000/day for every day past march 1st. doesn't mean march 1st never gets passed.

failure modes in the cloud are known too: your VMs are either working or they aren't, or maybe they're somehow degraded but you should fail over anyway. it's very similar to physical hardware. what will you think when a backhoe takes out your datacenter's fiber for 10 hours? that physical hardware and datacenters are now unreliable too?

as for a database that runs on a system that offers no guarantees -- isn't that all of them?

soamv• 15 years ago

Can you actually tell us a bit about the new DB you're building? (What is it designed to run on, what kind of failures are you aiming to tolerate, and so on.)

justinsb11• 15 years ago

Sorry, not too many details yet - let us finish it first :-) Commodity hardware though, and quorum-style failure-tolerance characteristics.

So long AWS, and thanks for all the fish: FathomDB (W08) focusing on its new DB

Related Stories