Ma.gnolia.com crashes hard -- no backups?

http://ma.gnolia.com/

by eli • 17 years ago

95 77 17 years ago

ma.gnolia.com

sam_in_nyc2• 17 years ago

His note says it's January 31st... What will get fixed first? The date, or the data? I'm betting on data.

whatusername2• 17 years ago

I don't know about you - but it's January 31st for me.. :P (although I do believe it was just a typo)

jjs• 17 years ago

Tell us what the future is like! ;-)

quoderat• 17 years ago

The backup strategy, in my experience, is the first or second thing you need to be thinking about when you set up a server, service or anything IT-related.

The first thing I ask when my boss tells me to set up something new: how are we going to back that up?

eli8• 17 years ago

It's a classic dilemma -- most people don't care about backups until immediately after it's too late.

mdasen4• 17 years ago

This is why it's so important to take some simple steps to ensure that you don't bork your site. For my production sites, I usually have the following setup:

* Replicated Database with one slave in a separate data center just as disaster recovery (not a performance thing with the separate data center). Heartbeat for the slave(s) in the same datacenter so that it stays up.

* Nightly backups offsite. Used to be rsync, but recently thinking of using tarsnap for the ease of it (and S3 puts it in several data centers).

* Files stored either in a MogileFS setup or S3 in multiple data centers.

It doesn't have instant failover to another data center, but the offsite DB slave should mean no data loss beyond a second or two. Ma.gnolia is a decent sized site. Maybe they did have a decent infrastructure and it will only be a little while before they've gotten everything back. Of course, after the fiasco with the blog site that used RAID as their backup system, I've started to think that many people don't take data as seriously as it needs to be taken.

timf11• 17 years ago

I like the idea of the offsite slave. I was also reading about another idea, forcing a slave delay to recover from oopses/corruption:

http://www.rustyrazorblade.com/2008/05/07/mysql-time-delayed...

eli8• 17 years ago

I do something like that for my outgoing email ;)

t0pj• 17 years ago

"take some simple steps to ensure that you don't bork your site"

unborked[.com] is currently available.

A site dedicated to stopping all of the borking going on out there.

===

I think some cheap, simple sql dumps from the db, burned to an external HD and taken home would've been a good first step -- your data would be dated but somehow restorable nonetheless without the need for off-site, replicated, heart-beat monitored databases.

IMHO you really need an expert database admin (of which I am not) to get all of this up and keep all of it going; a caretaker who's only job in life is the maintenance of any and all persisted data.

On another note, backups are really useless unless you have actually tested the restores.

jacquesm2• 17 years ago

what does 'burning to an external harddrive' do ?

is this some new tech that I'm unfamiliar with ;) ?

bprater3• 17 years ago

I'd really like to store incrementals of my MySQL at S3. Anyone using something they really like? Or should I just break-down and do full dumps each night?

fendale• 17 years ago

Then you can still lose a days data. Why not switch and ship the mysql binary log every x minutes? Last backup plus all the logs gives you much better recoverability

mdasen4• 17 years ago

Tarsnap.

Tarsnap lets you say: tarsnap -c -f backup01302009 mysql_dir/

And you can just adjust the date each day. It gives you the luxury of a full dump (anytime you want to restore, just reference backup01302009), but it only actually stores the deltas (making sure not to duplicate data that might be in backup01292009 or backup01282009 and so on). Tarsnap stores the data to S3 so that it's replicated in multiple data centers.

It costs a little more than S3 at 30 cents per GB, but it's metered out so that if you only use 1MB of storage, you'll only be charged 0.03 cents for that storage. You could try creating your own way of doing incrementals, but I doubt you'd get it as efficient as Colin (the math genius behind Tarsnap) and so I doubt you'd get it cheaper. Plus, this way you don't have to deal with it.

And remember, it's hard to fill up a database.* As the Django Book notes: "LJWorld.com's database - including over half a million newspaper articles dating back to 1989 - is under 2GB." So, if they were using Tarsnap, they might be storing 5 or 10GB tops at a whopping $1.50-$3 per month plus whatever the transfer of their deltas was for the month. Oh, and tarsnap compresses the data too. So, maybe they'd be paying $1 or something lower.

* Clearly, if you hit it big time, you might not want to continue paying for tarsnap. However, if you become the next big thing, you can hire someone to deal with it for you.

eli8• 17 years ago

Depends on the data and your budget, I guess.

Disk space is cheap -- I do a full dump of the database nightly (and a separate system dumps a few key tables every 15 minutes)

anuraggoel• 17 years ago

Reminds me of the Couchsurfing fiasco back in 2006.

http://www.techcrunch.com/2006/06/29/couchsurfing-deletes-it...

http://www.couchsurfing.com/crash_page.html

Couchsurfing had a backup process in place, but failed due to human error. It's highly likely that magnolia had backups of some sort. Before we all start berating magnolia (and website owners in general) for not backing up their data, we need to know more about the real reason for this failure.

Couchsurfing was able to come back from the dead because of limited competition. In this case, unfortunately, it will be easier for users to look elsewhere.

jodrellblank5• 17 years ago

Reminds me of LeafyHost.

(Summary: Small hosting company advertising multiple backups; their single server crashed and they had no backups at all. Long story involving terrible customer service ensues).

eli8• 17 years ago

You're right that we probably shouldn't jump to conclusions, but it doesn't look good.

And human error shouldn't really be possible if your backups are automated and you periodically do test restores.

timf11• 17 years ago

I'm planning on burning time on building a full failure solution. Records snapshotted at least daily and any single node/service can die entirely and there is an exact, tested manual recovery checklist or automatic rollover option in place for each permutation.

This runs counter to the more cavalier "release early, polish later" advice I keep seeing. Maybe I am doubly freaked out because the things I'm storing are not easily recovered or re-imported by the users themselves or any kind of algorithm/redux.

bprater3• 17 years ago

It'll be a time sink. Why not do it as dirty as possible now and go back and tune it as time goes on?

timf11• 17 years ago

See the original submission, that's why not :-)

I want to not only have a backup scheme but also make sure it's restore-tested. Maybe I wasn't totally clear, not planning on a beautiful failover in each place in the beginning (planning failover for the DB at least). Just a tested (even if manual) restore procedure in each situation.

jodrellblank5• 17 years ago

It also runs counter to "do the simplest thing that could possibly work" and "KISS" and "YAGNI".

Doesn't mean it's a bad idea though. But if it's that good, make sure you announce it and market it as a significant feature...

palish• 17 years ago

I wish Twitter allowed comments on tweets. Companies (like Magnolia) use it for important announcements / status updates, and yet there's no way to see feedback from the community.

tptacek• 17 years ago

That's what search.twitter.com is for.

timf11• 17 years ago

http://search.twitter.com/search?q=%40magnolia

jodrellblank5• 17 years ago

Hope Twitter have backups...

nickb• 17 years ago

From reading the notice, I get the feeling that they don't have any backups! Not very re-assuring at all.

bprater3• 17 years ago

Gut wrenching, for sure. To see your whole model explode, it's just not worth it.

If you don't have a backup plan -- close your browser, stop surfing Y!Hacker, and go write a shell script to dump your database and rsync to anywhere else on earth.

eli8• 17 years ago

...and then do a test restore!

jacquesm2• 17 years ago

amen to that! You should see the confused looks I get when doing technical due diligence somewhere and they get that question: "when did you last try to restore a backup ?"

then once they get the point they want to leave the room asap...

rarrrrrr3• 17 years ago

YES! This comment should be modded up to the top.

We talk to a lot of customers about their backups every day. I'd say more than half of the backup failure stories we hear come from failing to adequately test that data can actually be restored.

Some have been comically tragic. Like creating an offsite backup using encryption keys that are only archived as part of said backup... :(

jodrellblank5• 17 years ago

I doubt Google have backups, take a copy home on a CD, have a massive third party database with dedicated certified people, or backup to Amazon S3.

I suspect their file storage system is simply good enough to replicate data intelligently across many machines/sites like a giant RAID array.

I also think that it's a truly massive benefit they have over other companies, particularly small companies, and that it's rarely discussed as such.

eli8• 17 years ago

Magnolia ain't exactly indexing every page on the internet though, ya know?

I agree with you that that is one of Google's key strengths, though. PageRank is neat, but the real edge is that the site is fast and the infrastructure is based on commodity hardware, so it's cheap.

jodrellblank5• 17 years ago

The real edge is not that it's based on commodity hardware, but that it's based on scalable low-management software (I imagine all this, and skimmed their Google File System paper once. I don't work for Google).

It wouldn't matter if they were blades or Sun UltraWhatevers, the real benefit is that if one dies, nobody need panic - plug another in and "the system" will rebuild it. If a rack dies, plug another in and the system will rebuild it. If a datacenter dies, the others will cover for it.

No backup tapes, no manual tweaking each new build.

eli8• 17 years ago

What gets me is how people even do normal development without some sort of backup -- even an informal one.

Maybe I'm just lucky enough to work with relatively small datasets, but our QA server contains a complete copy of the production database and is at most a week old. So even if we didn't run any other backups (which we do), we could restore from that server and be back up and running within hours.

billturner• 17 years ago

They have a page/section at GetSatisfaction, but it doesn't seem to be updated anytime soon (though there are 3 employees set to answer questions): http://getsatisfaction.com/magnolia

raghus• 17 years ago

Early on the West-coast morning of Friday, January 31st, - poor guys must be frazzled. Jan 31 is Saturday.

seldo• 17 years ago

Or else this is a dire warning from the future! There's still time to save the data; this accident won't happen until tomorrow!

rarrrrrr3• 17 years ago

The trouble with convincing humans to back up is that it's important, but not urgent. For any given day of procrastination, the likelihood of consequences is small.

Our motto is "If it's not backed up, it's down." (Applies doubly so for us as a backup company.) For sufficiently seasoned administrators, non-redundant storage should cause sleeping difficulties.

Locke3• 17 years ago

It's easy to say, "What? No backups... how stupid can you be?" But, without knowing the particulars, I wouldn't necessarily jump to that conclusion. It's very easy to have regular automated backups fail in some way.

Unfortunately, it's not enough just to have backups. You have to actually verify that they're correct and up-to-date. Verification is easy when your database is small (for example). You can just load it in your development environment occasionally. But, how do you verify your backups if you have hundreds of gigs or even terabytes of data?

As an example, I've seen cases where backups were successful every night... but, they were being run against a slave db and replication had failed. The result: Excellent backups of weeks old data.

timf11• 17 years ago

That's nasty. Maybe you could automatically inject test data into a fake account and run test queries on the backup. That will not verify you are getting everything but at least that you got something recent.

Locke3• 17 years ago

Verifying backups is almost always many times more difficult than setting up the backups themselves. Checking that replication was in fact working turned out to be fairly easy to automate (although, short of checksumming all the records in all the tables you can't really be 100% sure everything is okay).

That's the thing with backups, they can fail or become corrupt in many ways. If you don't use them for something on a regular basis or have some regular verification process you may not know what you have until it's too late.

And, of course, I've also seen situations where subtle data corruption in the master database leads to weeks of subtly corrupt backups. By the time the corruption was discovered we faced the choice of rolling back several weeks to the last good backup or fixing the corruption. In the end we had to do a kind of merge -- it was a real pain.

silvestrov• 17 years ago

A backup is only a backup when you have restored from it successfully. Until then, it's no better than garbage.

A disk raid is only a raid when you have tried pulling a disk and having the raid successfully rebuild (while running in production!).

I once had a system administrator who didn't dare pull a raid disk while in production. After that, has wasn't a sysadm in my eyes, he was a lottery gambler.

Being paranoid doesn't mean nobody is following you.

rarrrrrr3• 17 years ago

An easy way to have a space efficient, perpetual backup of a database:

Run the dump file every N units of time.

Compress the dump file using gzip with the --rsyncable option. This increases size by 1% but makes it efficiently diff/patchable.

Use diff to make a patch going back to the previous compressed dump of the file. Keep the patch, discard the previous dump if you like. You now can apply the patches in succession to go back however far you like.

Finally, and most importantly, use par2 to store parity along with your backup files to protect against silent bitrot.

Note: --rsyncable is in newer gzip versions. It's too new to be included in OS X Leopard.

yeahit• 17 years ago

Wouldnt these two solutions be even nicer?

* Backup the Binlog of the DB (At least its called Binlog in MySql)

* Make a diff of the uncompressed dump and zip that. This diff would even be human readable.

philfreo• 17 years ago

... glad I'm on del.icio.us, but going to export all my bookmarks now anyways.

timf11• 17 years ago

Let's say you had an insane amount of bookmarks (or something precious like it). Would you pay to have DVDs burned and sent to you?

It seems like a convenient re-assurance for users to get DVD backups mailed to them every so often -- but it would be a total pain to support this as a small business...

timf11• 17 years ago

For example: http://www.smugmug.com/help/backups

But how do I offer this when we're in colocation or slicehost etc.? Can't be driving to the colocation center every time someone wants a DVD...

joshu3• 17 years ago

I have an insane amount of bookmarks. I can still back them up in a few seconds via wget.

Of course photos and whatnot are much harder.

tlrobinson• 17 years ago

I don't know if I'd pay for just my list of bookmarks (I don't have that many), but I might pay for a monthly DVD archive of the site's content for all of my bookmarks, plus a searchable index. I hate when I can't find a bookmark, or a bookmark has gone offline.

rbanffy• 17 years ago

A burned DVD full of bookmarks is, indeed, an insane amount of data.

At about, say, 2K per bookmark, that's a couple million bookmarks. If one bookmarks a site every 60 seconds, 24x7 that's about 4 years of work. A slightly more reasonable person, bookmarking every 60 seconds for only 8 hours a day, would have to be bookmarking websites continuously since about 1996 to fill up a DVD.

vaksel• 17 years ago

sounds like the same thing as Journal Space http://www.techcrunch.com/2009/01/03/journalspace-drama-all-...

sam_in_nyc2• 17 years ago

Does anybody use this site? I can never, ever remember the URL, and that bothers me a lot.

The way I see it, there are too many social bookmarking sites out there. If there's now one less, I wouldn't mind.

decadentcactus• 17 years ago

Well, I finally did it. I stopped what I was doing, and worked out my backup strategy, which includes (finally) getting a script put together, and now to adjust it to email me copies and also (contemplating) using S3 to store copies. Maybe also box.net, depending on a few things.

acangiano• 17 years ago

Backing up is like testing, everyone knows that they should do it, but very few people actually do. That said, I'm sorry to hear about their troubles. They are good people genuinely trying their best.

critic• 17 years ago

If YC crashes and nulls my karma, there's gonna be a lawsuit!

ahoyhere• 17 years ago

We worked out our backup strategy for our time tracking system (http://letsfreckle.com/) while in beta. While we're still small, we do full db dumps to off-site, once an hour.

This does not require a lot of sophistication. And yes, we might have some downtime while we restore in the event of a catastrophe, but we do have the data.

I can't even imagine the nerve of shipping a product to the public that doesn't have even something this rudimentary in place.

Ma.gnolia.com crashes hard -- no backups?

Related Stories