Moving 6 Billion Messages Without Being Noticed

http://dt.deviantart.com/art/Moving-6-Billion-Messages-Without-Being-Noticed-285571516

by kachnuv_ocasek • 14 years ago

89 9 14 years ago

dt.deviantart.com

waitwhat4• 14 years ago

They needed the Percona toolkit (previously Maatkit). http://www.percona.com/doc/percona-toolkit/2.0/pt-table-sync...

kfool3• 14 years ago

The toolkit documentation reports limitations.

What they really needed is ChronicDB http://chronicdb.com

waitwhat4• 14 years ago

ChronicDB is a hosted solution. That isn't even remotely what DeviantArt were looking for.

newman314• 14 years ago

If you are going to flog ChronicDB, please at least disclose if you have some sort of relationship with said company. Most of your previous submissions pertain to ChronicDB.

banksyw• 14 years ago

Disclaimer: I wrote that article.

Percona toolkit has some useful stuff. Thanks for the link.

In this case our solution ended up being pretty much the same as pt-table-sync except that it made use of our distributed task queuing system so was MUCH faster than if we had used a single threaded perl script. (Up to a few hundred chunks of rows being checksummed/synced in parallel.) I didn't talk about this aspect directly in the article to keep it focused.

Not sure if it would be possible to use pt-table-sync in that way, certainly would have taken a lot more work to get a third-party perl script to distribute the work through our job queue.

Moving 6 Billion Messages Without Being Noticed

Related Stories