Show HN: Arvados, a free software storage and compute platform for big data

https://github.com/curoverse/arvados

by tetron • 10 years ago

6 5 10 years ago

github.com

tetron3• 10 years ago

Hello all, Arvados developer here to answer your questions.

brudgers2• 10 years ago

How does Arvados differ from other big data tools?

It looks to be focused mainly on bio-informatics, do you see other use cases where it's features provide more leverage than general purpose tools?

tetron3• 10 years ago

There's several features that make Arvados unique.

* The content addressed storage system references hashes all data so you can unambiguously reference an immutable data set, similar to git but capable of handling huge invidual files (hundreds of gigabytes) and scaling to petabytes.

* Every compute job is recorded in a database with hashes identifying the inputs, Docker image, and outputs, so re-running past jobs is easy.

* Designed to federate multiple instances, to support both "hybrid cloud" setups within an organization, and allowing different organizations to share data.

These are all features that are particularly important to the bioinformatics community, but solve problems that are common to lots of informatics big data problems.

Show HN: Arvados, a free software storage and compute platform for big data

Related Stories