jhorey

Yes, plus some sort of analytics-focused data warehouse (Teradata, etc.). I am almost certain though their analytics team is outsourced to a major contractor like Lockheed Martin.

I forget the exact numbers, but a single year's worth of Medicare part D claims data will be on the order 1TB. That doesn't include the beneficiary and provider datasets (which links patients and doctors) which you'll need to join against. Also when detecting fraud like this, you may want to include the other Medicare parts (A, B, C) which are oftentimes larger than part D (being that D is the newest). So this leaves you manipulating on the order of 10TB for single year analysis. Finally, since Medicare bills can be corrected up to 3 years, you may end up joining multi-terabyte datasets.

The comments here do a pretty good job explaining the technical details and common use cases. If, however, you'd like a more narrative-based explanation, you might want to check out my PyData conference presentation (http://bit.ly/1l1PPSe). It's a bit long (about 20 mins), but I try to explain the circumstances in which Docker makes sense.

I would imagine something like Cassandra scaling better (both on the reads & writes), but I'd suspect you'd have to change your data model quite a bit to see those benefits. If you're interested in this line of inquiry, shoot me an email 'jlh' @ 'opencore.io'.

If you're using Vagrant, you'll have to manually shut off the VM via `vagrant halt`.

If you're just trying to get started with Docker, Boot2Docker (https://github.com/boot2docker/boot2docker) is probably the way to go.

If you're looking for an Ubuntu-based Vagrant box, you can try out my Vagrant box (http://ferry.opencore.io/en/latest/install.html#os-x). It's based off of 14.04. Be warned though, my box is very large (~4GB) as it contains many pre-baked images (Hadoop, Cassandra, etc.).

Here's mine: Ferry is a tool to help developers create (virtual) clusters on their local machines using Docker. Ferry supports Hadoop, Cassandra, Spark, and GlusterFS/Open MPI.

GitHub: https://github.com/opencore/ferry

I'd love help with:

Examples using these big data stacks. Examples can consist of simple Dockerfiles.

YARN/Spark integration. Currently Spark runs in stand-alone mode, but I'd prefer it to run as a YARN service.

Ability to support customized backend stacks so that users can include packages, etc.

If you're interested, please file issues on GitHub or shoot me a message on Twitter (@open_core_io).

For those know a bit more, is this basically equivalent to CQL for Cassandra except for HBase?

It's interesting, my experience with companies here in the U.S. (mainly SMBs) are that they feel much more comfortable using a SaaS compared to adopting an internal NoSQL store. Granted most of these companies aren't highly technical, but I suspect for them using a SaaS is easy and doesn't require much internal expertise. Adopting a new NoSQL store though may require training, etc. Of course I concede that things may be different in larger enterprises and companies outside the U.S.

I would bet, however, that most of the clients that do business with Mongo or DataStax still intend on using/maintaining a relational system. I haven't really encountered many companies that's decided to completely dump their relational systems in favor of something else.

I agree that's a bit short sighted, but I think it's one of perspective. The mistake most people make is to believe that Oracle (and other relational DB vendors) are competing directly against the various NoSQL stores (this includes doc stores, scalable key-value stores, etc.). I don't think that's true. I think that relational databases are really competing against industry-specific SaaS. So instead of implementing their own database for inventory & sales, companies may opt to use a service. These SaaS companies, in turn, are more likely to adopt a variety of technologies (including NoSQL and relational). Since the SaaS companies serve more than a single company, they're also more likely to adopt easily scalable technology like Cassandra. So fewer businesses will need to purchase databases (of any sort), but the ones that do will have greater scalability needs.

If you want a relatively simple way to provision a Hadoop cluster locally, you may want to try out Ferry (http://ferry.opencore.io). It's based off Docker, so in theory, you could also write a nice Dockerfile to deploy Mortar's recommendation engine. (full disclosure, I'm the author of Ferry).