Big Data Debate: HBase

http://www.informationweek.com/software/enterprise-applications/big-data-debate-will-hbase-dominate-nosq/240159475

by jeromatron • 13 years ago

42 54 13 years ago

informationweek.com

knodi• 13 years ago

Does it matter?

hayksaakian• 13 years ago

Blatant link bait

http://en.wikipedia.org/wiki/Betteridge's_law_of_headlines

duaneb7• 13 years ago

NoSQL is my least favorite buzzword of the decade. It means absolutely nothing but being counter cultural to... SQL culture. Many NoSQL "databases"—and I use that in the lightest term possible—even have SQL engines.

monstrado12• 13 years ago

Agreed, and some of these databases don't even embrace it. For example, HBase's main website makes no mention of being a "NoSQL" database (http://hbase.apache.org/).

Of course, given the right circumstances, you could totally use SQL to talk to your data which lives in HBase, but that doesn't make it a SQL database.

SQL is a query language, a database is usually made up of many different components, one of them usually being a query engine. Look at Impala (https://github.com/cloudera/impala) which decouples itself completely from the storage engine layer, it can query data CSV data, doesn't mean it's a SQL database.

otterley• 13 years ago

How to make a simple NoSQL database:

(1) Install MySQL.

(2) Create a database called "nosql".

(3) In the "nosql" database, create a single InnoDB table called "data" with 2 columns: a BLOB primary key "mykey" and a BLOB column called "myval".

(4) Write a thin wrapper over your favorite mysql client library implementing "get" and "set" operations (which are translated to SELECT and UPDATE statements).

(5) Enjoy your ludicrous speed.

monstrado12• 13 years ago

If you're comparing this to your riak, redis, tokyo cabinet, ... database than most likely.

With HBase, one of it's bread and butter operations is the start and stop row scan (otherwise known as a range scan). The only thing equivalent I know of in MySQL is using windowing functions, and even then I don't think that's an appropriate comparison.

I hear you though, most people equate NoSQL to some sort of KeyValue store and that's it.

e12e• 13 years ago

I prefer the (backronym?) Not Only SQL - as it better describes what NOSQL can help with -- as opposed to using only a normalized relational data store.

PaulHoule2• 13 years ago

I laugh at mongodb, but every time I've seen a shootout between Hbase and Cassandra, Cassandra has won.

monstrado12• 13 years ago

Could you please elaborate? It totally depends on use case...

PaulHoule2• 13 years ago

(1) I've seen more than one project failure involving mongodb,

(2) Every time I've done a performance/features shootout of mongodb vs. other projects, mongodb doesn't just lose, it comes back with two black eyes.

rjurney2• 13 years ago

MongoDB is actually great to use with Hadoop as well. Laugh away...

zzzeek• 13 years ago

NoSQL, what's that, is that like Big Data ?

dallasmarlow• 13 years ago

NoSQL is only a stepping stone to NoQuery, it's going to be awesome

capkutay2• 13 years ago

some people even describe hbase as nosql, because its not sql right? unless you use cloudera impala, which is sql, but impala is supposed to kill sql too. in the end no one wins.

monstrado12• 13 years ago

impala is supposed to kill sql too

I'm not sure what you mean by this, Impala is in no no way trying to replace "SQL". Impala is a general purpose distributed query engine that currently translates SQL to a query plan.

capkutay2• 13 years ago

I was commenting in the theme of tc-style enterprise tech journalism but thank you for clarifying.

m0nastic2• 13 years ago

At the risk of hijacking what seems like a linkbait article, I'll ask folks here a question:

Does anyone have good experiences/recommendations for storing a "reasonable" amount of unstructured logs/pcap files. By "reasonable", I mean, not petabytes (maybe a couple terabytes, over time).

I ask, because I keep thinking something like HBase is overkill (although one of my alternate solutions is to run an internal Openstack Swift cluster, which seems like pretty much the same amount of hardware/engineering).

If I could, I'd just send it to the cloud (to the cloud!), but I need it to be local and internally controlled (however, the developer niceness of S3 or Azure blob storage is what got me thinking about just making my own Swift object store).

monstrado12• 13 years ago

Disclaimer: I work at Cloudera as a Tools Developer

What do you mean by unstructured? Do you mean the data has yet to be parsed into a format which could be logically grouped into columns? Or do you mean that it's deeply nested?

Since log data doesn't really change, it might be overkill to use something like HBase (or any database for that matter). On the tools team at Cloudera, we've found that writing the data into HDFS and using Impala to analyze it works pretty well.

We typically analyze chunks of log data and then ingest it into HDFS (due to the use case), but if you're looking to ingest data in "real-time", you'll want to use something like Apache Flume.

With the data separated into partitions, we're able to run queries that analyze GBs of data in under a second (15 nodes). This is log data (LOG4J) that has been extrapolated into columns, and then loaded into a columnar storage format (RCFile, soon to be Parquet).

Let me know if you have any questions, glad to help.

m0nastic2• 13 years ago

Thanks, I've been testing out a bunch of hare-brained schemes and it didn't even occur to me to just use HDFS directly (and seeing you and karterk both suggest it helps).

Basically, at a high level, the system I'm working on aggregates and processes security information (It's a SIEM, if that product category means anything to you). At the point the logs get ingested, the server determines if they're "actionable" (which is determined by rules I load into Redis), in which case it parses them and stores them in a Postgres event table; or "not individually actionable, but may cause an action in conjunction with some other log" that I want to just store somewhere for batch processing.

I don't really need to tokenize those logs, as at the point I care about them I'm just going to be searching through them. So, they're "unstructured" in the sense that there's about 15 different collection points, each with it's own format (many just an ugly facsimile of syslog with some JSON in the middle).

So, I think your suggestion will work out very well.

Thanks again.

karterk3• 13 years ago

Store it in HDFS as sequence files. HBase (as the article mentions) is a beast to manage with many moving parts. But, HDFS is just rock solid and very stable. If you want to analyze logs, you can do so in a batch mode (I recommend using Scalding or Pig). If you want it to be more "searchable", then take a fraction of it using a MR job and index it in Solr or ElasticSearch.

trun• 13 years ago

Linkbait aside there are some reasonable points being made here, specifically "Failover means downtime" about HBase. We run into this pretty much every day in one of our customer facing applications or APIs and it's quite frustrating to have to explain that there's very little you can do to prevent it.

I haven't looked too closely at MapR yet, but "instant recovery, seamless sharding and high availability" are impressive claims. It's still decidedly differently than HBase in my mind given the cost.

monstrado12• 13 years ago

Which version of HBase are you running? If we have a node or two go down in our cluster, HBase is completely unaffected. Recoveries can take up to a few minutes, in the old ages it took hours.

I know that MTTR (mean time to recovery) is being worked on together by several large companies to get down to seconds.

jeremiahjordan• 13 years ago

"Minutes" is not "unaffected" if your site is now down.

samspenc3• 13 years ago

Also, if you use the Thrift service as an interface to HBase (we do), you can have it running on several machines on the cluster and be able to set them up using a failover architecture - as long as the cluster itself is healthy, Thrift should work from any machine you have it running on.

karterk3• 13 years ago

We use HBase pretty extensively and I have mixed feelings about it. On one hand, it's very clunky. Though the documentation is pretty good these days, setting up and managing a HBase cluster on production and in scale involves tackling many moving parts. Setting up Cassandra is almost a joke in comparison.

HBase-Hadoop integration is great, but Cassandra has caught up significantly on that front. If you want strong consistency, apart from HBase there is no other (non-sql) solution that's really battle tested. If you're okay with eventual consistency, you should take a hard look at Cassandra.

threeseed2• 13 years ago

I don't understand why people keep thinking that Cassandra is only eventually consistent.

It can be set to any consistency level you like for both reads and writes: http://www.datastax.com/docs/1.1/dml/data_consistency

jbellis5• 13 years ago

Even lightweight transactions in 2.0 for linearizability: http://www.datastax.com/dev/blog/lightweight-transactions-in...

duaneb7• 13 years ago

It doesn't even have transactions, I'm not sure how 'consistency' is even in the conversation.

EDIT: Just saying, failures and partitions are extremely difficult to deal with without atomic operations.

linuxhansl3• 13 years ago

And then you get slow writes and/or reads as multiple nodes need to be contacted, as well as temporary outages when a quorum cannot be formed.

karterk3• 13 years ago

I'm aware of consistency levels. But in general HBase performs much better out of the box when compared to tweaking Cassandra to be be more consistent. In other words, if I wanted consistency badly and at scale, I would go with HBase.

rjurney2• 13 years ago

Interesting that neither person has anything to do with Apache HBase. Both are in a position to belittle it.

samspenc3• 13 years ago

Heavy HBase user here. My two cents, FWIW.

I totally identify with both sides of this article, but if we had to do this again, we would probably go for HBase again. Its quite a pain to manage unless you have someone on your team with a PhD in HBase, but:

1. The main HBase committers are also the ones who contribute to the Hadoop project, so its in a forward trajectory with good velocity, since its fairly coupled to the Hadoop ecosystem. Cloudera is a big contributor, as are Facebook and Salesforce.

2. Facebook, Adobe and other BigTM companies use it in production and at scale. (Granted they have armies of smart people to maintain HBase.)

3. For all the pain it is, its reasonably documented and has a growing community that fills in the gap. I could be totally totally wrong about this, but Cassandra doesn't have that same level of community as HBase does.

jbellis5• 13 years ago

/author of the "con" position here

I can see why you might come to those conclusions -- a lot of people with their heads down in Hadoop just don't realize that there's a world outside HDFS. Not saying that in a mean way; that's just the way it is. If you're involved in that ecosystem, there's enough to keep up with without researching what others are doing.

1. True enough, but HBase tends to be an afterthought for the datawarehouse-focused Hadoop players. You see this manifest in a bunch of ways, but as just one example: when I evaluated HBase at Rackspace 4 years ago, they were exploring options for secondary indexes. They're still exploring.

2. Yes, you really do have to have expert-level knowledge of the internals to deploy it successfully. And there's a LOT of those internals, by design (hmaster, ZK, regionserver, and close ties to the HDFS infrastructure). Cassandra is much simpler, much easier to deploy and troubleshoot. When Cassandra gets evaluated vs HBase, it tends to win [1], but often HBase is the "default" choice because of "Hey, we already have HDFS" thinking. This is changing as awareness grows of alternatives.

3. I think that's the tunnel vision I mentioned. By any metric I can think of -- conference attendance (over 1100 at the Cassandra Summit in June), IRC activity, StackOverflow participation, jobs advertised -- the Cassandra community is larger and more active than HBase's. Here's a list of some of the users: [2].

Give it a look, I'll be happy to answer questions!

[1] http://vldb.org/pvldb/vol5/p1724_tilmannrabl_vldb2012.pdf [2] http://planetcassandra.com/Company/ViewCompany?IndustryId=-1

samspenc3• 13 years ago

Thanks for the reply!

1. True, the fact that we were deploying Hadoop was a big reason to go with HBase. Despite the challenges, it was a case of "better the devil we know than the one we don't."

2. Agreed, I played with Cassandra, but again, point #1 carried the day.

3. That's great! I was at HBaseCon 2013 and that only had 750-850 people. I concede this point. ;) Sorry I didn't explore this further, that was my bad.

[EDIT] One thing we like is that HBase re-partitions data really fast since data is in HDFS. Not sure how well Cassandra holds up there.

linuxhansl3• 13 years ago

Oh please.

On the one side we have a commercially competing entity arguing against HBase and on the other side we have an individual "defending" HBase who never contributed a single line of code to open source HBase promoting their own closed source solution.

HBase is a "Sparse, Consistent, Distributed, Multidimensional, Sorted map" and at that it is pretty good.

Other stores are either not consistent or not sorted by default (which means you cannot do range scans). Some can be configured to do that, but then suddenly all those nice claims made by the commercial entities backing them just vanish.

If you do not need consistency and range scans, then do not use HBase. If you do need those HBase will be a excellent fit.

Some of the most heavyweight entities on this planet have committers to HBase (Facebook, Intel, Salesforce, to some extend Twitter, etc) as well as commercial backers such as Cloudera and Hortonworks.

Disclaimer: HBase committer here.

dialtone• 13 years ago

Agreed. Out of the box none of the other storage systems actually allow you to not worry about range scans at all. While HBase will repartition your data to balance the used disk space automatically, in most other systems you'll end up having to manually partition your data in different ways and then aggregate the queries across partitions.

This is pretty literally the main reason why many companies use HBase. Then in the same companies you can also find a good usage for Cassandra but it'd be a different use case.

Big Data Debate: HBase

Related Stories