tlipcon

Colossus metadata is stored in Bigtable: see https://cloud.google.com/blog/products/storage-data-transfer...

Spanner stores its data in Colossus, so there would be some bootstrapping issues to resolve to move it to Spanner over Bigtable. (Bigtable also has the bootstrap issues but has solved them already and there are additional difficulties due to some details that I probably am not at liberty to share)

Spanner is used for metadata for many other very very large storage systems, though.

Just a small note: Kudu shares no code or provenance with Hadoop. Only commonality is ecosystem (eg you can use spark or Impala to query it)

For those interested in a technical overview, http://kudu.apache.org/kudu.pdf is our academic-style (but not submitted to any journal) paper.

Apache Kudu project founder here:

It's true that the project was initially developed at Cloudera, and employees continue to be the main driving force behind development. That said, we have committers and contributors from other companies as well. Roughly half the people who contributed a patch in the last 3 months have been non-Cloudera. Additionally we are very strict about doing all development upstream (eg with the first open source release we spent a lot of effort to open the entire development history going back to 2012, including JIRA, git, etc).

As for users, here are a couple examples off the top of my head who aren't currently paying for any support:

- Xiaomi (world's 4th largest smartphone maker) collects ~2TB/day of event data from >5million phones into a cluster which simultaneously runs analytics workloads (SQL, Spark, etc) - CERN is looking at using Kudu to store high energy physics experiment data from the ATLAS detector at the LHC. You can find some code at https://gitlab.cern.ch/zbaranow/kudu-atlas-eventindex and a poster here: https://indico.cern.ch/event/505613/contributions/2230964/at...

(of course lots more too whose names I dont have permission to mention)

Feel free to join our slack if you're interested in chatting with more - usually plenty of people online here: https://getkudu-slack.herokuapp.com

-Todd

Hopefully this isn't too "pitch"-y, but: if you're looking for a database that's good at time series, will always be open source, and does support scale-out and HA, you might be interested in Apache Kudu (incubating).

Feel free to drop by our Slack (http://getkudu-slack.herokuapp.com ) if you have any questions.

Suggestion: add an option to look at file contents and analyze cross-file dependencies (eg c/c++ #includes or java imports)

Todd from the Kudu team here if anyone has any questions, I'll check back throughout the day (or tweet at @ApacheKudu)

FYI on the date issue, lest anyone think we filed the patent trying to steal the work done by others, the patent application says:

"This application claims to the benefit of U.S. Provisional Patent Application No. 61/911,720, entitled “HYBRIDTIME and HYBRIDCLOCKS FOR CLOCK UNCERTAINTY REDUCTION IN A DISTRIBUTED COMPUTING ENVIRONMENT”, which was filed on Dec. 4, 2013, which is incorporated by reference herein in its entirety."

(which predates the creation of the cockroachdb repo and the hybrid logical clock paper).

Agreed -- personally I'm against offensive use of patents like this as well, and it's my understanding that Cloudera doesn't intend to use this patent offensively. If it did, I would be upset and would consider leaving the company - I know many other employees feel the same way. The reason I agree to help write patent applications as an engineer is that I've seen the distraction and damages caused by patent trolls (or even other companies) and the importance of having a defensive portfolio.

Disclaimer: Obviously I'm not speaking for the company or making any promises here :)

-Todd

Kudu uses a similar algorithm which we call HybridTime. You can read the tech report here: http://pdsl.ece.utexas.edu/david/hybrid-time-tech-report-01....

and the source here: https://github.com/apache/incubator-kudu/blob/master/src/kud...

Both are probably more readable than patent-ese :)

Typically QSBR algorithms don't require blocking the world, or even blocking any single thread. They just require each thread to periodically check in and run a bounded amount of code which amounts to "hey, I'm not currently looking at the map".

Some other background collector thread (which is going to actually delete removed objects) just has to wait until it sees every mutator thread cross a safepoint, at which point it knows that none of those threads could be hanging onto references that have been unlinked from the data structure.

I'd recommend reading some surveys of RCU and SMR algorithms if this stuff is interesting to you.

Also worth noting it's an open source project so if you're interested in contributing in this area, we'd love to have you on board.

We've just updated to C++11 as of a couple weeks ago. Partially the issue was when it started, partially the issue is that we have to support older platforms and we have a C++ client library to worry about.

We have a few entries in the FAQ about this: http://getkudu.io/faq.html#what-is-kudus-consistency-model-i... as well as some more in-depth docs: http://getkudu.io/docs/transaction_semantics.html

Basically the short answer, though, is that reads are not done by quorum, but the client can specify if they need "up to date" reads. If they do, currently, we force reads to go to the leader, but have a roadmap for how to read from other replicas (see the Spanner paper for a rough idea how that can work).

It doesn't yet. It's on our nebulous "we'd like to do this some time" roadmap, but currently concentrating on some more basic stuff around stability and time series features.

Of course this is a huge optimization for data warehousing applications, where two co-partitioned tables can be joined without any network data transfer, and in some cases could even use merge join instead of hash based strategies. But, it's the usual time/scope/quality trinity, and we'd rather not compromise the third element.

You can also join us on Slack here if that's more your style: http://getkudu-slack.herokuapp.com

Thanks, Alex! Appreciate the kind words.

Todd from the Apache Kudu (incubating) team here. I'll check this thread throughout the day in case there are any questions (and try to check the original post for comments as well).

Yep, I've been taking part in those design discussions. We hope to have Kudu tablet servers support generating this in-memory format in shared memory as the result of scans, so the Impala server (client from Kudu's perspective) can directly operate on the data. We're expecting a 20-30% speed boost from this for some queries, though haven't done any tests at scale of the prototype.

Yep, that's correct. HDFS+Parquet is more accurate but doesn't fit quite as well on slides and short descriptions.

The idea is to get the analytic scan performance of Parquet while still allowing for in-place updates and row-by-row access like HBase.

HDFS (with Parquet or other formats) will still be better for unstructured or fully immutable datasets. HBase will still be better when your top priority is ingest rate, random access, and semi-structured data. Kudu should be good when you've got tabular data as described above.

9 worker nodes plus one master node. Each with 48GB of RAM, 3 data disks, and dual 4-core Xeon L5630 processors at 2.13GHz.

So, pretty old stuff. But note that the 100G dataset fit comfortably in RAM. If you don't fit in RAM, then the latencies are obviously going to be much worse on spinning disk. Our bet is that for OLTP workloads, you're probably going to be on flash or mostly RAM-resident these days, given the dropping prices and increasing speeds of solid-state storage.

Quick example of how TSAN makes it easy to understand and fix races -- here's a commit message which shows its output: https://github.com/cloudera/kudu/commit/e402d5ed79a9c98283b6...

Apparently Google has this same tool for Java internally, but hasn't open sourced it due to their litigation with Oracle over Java stuff.

I spent a lot of time in 2011 or so struggling with GC on the JVM: http://blog.cloudera.com/blog/2011/02/avoiding-full-gcs-in-h... has some of the gory details. Even hacked a bit on G1: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2011-A...

With a lot of effort by many folks in the community, HBase has mostly tackled the full-GC problem, but still has occasional issues with some workloads.

So, GC was definitely one factor - not having GC means we can give 99th percentile numbers in the single-digit milliseconds, which is pretty nice. Our master process actually has shown <1ms 99.99th percentile for tablet location requests on an 80 node cluster. So again, numbers that are super difficult to get on the JVM unless you take an allocation-free approach like the HFT guys do.

Another factor was ease of integration of platform-specific code for performance reasons. For example, we make use of SSE prefetch instructions to improve scan speed in our concurrent B-tree by 30% or so. The b-tree itself would be difficult to implement in Java due to lack of control over object layout, etc. While you can eventually get the same performance with enough off-heaping and sun.misc.Unsafe, my feeling is that, by the time you've gone down that road, you might as well be using C++.

I'll admit that, after many years of not writing native code, I was a bit nervous of diving back in. Segfaults are never fun. But, we soon realized that the native code tooling has improved a _ton_ in the last decade. We run all of our tests precommit using the excellent Sanitizer tools from Google (ThreadSanitizer, AddressSanitizer, LeakSanitizer) and those make it nearly trivial to diagnose a leak or crash. We also have pretty strict guidelines around use of pointers, based on the Google C++ guidelines. Many will complain that this is a neutered form of C++, and they're right. But it's also a relatively safe form of C++.

I could probably write a lengthy blog post on our experiences of C++ vs Java, but hopefully the above gives you a taste. Overall I've been happy with the decision. Slightly more time spent on crashes. Less time spent on chasing hard-to-reproduce performance or memory consumption issues. And the thread checking tools are actually far superior, so I'd say less time spent chasing races.

We're working on running some more thorough YCSB benchmarks, but here are some of the percentiles on the uniform "workload A" running on a 9 node cluster for 1 hour:

Throughput: 28280 ops/sec Read: 2821us avg, 467us min, 3519us 95p, 6843 99p Update: 1688us avg, 714us min, 1983us 95p, 8855us 99p

Workload D, which has some inserts (and reads recently written data):

Throughput: 36286 ops/sec Read: 1765us avg, 491us min, 2537us 95p, 4259us 99p Insert: 1614us avg, 838us min, 1595us 95p, 11575us 99p

Hope that helps. I'll try to push our latest YCSB bindings to github later this afternoon/evening if you'd like to reproduce on your own.

We haven't scoped out the security features. Cell level security can be difficult to implement efficiently, but if we see enough demand for it, I could imagine it happening.

My guess is that the first pass will be table and column level authorization, plus of course strong authentication. Row/cell/predicate-based security could be added in a later release, but it's a feature that's less commonly required.

As for encryption at rest, I imagine that will also be fairly high priority as we move towards GA or the first few releases after GA. But again, we haven't done the scoping exercise yet, so I'm cautious to throw out dates :)

If you're interested in helping to contribute either feature, let us know! kudu-dev@googlegroups.com

Should add that those looking for a technical deep dive might enjoy our draft paper: http://getkudu.io/kudu.pdf and/or browsing our source: http://github.com/cloudera/kudu

Todd from the Kudu team here. If anyone has any questions, feel free to ask them here, will try to check back throughout the day.

True it's a pain to embed. We embedded it in my current project, and it only took a few hours to add a CMakeLists.txt and make it reasonably embeddable.

I guess if your project doesn't want to include C++ (and you don't want to write a simple C wrapper), you've got a fair point. But it sounds like their implementation is significantly faster than the one in the linked post.

Existing library which does the same thing, and faster, with arbitrary polynomials: http://code.google.com/p/crcutil/

If this were more like Spotify for sheet music, I'd pay for it -- instant access to music for any published song is valuable. Otherwise, I'm not sure how it's much better than dropbox and PDFs.