jodok

thanks for the comment, we'll update the indexes used in postgres. we tried to be as objective as possible and of course show our sweet spot. we don't want to treat postgres unfair, if you have suggestions on how to improve indexing in postgres we'd be happy to take that into account. (full disclosure: i'm co-founder of crate.io).

@ralala - would love to hear your feedback once tried crate (jodok@crate.io). we've been working directly with kyle on the issues discovered. the current state of our work is documented on our website: https://crate.io/docs/scale/resilience/

Thankfully, for most use-cases, if you our follow best practices, you are extremely unlikely to experience resiliency issues with CrateDB.

why not run the database directly in the container as well? we're doing that with https://crate.io - just expose your local instance store as volume, crate will take care of the rest

https://crate.io Co-Founder here. We're participating in this game by working hard to build a fully distributed, shared-nothing SQL database.

In our vision, a app in a container is able to access the persistence layer like in SQLite - just import it. another cluster of containers - preferably having one instance node-local - takes care of the database needs.

The database is distributed and makes sure enough replicas exist on different nodes. it's easy to scale up and down, local ressources are being utilized whenever possible (no NAS/SAN like storage).

cool stuff

full disclosure. i'm one of the co-founders of crate :)

In December 2010 we found about Elasticsearch and were truly amazed by it's simplicity, speed and elegance. We built our service and consultancy business around it.

In 2011 we've built some of the largest ES applications at that time (6TB, 120node cluster, http://2012.berlinbuzzwords.de/sessions/you-know-search-quer...) and started to develop a set of plugins, such as the in-out-plugin to allow distributed dump/restore.

With this background - and the mission to build a datastore that is as easy to use and administer as Elasticsearch we founded in 2013 Crate and raised some Seed money. Since that we're working hard to make this vision become true. We're often confronted with the results of the so-called Jepsen-Test (https://aphyr.com/posts/317-call-me-maybe-elasticsearch) that Aphyr published in 2014. Don't forget: Lucene, Netty, Elasticsearch, Crate - all are Open Source products (APL) and rely on all kinds of contributions - such as this analysis! No matter be it bad news or good news. We can only improve based on hard testing and feedback. However, this caused a lot of rumblings in the Elasticsearch ecosystem and the reaction of Elasticsearch was exemplary:

1) explain the reasoning and make an official statement: https://www.elastic.co/blog/resiliency-elasticsearch/

2) list all the issues and hunt them down. one by one: http://www.elastic.co/guide/en/elasticsearch/resiliency/curr... (and add new ones as they occur).

3) stay in contact with the community that reported the issues: https://twitter.com/aphyr/status/525712547911974913

All that being said. We see many people using Crate as primary store (and of course backing up their data) but we also see people that don't put that much trust in a younger database and keep all their primary data in another location and sync/index to Crate.

ALWAYS make backups (COPY TO / COPY FROM), make sure you have replicas, and most important configure minimum_master_nodes correctly to avoid split brain.

At Crate we stand on the shoulders of these great Open Source project, try to be as good citizens as possible and focus mainly on our Query engine (Analyzer, Planner, Execution Engine).

It's named Blender Pro and has been designed by our swiss friends from binnenland. We're super happy that they provided us with a generous license to use it within Crate. More about the font: http://www.binnenland.ch/notes/view/about-the-blender-typefa...

Crate-only queries - like exact counts, distinct counts, (soon JOINs) - are not exposed via this API. The Crate query engine is accessing Lucene directly and not running on top of the ES query engine.

We run aggregations fully distributed and when iterating over the values we heavily rely on the field-caches. They hold the values of the latest used fields in memory and therefor allow in-memory performance on them. for example they don't grow linearly with the amount of rows stored, but depend on the cardinality of the fields. Running aggregations over non-indexed data is not supported.

not yet. but we have some "tweaks" for exactly your use-case on our backlog. using the client libraries should make it mucn easier (e.g. https://crate.io/docs/projects/crate-python/stable/sqlalchem...). so right now you would need to do a refresh. on a side note: it's not an index delay. it's the readers that "sit" on the lucene index. they are being repurposed for performance reasons (and meanwhile other writes are appending). like the client libraries you can force reopening them (https://crate.io/docs/en/0.47.8/sql/reference/refresh.html) - of course at the cost of performance.

we recommend to expose a host directory to crate ('docker run -d -p 4200:4200 -p 4300:4300 -v <data-dir>:/data crate') and configure replicas. if one of the crate containers disappears, replicas will be promoted as primary shard and new replicas created on the fly. it's also possible to expose multiple directories (e.g. on multiple disks for more performance), you can configure crate to use them in parallel.

yes, you can enable the elasticsearch API ("es.api.enabled: true" in the crate.yml config file). however, it's not officially supported and we recommend to use it read-only as we store additional information about tables,...

No, it can hold as much data as the sum of all the nodes in the cluster have (also depending on the amount of replicas you configure). Some background: All data is being sharded by default. These shards are distributed evenly between the nodes. Nodes can hold multiple shards. Furthermore shards can have replicas, being hold by other nodes. Memory is being utilized to hold indizes in RAM and to cache values that are being needed frequently for e.g. aggregations ("field caches").

we're good friends with the original owner "dstufft" and took over the domain after he took over an official pypi position (more than a year ago). read his own blog post: https://caremad.io/2014/03/crate-io-new-ownership/

arthur, that's how we started two years ago. meanwhile the queries are being handled by our own engine. https://github.com/crate/crate

the elastic team is working hard in fixing the issues discovered by aphyr (jepsen test). you can find the current status here: http://www.elastic.co/guide/en/elasticsearch/resiliency/curr... - we're doing our best to support them. (btw. here's what aphyr thinks about that document: https://twitter.com/aphyr/status/525712547911974913)

we just moved our very own https://crate.io site from wordpress to cactus (python-based, https://github.com/koenbok/cactus). just opened our github repo: https://github.com/crate/crate-web it also includes a script to pull in existing wordpress blog posts.

yes, on the same host it's fine - we're mentioning that in our dockerfile readme (bottom): https://github.com/crate/docker-crate

bummer! good catch. video was taken before this bugfix: https://github.com/crate/crate/blob/master/CHANGES.txt#L160 we'll re-captchure it: https://twitter.com/burndee/status/457256576345448449

sorry for my ignorance, but Crate and "SQL on top of Hadoop" fulfil quite different needs. How would you handle tens of thousands concurrent queries on a Hadoop platform? Read and write at the same time? On the other hand side - crate will never be able to batch-oriented, complex map/reduce workload.

we took crate.io over in december. some people still haven't updated their links. that's why we keep some subdomains and redirects up and running (https://crate.io/thanks-dstufft)

we're unrelated. but we keep some redirects and subdomains from the "old" crate.io. see https://crate.io/thanks-dstufft.

we've worked with upt to 25bn records (small size) in crate, that was about 3TB (without replicas). if you have larger records or blobs, you're soon in the 10th of TB.

thanks for your summary and valuable feedback. a few notes from the crate team: 1. JOINs are on our roadmap. we might never reach full JOIN support anytime soon (INNER, LEFT OUTER, RIGHT OUTER, FULL OUTER and CROSS), but simple use-cases (1:n), equal joins shouldn't be too hard. 2. after re-reading our documentation on sharding i need to admit that we need to improve it. we mention it briefly here https://crate.io/docs/current/sql/ddl.html#sharding, more will be added asap 3. if we have growing datasets we typically work with parted tables. we just pushed that change just a few days ago (https://github.com/crate/crate/blob/master/docs/sql/partitio...), not yet in the RPM build. 4. no, you can specify routing https://crate.io/docs/current/sql/ddl.html#routing 5. agreed, that the doc can be improved. and we're at the very beginning https://crate.io/docs/current/sql/ddl.html#replication - rack awareness,... there are quite some replica settings on our roadmap. i'd be happy to hear your most important additions to replica policies.

i promise, we're working hard and try to be there faster than in a few years :)

we here at crate were living in the hadoop ecosystem for some time. but when we came across the beautiful architecture of netty.io (async, event-driven) and the way elasticsearch orchestrated it - since that time we know there will be room for newcomers :)

spatial data types are high on our backlog. you'll be able to store geo points, query them by location, bounding box, polygon, sort them by geo distance, and also aggregate them based on geo-distance (e.g. 0-10km, 10-50km,..). we haven't thought about additional topics on top of lucene - but that's our list for a start. feedback, github issues welcome!

cassandra is a super choice for write oriented workloads with really high performance. it's also super stable. we want to be in the spot where you also need advanced realtime search, high numbers of concurrent read/writes. and yes, we focus to make administration as easy as possible

hi, i'm jodok from crate. yes, right now we only use the sql parser (and extended it)