Jepsen: Dgraph 1.0.2

by aphyr • 8 years ago

183 124 8 years ago

Sir_Cmpwn18• 8 years ago

Important: contrary to the marketing material, dgraph is not open source - it uses the Commons Clause we've been discussing over the past few days. Discussion here:

https://github.com/dgraph-io/dgraph/issues/2545

jadbox• 8 years ago

Doesn't open source just mean you have access to read and modify the software? It's the stronger term Libre that implies that you grant all distribution/selling rights to the public.

Sir_Cmpwn18• 8 years ago

No, open source is defined by the Open Source Definition, published by the OSI:

https://opensource.org/osd

brazzledazzle2• 8 years ago

I don’t have any problems with the commons clause in principle but I think they (the commons coalition or whatever they are) should just create a new license. Like commons-apache2.0 or something. Saying you’re Apache licensed is kind of dishonest because it’s not really.

tunesmith• 8 years ago

When did the definition of "open source" change? I always took "open source" to be an umbrella term that contained both Apache-type licenses and GPL/FSF-type licenses. I get that Commons Clause is more restrictive in ways than GPL, but GPL is definitely open source. So what makes Commons Clause not open source, if GPL is?

ealexhudson• 8 years ago

You can sell GPL'd software (on its own or as a service), you can't sell Commons Cause software as a service.

brazzledazzle2• 8 years ago

It’s not really Apache though. That’s the problem I see with it. If you tell me you’re Apache licensed I shouldn’t be surprised when I open your license file.

Sir_Cmpwn18• 8 years ago

The definition of "open source" never changed. Apache and the GPL are both open source licenses. Appending the Commons Clause removes the zeroth freedom (FSF lingo) or the first clause (OSI lingo).

https://opensource.org/osd

https://www.gnu.org/philosophy/free-sw.en.html

marknadal2• 8 years ago

The definition has not changed, "Open Source" arose as a reaction against GPL/FSF/etc. so that code could truly be open for any and all.

striking2• 8 years ago

From https://commonsclause.com:

It [sic] this “Open Source”?
No. [also sic]
“Open source”, has a specific definition that was written years ago and is by the Open Source Initiative, which approves Open Source licenses. Applying the Commons Clause to an open source project will mean the source code is available, and meets many of the elements of the Open Source Definition, such as free access to source code, freedom to modify, and freedom to re-distribute, but not all of them. So to avoid confusion, it is best not to call Commons Clause software “open source.”

Additionally, "Apache 2.0 + Commons Clause" is not an Apache license. I have seen a number of people be confused by this. I hope Apache implements some guidance with regards to trademarks on this matter, mostly so I don't have to keep saying it.

mrjn7• 8 years ago

We changed the references to say liberally licensed. I'll try and write a bigger blog post about the reasoning behind Commons Clause and Dgraph and other companies' adoption of it.

Sir_Cmpwn18• 8 years ago

I think it's disengenuous to say liberally licensed. Liberally licensed might apply to MIT or even Apache alone, but not when you include the Commons Clause. But it's a subjective measure, so you're not making factually incorrect statements even if you are being dishonest.

Your updated statement also says "Dgraph is available under an Apache v2.0 based liberal license", which I think is unfairly using the Apache 2.0 name without clarifying that your license in fact is not open source. The language is still dishonest.

marknadal2• 8 years ago

Commons Clause, GPL, FSF, etc. are not liberal though.

MIT, BSD, ZLIB, Apache2, etc. are liberal.

Meai• 8 years ago

Well I guess I learned something new today, open source is apparently a term that was defined by somebody and has a counter intuitive meaning. To me it has always meant that the code is available for viewing - nothing more. I assumed this without knowing anything about the OSI.

Reference: https://opensource.org/osd

slavak2• 8 years ago

Yup. Open-source is a whole philosophy. What you're talking about is called source-available software.

sanderjd8• 8 years ago

What does this have to do with the Jepsen test discussed here? I'm pretty tired of so many comments sections turning into philosophical debates of open source licensing terms the past few days.

Maybe this will naturally bubble down so that the far more interesting technical discussions can bubble to the top, but I fear this sub-thread will eventually drown out the on-topic discussion.

Sir_Cmpwn18• 8 years ago

I normally wouldn't comment, but in this case we're talking about a promotional article that leads readers down a conversion funnel. Because some steps at that conversion funnel are dishonest, I want to make sure that people are aware of what they're actually looking at.

BMarkmann2• 8 years ago

I honestly don't know that there are any posts I get more excited about on HN than Jepsen analyses. The number of landmines distributed systems create and the ability to suss them out in such detail / depth is really incredible. Kudos.

striking2• 8 years ago

Totally agreed. My one and only wish is that they might offer an RSS feed.

aphyr6• 8 years ago

Oh, yeah, that's something I could put together. In the meantime, there's a mailing list and twitter you can subscribe to! https://jepsen.io

amelius• 8 years ago

Agree. But ... I sure hope that the new way of developing distributed systems doesn't follow this pattern:

1. write some code

2. submit to Jepsen

3. if errors, goto 1

The problem with this approach is obvious.

BMarkmann2• 8 years ago

You forgot step zero, which is "find terrible example code on SO and copy it"... then 3 goes back to step 0.

yashap• 8 years ago

With most of the DBs he tests, the authors have very good understandings of distributed systems, and make very carefully considered design choices. They still make mistakes though, because they’re human. Very thorough testing is simply a great way to make any software more robust, distributed systems or otherwise.

namibj• 8 years ago

I'd hope they find themselves one or more with a master in math on logic proving in ways that a computer can help, and actually prove that their code, at least if compiled to LLVM bytecode, is adhering to the high-level architectural proof. Considering that they'd only need to prove consitency, nothing more, and especially not every little piece of code they are writing, but just the core that handles transactional isolation, not even that the code actually resolves in all cases (being stranded with a stuck cluster that is made to handle a hard reboot without problems is better than a cluster that silently violates constraints), it should not be an infeasible goal.

The main issue is that distributed database consistency without trivial, performance-killing locking schemes is too complex to prove when writing or using any trivial, local methods based on e.g. SMT solvers or so.

If something like cockroachdb would be proved-consistent on that level, it would be used for applications currently employing pessimistic locking due to a lack of trust in their database, or scaling vertically without really needing to (there are cases which make horizontal scaling cost-prohibitive due to the dependency chains in the algorithms that can solve them, but they can be replaced most of the time).

danr4• 8 years ago

Me too. and I don't know a thing about databases and distributed systems. I guess I still have a soft spot for the snarky ones from a few years back.

makmanalp• 8 years ago

Slightly tangential but dang, this is the best graph and accompanying explanations of consistency models I've ever seen: https://jepsen.io/consistency

jdcarter• 8 years ago

Does anybody else miss the style of Aphyr's older posts, with animated Barbie dolls and such mocking the product under scrutiny? Examples:

https://aphyr.com/posts/317-jepsen-elasticsearch

https://aphyr.com/posts/284-jepsen-mongodb

I understand this analysis was paid for by Dgraph so GIFs and memes aren't appropriate... but I do miss them.

mberning• 8 years ago

aphyr6• 8 years ago

Oh, I certainly took no shortage of flack for those early analyses, but I'm glad you appreciated them! Some of my clients have even expressed disappointment that my jepsen.io analyses for them weren't as snarky as the aphyr.com ones! It's a tough situation though--if I were to write snarky analyses, even for unpaid clients, it would raise the possibility that clients could pay for favorable treatment. So... I go through an extensive process, often involving weeks of review with my client and peers, to produce as accurate, professional, and neutral a report as I can.

There's another aspect to this: now that I've established a reputation and people take my work seriously, I don't have to be snarky or aggressive to get attention drawn to these issues. And the DB landscape has changed a lot in the last five years! Engineers want to provide formalized safety guarantees. Vendors are taking these failure modes seriously instead of dismissing them as irrelevant! Since I'm getting paid, I can invest more time into making these analyses real collaborations with the vendors. So I see my role now as more about helping people reach their safety goals, and a little less about shaming folks for making systems that didn't live up to their claims.

My other motivations--letting users know how to work with the databases they've chosen, and giving case studies to help other engineers test and improve their own systems, well, those are unchanged. :)

kjeetgill• 8 years ago

Thank you for all of your work. I dove into those posts at the start of my career and they've shaped how I think about my work tremendously. (Along with Brendan Gregg's work!)

I still reread this and pass it around at least once our twice a year: https://aphyr.com/posts/313-strong-consistency-models

It's dense and takes some time to digest but I consider it required reading for anyone working on or with distributed systems.

clebio• 8 years ago

now that I've established a reputation and people take my work seriously, I don't have to be snarky or aggressive to get attention drawn to these issues.

like the parent and others here, I've lamented this. I appreciate your considered reply. This callout, in particular, is something I hadn't accounted for, but it makes perfect sense (the general idea that "we're getting serious and our clients are serious now too, so posts have to be serious" is just sort of obvious, and still likely true). Cheers to fighting the good fight!

yoklov• 8 years ago

These are great, thanks, I hadn't seen them before.

mrjn7• 8 years ago

Author of https://dgraph.io here. I'd say this is one of the best collaborations we've done. Due to Jepsen tests, we were able to proactively identify and fix issues in a complex real time distributed graph DB, that is Dgraph. Dgraph's graph sharding, replication and transaction implementation is unique and it took more work to make it all work correctly, while still achieving good performance, and without utilizing any third party system.

I'd like to thank Kyle for his work on Dgraph. We'll continue to expand Jepsen tests to try and identify more issues and fix the remaining ones.

I'm around if you have questions! Cheers, Manish

jimktrains22• 8 years ago

Dgraph has made significant progress, but 4 of the 23 issues we identified remain unresolved, including the corruption of data in healthy clusters. This work was funded by Dgraph,

Between this and your comment above, it restores a little faith in humanity that there are people out there who honestly want to make their work as good as possible and not paper over cracks.

manigandham• 8 years ago

If you're talking about funding, many of the recent analyses were funded by the company whose product is being tested. Most companies do this kind of testing internally but it's good to pay for the recognized expertise and thorough process of jepsen.

mrjn7• 8 years ago

Thank you! This is the best acknowledgment of our work I've seen so far. A lot of blood, sweat, hairs, frustrations, and tears has gone into building Dgraph -- and this comment by you makes me feel proud of what we're building, and the community of people using it.

beagledude2• 8 years ago

Were there any badger related issues? There doesn't seem to be reading the article.

mrjn7• 8 years ago

None were found based on Jepsen testing done.

We had done a separate testing for Badger before Jepsen, where we had identified and fixed a few issues related to crashes. You can read that blog post here:

https://blog.dgraph.io/post/alice/

aphyr6• 8 years ago

Yeah, we did talk about Badger a bit, but I think Dgraph's internal testing was likely going to stress it more than I could over the network. We have occasionally found storage-engine level bugs with Jepsen (Tendermint, for instance), but usually the low request rates mask race conditions in storage libraries. :)

billrobertson423• 8 years ago

I assume more testing will occur after the four bugs are addressed. Will that be published on the Jepsen blog?

steven19824• 8 years ago

No man, the bugs are open from long time. The team is no more, don't know why. Only the founder is doing commits from months. I had so much hopes on dgraph, it was way better than other graph dbs but immature

mrjn7• 8 years ago

Yeah, we're working on fixing the existing bugs and working with Kit Patella (https://github.com/mkcp) to further extend Jepsen's Dgraph tests.

The Snapshot Isolation bank test bug, in particular, is proving a lot harder to reproduce now -- it takes many hours (even up to 8h) to show a violation; which makes debugging harder obviously. The rest 2-3 are sort of related and easier to fix.

I doubt Jepsen blog post would be updated, but once all the remaining bugs are fixed, we'll write a blog post about it on https://blog.dgraph.io.

truth_seeker• 8 years ago

Are these fixes also applicable for OpenCypher engine in Dgraph ?

mrjn7• 8 years ago

Dgraph currently does not support OpenCypher.

staticassertion4• 8 years ago

I've been using dgraph for a side project. I chose it because I have a really high performance requirement and I find the query language feels 'right'.

Glad to see the progress made, and bugs found and fixed, for these Jepsen tests.

As a side note, interesting to see a largely Go-based, real-world project plagued with race conditions. Quite a few bugs read "but it was done in a goroutine" or "but mutable capturing in a goroutine", and even a segfault.

baq• 8 years ago

Go probably is the easiest language to get something working with any kind of performance, but it doesn't help much to get something working correctly. After dabbling with Rust I'm amazed that multithreaded programs work at all...

gazarsgo• 8 years ago

For each Jepsen release my favorite thing is the code for the tests themselves -- https://github.com/jepsen-io/jepsen/tree/master/dgraph/src/j... -- which seems surprisingly unreferenced in the post itself.

aphyr6• 8 years ago

Links to the code are throughout the "test design" sections of each report. :)

rdxm• 8 years ago

gotta say, if you do distributed systems, subject yourself to Jepsen analysis and publish it, you win a truck-load of points in my book.

steven19824• 8 years ago

Dgraph has been bragging about being used by big companies. Jepsen test clearly shows that there were critical bugs and no sane company can use it in production as primary database, unless they don't care about their data.

vemv• 8 years ago

Same could be said about almost every system tested by Jepsen, so your observation seems unfair.

steven19824• 8 years ago

Dgraph released 1.0.6 and replication stopped working, I gave up on dgraph. It's just not reliable and immature. It's expected since there is only single developer making commits from months.

antpls• 8 years ago

Was a set of testing tools used to build the analysis? It would be nice to be able to reproduce the results.

drej• 8 years ago

FYI: There's a very recent podcast with Manish, the author of DGraph, https://www.dataengineeringpodcast.com/dgraph-with-manish-ja...

Jepsen: Dgraph 1.0.2

Related Stories