Show HN: Nebula – a distributed graph database written in C++

by jamie-vesoft • 7 years ago

178 102 7 years ago

Interesting stuff, thanks for sharing!

How did you decide to use an SQL-like query language rather than a declarative query language like Neo4j uses (Cypher query language)? What do you see as the pros and cons of that decision?

Do you have any plans currently to design an ETL tool to make extracting data from a RDBMS and loading into NebulaGraph easier?

I didn't see this in any of the documentation (though I did admittedly skim), but is there any sort of visual front-end built in? If not, do you have any plans to make one?

vvern• 7 years ago

SQL is a declarative language. Cypher is somewhat SQL-like, also declarative language which exists to make the expression of graph queries with predicates over nodes and edges easier to express.

FillardMillmore2• 7 years ago

Thank you for the correction. You are correct, SQL is a declarative language. It would've been more accurate for me to refer to Cypher as 'a more expressive declarative query language' (specifically for the graph database paradigm) than a typical SQL-like query language.

jamie-vesoft24• 7 years ago

Thanks for your interest! :)

Good question re data loading! Currently there are two tools for this purpose: 1) Spark Writer which enables you to load data from DWH to Nebula Graph via Spark: https://github.com/vesoft-inc/nebula/blob/master/docs/manual... 2) CSV importer if you don't have to import very large data from RDBMS to Nebula Graph: https://github.com/vesoft-inc/nebula-importer

The visual will be released by the end of this week. :)

ragerino2• 7 years ago

Apache Drill is nice for querying structured files like CSV.

https://drill.apache.org/

cetra3• 7 years ago

It's not open source, as it is licensed under Commons Clause according to the README, which according to the FAQ is not open source.

I'd be interested in knowing whether the commons clause license has been challenged as the wording is rather simple

adrianN2• 7 years ago

I don't understand why projects don't simply use AGPL and offer a commercial license as an alternative.

fiberoptick• 7 years ago

There's a subtle difference between AGPL and the Commons Clause licenses.

AGPL requires network-accessible code to be disclosed & licensed under an AGPL-compatible license.

The Commons Clause license outright prohibits SaaS-style offerings of the licensed code.

A lot of startups licensing their code under AGPL might still have AWS et al. eat their lunch, becuase all Amazon needs to do to remain compliant is to publish any modifications made to the AGPL-ed code.

stockkid• 7 years ago

Another approach I have seen is BSL (Business Source License), which is kind of like Commons Clause in that it prohibits commercial offerings of the software, but after a rolling time limit, converts to an open source license. I might be wrong, so please correct me.

https://www.cockroachlabs.com/blog/oss-relicensing-cockroach...

codycraven• 7 years ago

I get that Commons Clause isn't "open source" but I really love the concept. If a company wishes to productize a creator's work it seems reasonable to pay the creator to alternatively license it (if the company doesn't want to put their secrets out to the public).

Meanwhile the creator gets to share their work freely with anyone who wishes to use it as a component of their own product/software in the spirit of open source.

jamie-vesoft24• 7 years ago

That's exactly what Common Clause is pursuing IMO.

jamie-vesoft24• 7 years ago

As far as I know, Common Clause can be attached to any open-source projects. The main purpose is to prevent cloud providers monetizing from the project without contributing back. So Nebula Graph's main license is Apache 2.0, meaning that to most users it is open source, no different than any other open source projects. :)

zozbot2345• 7 years ago

to most users it is open source

Discriminating by field of endeavor is contrary to the definition of open source software, and has been since before the term even existed. It's not open source, it's effectively Shared Source and developers who care about open source should stay away from this.

bluejekyll• 7 years ago

The real question that has to be answered, and this is the hard one, when does the product begin to be monetized?

Let’s say it’s a full DB option as part of AWS RDS (or whatever that graph DB equivalent is). That probably is clearly monetizing the product. But what if they completely abstract the API and not expose the original one, it’s just the backing engine for a graph DB product?

Now moving away from a direct product, what if it’s just the backing DB AWS uses for managing all of their infrastructure? It’s not being directly monetized at that point but it might be the most critical component for the AWS operations, which means that it is helping them monetize other products. Do they owe in this case? (I’m speaking about the license here, not whether or not they should or should not based on goodness or feature improvements they want to pay to see).

As the DB moves further away from profit centers in an organization, at what point is it no longer being monetized?

Personally, I’d like to see a model where the OSS developers can and are paid in all of these cases for their work, but I’m not always sure there is anything better than a contract to support and build new features (classic OSS support model).

thunderbong• 7 years ago

Honestly, I don't see what's wrong with expecting payment for your work if someone else decides to sell it. Why should 'open source' get conflated with free (as in gratis)?

For me, open source has been an incredible way to learn software - it's syntax, it's architecture, it's control flow, it's gotchas.

From my understanding of the license [1], you can see the code, learn from it, do whatever you want with it, modify it if you so please, improve on it, whatever. The only thing you cannot do is sell it. Because you've taken someone else's idea in the first place.

I see this happening all the freakin' time and it pisses me off no end. If I suggest a software to someone, the first thing they as is 'Is it open source?' What they really mean is 'Is it free?' Why? If someone is expecting to get paid for creating software for others, why is the feeling not reciprocated towards the person who's created the software in the first place?

From what I've seen, most managers and software engineers, expect to get paid for their work but all the software which helps them make that money, they expect for free.

I find that attitude extremely hypocritical, honestly.

[1] https://commonsclause.com/

zozbot2345• 7 years ago

Why should open source get conflated with things that are NOT open source? Putting restrictions around "commercial" use (which is notoriously hard to define) is not open source. Discriminating against fields of endeavor is not open source.

If you want to get paid for developing genuine open source software, there are things you can do to that effect. Get paid for support (even maintaining the code is support). Offer to highlight companies that support your software (even if the highlighting is quite trivial, this is enough to unlock 'marketing' expenses and make it easier for business-oriented entities to support you). Start a Patreon page. There are lots of things that can be done without adding any licensing restrictions.

microcolonel• 7 years ago

Then don't call it open source, because that word in almost all cases involves free use regardless of commercial interest.

It's all well and good, and nothing immoral is done by offering code under this license, but that doesn't make it open source.

e12e• 7 years ago

Oh wow, this really is just source available, isn't it?

For purposes of the foregoing, "Sell" means practicing any or all of the rights granted to you under the License to provide to third parties, for a fee or other considerationon (including without limitation fees for hosting or consulting/support services related to the Software), a product or service whose value derives, entirely or substantially, from the functionality of the Software.

So, you cannot pay a contractor to set this up, because they can't deliver to you if they charge for setup or hosting?

jamie-vesoft24• 7 years ago

Really appreciate the feedback and discussion! We will seriously consider the license issue.

Please DO let us know if you have any better license options than Common Clause that can help provide an open-source project for the community while stop cloud vendors from monetizing without contributing back?

Thanks again!

pbowyer• 7 years ago

I follow a few licensing blogs, and I know people are talking about and working on alternatives to Common Clause. As far as I can tell, all will disappoint those who believe the OSI's definition of "Open Source" is the one true open source.

My bookmarks: https://katedowninglaw.com/blog/ https://writing.kemitchell.com/ (blog & the blogroll for finding others). See also: https://www.google.com/search?q=site%3Awriting.kemitchell.co... http://www.blueoakcouncil.org/

bane3• 7 years ago

This is very interesting. Anybody use it? Alternatives?

jhoechtl3• 7 years ago

Alternative:Dgraph

https://dgraph.io/

In architecture and goals it actually closely resembles Dgraph, would love to see an (opinionated) comparison by Manish, the CEO of Dgraph

w3clan3• 7 years ago

I had the very same feeling, dgraph is older and has a larger community plus additional features like:

- geospatial features

- good speed as it is based on badgerdb key value database and ristello cache library.

- http library and other features

One of the advantage I saw in nebula graph is security role based access which is not available in dgraph until today.

I am very curious about benchmark between nebula graph and dgraph.

Also what is storage system used in nebula graph.

mrjn• 7 years ago

(Manish here) Don't know much about Nebula. Feels quite inspired by Dgraph.

emmanueloga_2• 7 years ago

Check the list of implementations of SPARQL [1].

One of the most interesting picks: RDF4j (java based). It can connect to a lot of different SPARQL servers, but the rdf4j Native Store should be good enough for data sets in the order of the "100 million triples", according to the docs.

I don't know much about it, but not long ago they announced integrated support for "federated queries", which means that if you data set can't fit in a single node, they have a solution to query different servers in the same query [2].

I'm slowly learning through the forest of related technologies, one of the most useful is SHACL [3], which is a language to validate and extract pieces of the graph that match a pattern (very loosely, think a "schema" for graphs).

1: https://en.wikipedia.org/wiki/List_of_SPARQL_implementations

2: https://rdf4j.org/news/2019/10/15/fedx-joins-rdf4j/

3: https://rdf4j.org/documentation/programming/shacl/

rapnie3• 7 years ago

Before using RDF for graphs one should inform themselves on the differences between labeled property graphs and triple stores, and choose the model that best fit their use case.

Take your time and beware of objectivity of article. Vendors try to lure you in. Following has some good info (but a Neo4j bias): https://dzone.com/articles/rdf-triple-stores-vs-labeled-prop...

sfgweilr4f• 7 years ago

I'd say dgraph (https://dgraph.io/) is closest competitor but Neo4j (https://neo4j.com) as well which has longer heritage.

I'd also include redis because of the graph module (https://oss.redislabs.com/redisgraph/).

I've likely missed a bunch of others. Add them as I'm interested in graph db and have only scratched the surface myself.

rapnie3• 7 years ago

There is also decentralized Gun DB: https://gun.eco/

OrientDB is an alternative to Neo: https://orientdb.org/

There are a bunch of DB's compatible with Tinkerpop and e.g. query-able with Gremlin: http://tinkerpop.apache.org/

lmeyerov4• 7 years ago

We help bring gpu visual analytics & investigation automation to users of all sorts of graph DBs (think tableau & servicenow for graph), so based on our enterprise/big tech/gov/startup interactions:

1. Shortlist (and in no order): Neo4j, AWS Neptune, Datastax Graph, TigerGraph, Azure CosmosDB, and JanusGraph (Titan fork) are the ones we see the most in practice, and not in production but rumor-mill, Dgraph, RedisGraph, & ArangoDB. The three-and-four-letter types seem to roll their own, for better or worse. There are also some super cool ones that don't get visibility outside of the HPC+DoD world, like Stinger & Gunrock. Interestingly, the reality is a ton of our graph users aren't even on graph DBs (think Splunk/ELK/SQL), and for data scientists, just do ephemeral Pandas/Spark. As someone from the early days of the end-to-end GPU computing movement, we're incorporating cuGraph (part of nvidia rapids.ai) into our middle tier, so you get to transparently benefit from it while looking at data in any of the above.

2. I now slice graph DB's more in terms of OLTP (neo4j, janus, neptune, maybe tiger) vs OLAP (spark graphx, cugraph) vs batch (janus, tiger) vs friendly BI/data science (neo4j) vs friendly app dev / multi-modal add-on (CosmosDB, Neo4j, Arango, Redis). Curious to see how this goes -- given the number of contributors, I'm guessing it's doing well in at least one of these. +1 to hearing reports from others!

bane3• 7 years ago

Thanks, I really appreciate the comprehensive write up of what your team is seeing. Any chance of a longer blog post that expands on this, especially pro-cons and performance?

derefr• 7 years ago

For someone who just wants to run some (intensive) OLAP graph queries on the “graph formulation” of a relational or hierarchical dataset every once in a while (maybe batch, maybe user-initiated, but either way <1QPS), but doesn’t yet have a graph DB and doesn’t really want to maintain their data in a canonical graph formulation, which type of graph DB would you recommend as the simplest-to-maintain, simplest-to-scale “adjunct” to their existing infra?

I.e. what’s the graph DB that best fits the use-case equivalent to “having your data in an RDBMS and then running an indexer agent to feed ElasticSearch for searching”?

jamie-vesoft24• 7 years ago

Currently the project has been deployed in multiple leading internet companies in China, including Tencent, MeiTuan (Chinese Yelp), Red (Chinese Pinterest), Vivo, and so on.

bane3• 7 years ago

That's pretty impressive. I'd love to see some details blog posts about setting it up, or using it in production (things to watch out for, good practices for provisioning hardware, etc.).

gibsonf12• 7 years ago

We're using Blazegraph (The db that Amazon Neptune uses) with great success. We only use the Sparql api with quads so we get nested graphs.

rs23296008n1• 7 years ago

This looks good. I like the query language.

Whats the status of the C++ client?

https://github.com/vesoft-inc/nebula/blob/master/src/client/...

I see it has connect/execute/disconnect. This seems like the minimum required. What can't I do with that basis?

jamie-vesoft24• 7 years ago

Glad you loved the query language!

The C++ client is PR ready now:

https://github.com/vesoft-inc/nebula/pull/1013

maxpert• 7 years ago

Any production usages that can tell us about how much it can handle? How does it compare to dgraph?

jamie-vesoft24• 7 years ago

Thanks for showing interests in Nebula Graph!

In some production deployments, the graph size reaches hundreds of billions of edges and data size reaches dozens of TB.

lmeyerov4• 7 years ago

Wow, not sure how I missed this! Bravo!

If anyone involved would appreciate access to Graphistry to experiment with a gpu client/cloud visual analytics side (e.g., jupyter & react), let me know (leo@....com). Would love to see them together!

jamie-vesoft24• 7 years ago

Exciting! I know Graphistry from Twitter.

slowmotarget• 7 years ago

It looks really neat! What's your plan on making this project viable in the long run? Do you envisage to monetize hosting, or maybe create a community vs. paid edition?

jamie-vesoft24• 7 years ago

Glad you liked the project! Hosting service would be the main monetization method. In addition, we will be providing consulting, training and all sorts of enterprise services.

foota• 7 years ago

The architecture diagram looks interesting, would love to read more about it if anyone finds something.

rainyi2007• 7 years ago

I found some articles about the architecture and design of the database in their repo:

Overall structure: https://github.com/vesoft-inc/nebula/wiki/Nebula-Graph-Archi...

Storage engine design: https://github.com/vesoft-inc/nebula/blob/master/docs/manual...

Query engine design: https://github.com/vesoft-inc/nebula/wiki/Query-Engine-Overv...

Hope that helps. :)

ioli• 7 years ago

Thanks

winrid• 7 years ago

Love the query language! Very easy to dive into. Any companies using this in production?

jamie-vesoft24• 7 years ago

Glad you loved the query language. Simplicity and versatility are our design goals for the language.

Currently the project has been deployed in multiple leading internet companies in China, including Tencent, MeiTuan (Chinese Yelp), Red (Chinese Pinterest), Vivo, and so on.

pojntfx• 7 years ago

It's a proprietary license, false advertising ...

SQueeeeeL2• 7 years ago

Huh? I'm ready to dunk on any project, but that's just not true... https://github.com/vesoft-inc/nebula/blob/master/README.md#l...

jamie-vesoft24• 7 years ago

Thanks for your feedback!

While our original intention is to provide a real open-source graph database project for the community, we also want to prevent cloud vendors from monetizing the project without contributing back to the community. Exactly like what's explained in this TechCrunch article: https://techcrunch.com/2018/09/07/commons-clause-stops-open-...

That being said, Common Clause seems to be the only license that can be used. Quote the article: "Academics, hobbyists or developers wishing to use a popular open-source project to power a component of their application can still do so. "

However, we will seriously consider the license issue. Please do let us know if you know any better licenses that can be used.

Much appreciated!

dang3• 7 years ago

We changed the URL from https://github.com/vesoft-inc/nebula to the project page.

jamie-vesoft24• 7 years ago

Hey, thanks for letting me know. May I ask why?

dang3• 7 years ago

Because it's more project-specific and thus stands out more. There are a ton of Github links posted to HN, obviously.

If you'd rather we switch to the other URL, let us know and we can do that.

essive2• 7 years ago

Why is this project tied so specifically to China? Honest question here....

flohofwoe2• 7 years ago

What are those specific ties other than the company behind the project being located in China?

apta4• 7 years ago

Still a red flag.

essive2• 7 years ago

I just thought it seemed a bit odd to me to have the project tied to large organizations in China so much. I’m not saying to “buy American” but I do think it’s reasonable to be perplexed.

apta4• 7 years ago

Better watch out for backdoors then.

gibsonf12• 7 years ago

SPARQL?

ablekh2• 7 years ago

Looks interesting. Good luck! Couple of questions: 1) why did you decide to create your own graph query language instead of trying to follow recent graph query languages standardization trend (e.g., see https://www.tigergraph.com/2019/02/25/the-road-to-standardiz... and https://www.tigergraph.com/2019/03/15/the-road-to-a-standard... 2) do you plan to support semantic features [i.e., RDF/S, Gremlin, SPARQL, inference]; 3) do you plan to have SDKs for popular languages [e.g., Python, TypeScript]; 4) did you benchmark (or plan to do so) Nebula Graph against competition in terms of performance [i.e., data import, query throughput and latency, inference speed, if/when supported]; 5) have you figured out which features will remain open source and which ones will be enterprise-only (I assume that you plan to follow the Open Core business model)? Thanks!

jamie-vesoft24• 7 years ago

Thanks for the questions! Good ones indeed.

1)nGQL will be compatible with the international standard GQL.

2) Currently no

3) So far the following SDKs are available:

   Python https://github.com/vesoft-inc/nebula-python 

   Java https://github.com/vesoft-inc/nebula-java

   Go https://github.com/vesoft-inc/nebula-go

4) We are working on the benchmark. Stay tuned.

5) Yes we'll follow the Open Core business model like you said. All the features that are open source today will remain open source. :)

Please let me know if you have any further questions.

ablekh2• 7 years ago

It's my pleasure! Thank you for the clarifications. I will definitely follow your efforts. :-)

shermanye2• 7 years ago

Nice to meet everyone here. As a newcomer, I would like to introduce ourselves a little bit. Nebula is inspired by the Facebook internal project Dragon (https://engineering.fb.com/data-infrastructure/dragon-a-dist...). Fortunately I was one of the founding members of the project. The project was started in 2012. Since then I've been spent all my time working on the graph databases.

The goal of Nebula is to be a general-purposed, distributed graph database. We welcome any positive feedback and technical discussion. We would love to learn to the community and to provide a product which truly satisfies customers' needs.

ragerino2• 7 years ago

I gave the tutorial using docker a try. The SQL like query language is OK until it gets to doing queries with where clauses on the data. Some form of help and auto completion would be great.

I recommend to align with the GQL standard: https://www.gqlstandards.org/

And Apache Tinkerpop. E.g. a RDF Sail implementation for Nebula. http://tinkerpop.apache.org/

jamie-vesoft24• 7 years ago

Thanks so much for trying Nebula! We really appreciate it.

Sorry about the where clause issue. Do you mind bringing an issue in this regard on our GitHub repo? So that we can assign it to relevant staff.

As to the query language, thanks for your suggestion and nGQL will surely be aligned with the GQL standard. We are keeping a close eye on it. :)

We are planning to support OpenCypher in the first half of 2020 and TinkerPop would be the next.

Thanks again! Here's our slack group btw and you may raise any question there: https://join.slack.com/t/nebulagraph/shared_invite/enQtNjIzM...

Show HN: Nebula – a distributed graph database written in C++

Related Stories