Graph Databases 101

http://www.cray.com/blog/graph-databases-101/

by BooneJS • 10 years ago

276 105 10 years ago

cray.com

valine2• 10 years ago

Question as someone new to graph databases: Are there any open source graph databases worth looking into?

timClicks• 10 years ago

Neo4j is a very good option.

iod• 10 years ago

ArangoDB is free open source multi model no-sql db that has decent¹ perfomance with graph support:

https://www.arangodb.com

¹ https://www.arangodb.com/2015/10/benchmark-postgresql-mongod...

rspeer7• 10 years ago

"The performance will suffer if the dataset is much bigger than the memory."

That is a huge drawback when compared to relational databases.

A good follow-up question would be: which open-source graph databases can reasonably import and store graph data that's not small -- that is, more data than fits in than RAM? Without proprietary extensions?

kinow2• 10 years ago

Depends on what kind of data and graph you are going to store/use. Neo4j is quite popular, cypher isn't very hard to learn, and it has lots of examples. Might be a good choice for a beginner.

https://en.wikipedia.org/wiki/Graph_database#List_of_graph_d...

valine2• 10 years ago

cypher isn't very hard to learn

Oh but I love a challenge. Are there reasons to choose cypher besides a gentle learning curve?

LunaSea• 10 years ago

But if you want to scale to more than two instances you will have to pay a $50k license. $50.000,00 dollars. Which is far too expensive for startups.

kawera3• 10 years ago

Cayley is a good option; we use it in production.

https://github.com/google/cayley

jalfresi• 10 years ago

Might I ask what sort of dataset size, servers etc you are using? I'm looking for an graph database and Cayley seems the best fit, though I'm not sure what sort of limits on the data there would be in the real world.

jedc• 10 years ago

Is Cayley mature enough for production? I thought it was still relatively new. Would love to know a little more about how you're using it.

arupauqa• 10 years ago

What's the story on inserting data into Cayley? I think every single code example I have seen on it only shows traversing graphs with it.

Also, if you don't mind me asking, how does it not being a property graph affect modelling your data and queries? At a glance it seems that queries would get significantly more complex if you wish to take several properties of a vertex into account.

twic• 10 years ago

Or if you like the idea of using a graph database developed by a secretive organisation full of geniuses dedicated to collecting and organising the world's information, but would rather it was public-sector, there's Gaffer:

https://github.com/GovernmentCommunicationsHeadquarters/Gaff...

emehrkay• 10 years ago

Yes. I love the Tinkerpop stack (http://tinkerpop.incubator.apache.org). I am currently writing a Python library as I develop an application around it called gizmo (https://github.com/emehrkay/gizmo).

jerven3• 10 years ago

In the rdf space there are a whole bunch. Graph as in sparkling gives you : Virtuoso Blazegraph Jena Rdf4j Ruby.rdf

There are more but these are opensource and I know them. And money more commercial ones.

rusabd• 10 years ago

Virtuoso

SanderMak• 10 years ago

We use http://orientdb.com/orientdb/, seems decent so far.

whazor• 10 years ago

There are multiple systems out there, however I have my doubts. It is important that your data does not get corrupted, and that your transactions will not get lost. Furthermore, speedups are possible with certain indices. That is why I personally would want to see some more safety/speed analysis and comparisons between the different systems.

marknadal3• 10 years ago

(Full disclosure: I'm the author, we are VC backed) https://github.com/amark/gun is an Open Source graph database with Firebase like realtime synchronization.

phpnode5• 10 years ago

We've had this discussion before but that product is not a graph database, it has no graph traversal features

rail2rail• 10 years ago

We're using TitanDB. One of the main benefits for us is that AWS has provided backend integration with DynamoDB. This affords you practically infinite and painless scaling on a pay-as-you-go model. Love it.

https://aws.amazon.com/blogs/aws/new-store-and-process-graph...

espeed3• 10 years ago

Look at Blazegraph, an open-source GPU-accelerated distributed graph database.

See previous discussion: https://news.ycombinator.com/item?id=11197880

karussell• 10 years ago

Here is an (old) overview: https://docs.google.com/spreadsheets/d/1XGapLHpSd2Ta8019VlwY...

d0ne• 10 years ago

http://stingergraph.com/ - From Georgia Tech

TimPrice• 10 years ago

1-Would it be more efficient to store objects that contain its relations if you only do (simple) read operations? (e.g. JSON database)

2-Instead, do graph DB engines try to break through bottlenecks for big data and analytics scenarios?

thesz6• 10 years ago

It introduces false dichotomy "graph vs relational".

In fact, most (if not all) graph algorithms can be expressed using linear algebra (with specific addition and multiplication). And matrix multiplication is a select from two matrices, related with "where i=j" and aggregation over identical result coordinates.

The selection of multiplication and addition operations can account for different "data stored in links and nodes".

So there is no such dichotomy "graph vs relational".

grandalf3• 10 years ago

True but highly irrelevant to why anyone might choose to use a graph database (or choose not to use one in favor of a relational database)...

sklogic10• 10 years ago

Of course you can express anything on top of a relational model. But for graphs such a representation would have been awfully inefficient. For this reason, CADs never even tried to switch to a relational data storage once that fancy new relational databases appeared, most of the professional CADs are still using good old graph databases.

thesz6• 10 years ago

I beg to disagree.

I am part of the team developing Russian CAD system [0]. It uses what one can consider a hypergraph db (relation includes many objects), but that DBMS system has queries on par with SQL. And they prove themselves very useful in development of CAD.

What you describe can be explained with development inertia. Most CADs are C/C++ and these languages are not very well suited for changes that go through all code base (change of storage engine and data model).

I also made experiments during a dev of graph analytics engine in one part of my experience. The relational model (actually, linear algebra model) has proven itself very competitive. It allows for easy distribution of data, the operations over distributed data are close to optimal, etc, etc.

[0] http://dd.ru/

sophacles2• 10 years ago

Strictly speaking yeah. Practically speaking: Not really true.

Just because something can be done, doesn't mean it can be done easily or well. I've done a lot of work with relational databases, and I love them for a lot of data sets. But I also have done a lot of work with graph databases - and they make working with graph shaped data a pleasure. I could do a graph in SQL, it's even moderately straight-forward in postgres these days by using WITH RECURSIVE - but it's still not as simple as just loading orient or arango for those tasks.

It's the same reason I keep multiple knives in my kitchen. Sure I could do everything with an 8" chef's knife, but the paring knife and the boning knife just make some tasks easier.

tkinom2• 10 years ago

Facebook has a very good paper descriptions on how they do graph on top of relational database. Google "facebook tao" for details.

I read that and implement my own version with SQL in < 500 lines of python code and found it just perfect for my own use cases. I can easily query any edges, notes in web speed ( < 10 ms) from databases with millions of nodes, edges, GBs of info.

I am curious what I might be missing with that approach as compare to a real graph database?

polymeris• 10 years ago

Curious, since you seem to have experience with both graph and relational databases (and I have not really worked much with the former)... if I had a graph where I just want to compute the shortest weighted path between two nodes, which model would suit better and how much difference would it make in terms of performance?

cbsmith• 10 years ago

linear algebra != relational calculus

Your argument is effectively because Haskell can be implemented in C, there is a false dichotomy between the two languages.

AdamN• 10 years ago

Everybody's focused on graph databases here but let's talk about Cray! One of the most forward-thinking computer technology companies ever to exist is starting to get out there again. If they got a few hundred million dollars from an outside investor, they could do friggin' incredible things. They already do incredible things but not out there in the way it so easily could be.

fulafel2• 10 years ago

Cray is a brand name that has been passed around between half a dozen companies (including Sun and SGI) dotted by various kinds of product reboots and commercial failures. Cool stuff but supercomputing isn't the most financially sound business it seems. The current name holder is the company previously called Tera, originally famous for making an aggressively multithreaded HPC computer.

mitchty2• 10 years ago

I'm not clear how Sun ever owned Cray care to explain?

The provenance was Cray Research -> SGI -> Tera/Cray according to those that have been around since the Cray Research days.

Source: err, I work here and asked a couple people a few cubes over. :)

The Sun deal was apparently more SGI wouldn't be caught dead with a supercomputer that ran on sparc so it got sold off to Sun.

pinewurst• 10 years ago

"an aggressively multithreaded HPC computer" - still sold as Urika-GD, marketed as a dedicated graph appliance. That's where Cray Graph Engine originated.

rrrrtttt• 10 years ago

Cray nowadays is purely a government contracting shop (think healthcare.gov in terms of technology chops).

kendallgclark• 10 years ago

Wow. That's just patently false. Good job bro!

SloopJon• 10 years ago

The author's next post describes RDF and SPARQL in the context of the Cray Graph Engine:

http://www.cray.com/blog/how-cray-graph-engine-manages-graph...

gtrubetskoy• 10 years ago

I have spent a lot of time figuring out how to deal with a large graph a couple of years ago. My conclusion - there will never be such a thing as a "graph database". There are many efforts in this area, someone here already mentioned SPARQL and RDF, you can google for "triple stores", etc. There are also large-scale graph processing tools on top of Hadoop such as Giraph or Graphx for Spark.

For the particular project we ended up using Redis and storing the graph as an adjacency list in a machine with 128GB of RAM.

The reason I don't think there ever will be a "graph database" is because there are so many different ways you can store a graph, so many things you might want to do with one. It's trivial to build a "graph database" in a few lines of any programming language - graph traversal is (hopefully) taught in any decent CS course.

Also - the latest versions of PostgreSQL have all the features to support graph storage. It's ironic how PostgreSQL is becoming a SQL database that is gradually taking over the "NoSQL" problem space.

yeukhon• 10 years ago

Years ago PostgreSQL already support recursive query, and in Oracle you have CONNECT BY. I have only used the recursive with once and it was just a quick demo, but my understanding is update is extremely expensive.

amirouche5• 10 years ago

In my point of view, the fact that you can add an expert index very easily to a graph database written in a modern language (say no C/C++) makes it even easier to customize an existing graph database to suit your direct need. In turn, storage and runtime can be tunned more easily. Making so easy to have the performance you need. But at the end of the day not dealing with algreba is the best.

valhalla• 10 years ago

If anyone's curious about Network Science/Graph Theory in general here's a free online textbook used by a grad student friend of mine

http://barabasilab.neu.edu/networksciencebook/downlPDF.html

rfreytag• 10 years ago

Blasted down arrow is so close to the up-arrow I clicked down when I meant up. Someone please cancel out my mistake.

GFK_of_xmaspast• 10 years ago

Be careful about Barabasi: https://liorpachter.wordpress.com/2014/02/10/the-network-non...

(FWIW, I had previously read some Barabasi papers and had come away seriously unimpressed, see also https://news.ycombinator.com/item?id=9555547)

gilleain• 10 years ago

I misremembered Albert-László Barabási for Laszlo Babai - I was wondering why you were unimpressed!

Yes, scale-free networks (and so on, and so on), are oversold. Is his work really that bad though?

amirouche5• 10 years ago

I am huge fan a graph-y stuff. I did several iteration over a graph database written -- in Python -- using files, bsddb and right now wiredtiger. I also use Gremlin for querying. Have a look at the code https://github.com/amirouche/ajgudb.

Also, I made an hypergraphdb, atom-centered instead of hyperedge focused in Scheme https://github.com/amirouche/Culturia/blob/master/culturia/c....

Did you know that Gremlin, is only srfi-41 aka. stream API with a few graph centric helpers.

edit: it's srfi 41, http://srfi.schemers.org/srfi-41/srfi-41.html

lobster_johnson• 10 years ago

I've seen people using graph databases as a general-purpose backing store for webapps/microservices. What are people's opinions about this?

My feeling is that graph databases are not suitable/ready for — for lack of a better term — the kind of document-like entity relationship graphs we typically use in webapps. Typical data models don't represent data as vertices and edges, but as entities with relationships ("foreign keys" in RDBMS nomenclature) embedded in the entities themselves.

This coincidentally applies to the relational model, in its most pure, formal, normal form, but the web development community has long established conventions of ORMing their way around this. The thing is, you shouldn't need an ORM with a graph database.

Xyik• 10 years ago

One of the biggest challenges in databases is handling concurrency and sharding, wish this would have talked a bit more about how that changes between a graph database and a relational database.

jmartins• 10 years ago

Anybody know dgraph.io? it's a Scalable, Distributed, Low Latency, High Throughput Graph Database over terabytes of structured data. DGraph supports facebook GraphQL as query language, and responds in JSON and the storage engine is facebook rocksdb a very fast database. see more in https://github.com/dgraph-io/dgraph

Graph Databases 101

Related Stories