Cayley – An open-source graph database

http://cayley.io

by _pius • 12 years ago

289 86 12 years ago

cayley.io

ForHackernews• 12 years ago

Not a Google project

riffraff• 12 years ago

and obviously, not "google's new graph database".

nostrademons• 12 years ago

It looks like a Googler's 20% project.

However, I'll point out that most of the actually-used open-source projects coming out of Google aren't actually "Google projects", they are "projects by Googlers released under the OSPO process". LevelDB (Jeff Dean & Sanjay Ghemawat), Protocol Buffers (Kenton Varda), Guice (Bob Lee, Jesse Wilson, and Kevin Bourillion), Gumbo (myself), and angular.js (a team within DoubleClick) all started out as small internal projects built to scratch an individual's itch that were then released externally because hey, why not. The "corporate" open-source projects have been things like Android, Chrome, GWT, Closure, Polymer, etc.

panarky• 12 years ago

  Not a Google project, but created and maintained by a
  Googler, with permission from and assignment to Google,
  under the Apache License, version 2.0.

cnbuff410• 12 years ago

Here is a brief introduction

http://google-opensource.blogspot.hk/2014/06/cayley-graphs-i...

nicklovescode• 12 years ago

I've been using a graph db as my primary database with my last project(neo4j) and it has been a pleasure. I wish a good graph database was hooked in with the new Google Cloud stuff so it could be queried/visualized/performance analyzed in a similar fashion to the BigTable demo earlier

cihangirsavas2• 12 years ago

are you using it for production purposes or for your small projects?

if it is for production, how is your read/write performance?

kaonashi• 12 years ago

Neo4j wants everything in memory, so the bottlenecks would come after your data-set size outstrips the memory available.

tehprofessor2• 12 years ago

I'm curious as to how you're dealing with the licensing costs, as they shut down most hopes I had. Granted, Neo4j said they had a "startup" license but due to family issues i never followed up. Still seems a bit extreme to me for the price, but I also (at the time) couldn't really compare it to much else.

And on the topic of Cayley... I'm stoked. I'll give it a try tonight. Easy to use, and graph databases, haven't really gone hand in hand for me.

JoshTriplett2• 12 years ago

I'm curious as to how you're dealing with the licensing costs, as they shut down most hopes I had. Granted, Neo4j said they had a "startup" license but due to family issues i never followed up. Still seems a bit extreme to me for the price, but I also (at the time) couldn't really compare it to much else.

Did you want to get support from the company that makes neo4j? If not, then you don't need a license.

hugofirth2• 12 years ago

Awesome - all my work is with graph data and graph databases any additions to the space are great news.

hugofirth2• 12 years ago

One question: I'm curious what your motivation was for providing a Gremlin style query language? As opposed to something like Cypher? Was it a case of expressiveness, personal preference etc....?

barakm5• 12 years ago

I wanted something easy to pick up -- Javascript seemed natural. I also wanted to be as agnostic as possible, because graph query languages are interesting and worth experimenting with. That's why it has submodules per-language and it's (relatively) straight-forward to write another.

Cypher's not bad either, but it's not "just Javascript". But I'm totally taking commits if someone wants to port it :)

JasonL90005• 12 years ago

Getting performance out of a graph often depends on being able to express your query effectively. The last thing you want to do is plunk down a cursor on a node and have your client start wandering around the graph, following edges. All that back-and-forth chattiness is a non-starter network-wise and it gives the database engine essentially no chance to optimize the query. I've been favoring scripts that let me use free variables--like Prolog logic variables--to describe shapes in a graph and then let the database server find bindings that match. Like 'print a.name, b.name for a, b, x, y where a is_friend_of b and a is_friend_of x and b is friend_of y and not x is_friend_of y and x.city == y.city'

mpark• 12 years ago

I'm a developer on Orly and I focus on the language portion of the project. Orly takes a different approach in this regard by providing a general-purpose functional language as its "query" language, rather than providing a language specifically designed to traverse/query the graph. This approach allows us to perform not only queries but also arbitrary computations on the server-side and take advantage of the powerful resources available on the server such as large memory, CPUs and perhaps most interesting are GPUs.

dingdingdang• 12 years ago

Demo version somewhere?

barakm5• 12 years ago

http://cayley-graph.appspot.com is a live demo on App Engine.

bratsche• 12 years ago

For people here experienced with graph databases, do you typically use the graph db as your primary data store or do you use it in combination with something like postgresql? If you're using both, can you talk about how that works and if it's been successful for you?

I'm curious because I've had a couple situations where I thought using neo4j (or some graph db) would be a natural fit for something I wanted to do, but otherwise I thought most of my other data fit into postgresql just fine. My instinct is that if I'm doing this in a web app then querying from two different databases is going to slow down my responses a lot.

klapinat0r• 12 years ago

I don't currently use graph databases in production, but I do, however, have some experience.

I use both, in a similar vein of "using Elastic Search". It could be your primary store, but it's sometimes it's more pragmatic to have two sets, a "solid" base.

This is not to say that it can't be done. What I'm stressing is that larger "changes" are hard and difficult to handle - which means a lot in the start of your process, and less in the end, as in, when you're deciding how to model your data. For instance node layout (new properties? different type? other constraints?), and mass updates are also a bit cumbersome.

Usually I have more than one SQL table (naturally) since the data I've used in graph databases is mix and match (otherwise I'd just use a fixed schema and some relational DB).

-- As for "how that works", for me it's:

routinely update from my base database with queries alike: ID > last ID.

This has worked as expected, in terms of what data you get in, and which limitations you impose (e.g. timeliness).

I'm currently making a shift to running all data in my graph database as I've settled on a model (which edges, which nodes, which properties).

querying from two different databases is going to slow down my responses

True, but depending on your data (do you know one of the queries beforehand - e.g. is your postgresql query enriching whatever your graph query returns) you might have success tying (inserting) some of the SQL data to your graph database.

nchuhoai• 12 years ago

I have actually started with using neo4j as a primary store and then moved to using it as a secondary store next to postgres. I wrote some stuff up here: http://nambrot.com/posts/14-credport-technical-post-mortem/

It led to itself as neo4j was only useful for parts of our queries and fitting everything into neo4j was just hassle when most of our data was relational.

cihangirsavas2• 12 years ago

i had the same history with neo4j

kennybastani3• 12 years ago

From your post: "The biggest issue was that we had data in the graph, that just didn’t feel right in the graph instead of a relational DB." -- What exactly was the problem other than managing complexity? I read through your posts and I didn't see any mention to some of the technical aspects of your issues with Neo4j. Was your data just so large that going the relational table route gave you a better understanding of this complexity?

JasonL90005• 12 years ago

A graph can do what a table can do and a lot more, but that's usually not the whole issue. In practice you need to consider things like speed, volume, scale, consistency, redundancy, computation, ad-hoc vs. planned operations, use of resources (disk, memory, CPU, GPU), etc. And as most NoSQL systems just aren't as mature as their table-based counterparts, you'll also have to factor in your tolerance for issues and general system crankiness. All that being said, some applications just cry out for graphs, particularly apps that involve items linked in pairs. Social apps (people linked by friendships), travel (places linked by flights), communications (people linked by messages), all of these can play hell with an SQL database but are naturals for graph databases.

jmlvanre3• 12 years ago

I agree with the idea that tables are just strict graphs and as such a graph database is usually capable of substituting a relational database. I think many graph DBs lack a sophisticated enough query language to bridge that gap. At Orly (https://github.com/orlyatomics/orly) we're working on a powerful query language, and it's nice to see that Cayley is doing the same.

querying from two different databases is going to slow down my responses

I think querying 2 different systems tends to be slower, but more importantly you lose transactionality. If you can use a single system that is at least on-par with your relational system for your run of the mill data and have a very powerful graph then that's a big win.

sebastianconcpt• 12 years ago

I use only object databases. No relational. BTree response times for the indexed objects so life is good.

Some projects I use NoSQL, mongo.

I do use Redis for reactivity and caching (but redis isn´t a database so)

Congratulations for the progress, thanks for sharing your work. I do want to see more great odbs with friendly APIs

devniel• 12 years ago

Currently, I'm using a graph database (neo4j) in a project about ontologies.

shuzchen2• 12 years ago

Actually you can have a graph database in postgres as well! Look into queries using "WITH RECURSIVE", and you can do pretty much anything a graph database could do. From the specific use case I had, there was actually no difference in performance between neo4j and postgres. I really enjoyed using cypher, and it was a pain to translate a query written in a graph-specific query language to a postgres equivalent with "WITH RECURSIVE", but because postgres was already a part of the stack I stuck with it.

baq• 12 years ago

how do you cope with cycles in the graph? do you even have cycles?

spyder• 12 years ago

If you query the two databases parallel then the response time should be equal to the slowest one of the two database responses (not the sum of them). But if you use two database then you have to maintain both of them and if they are on the same server then sharing the same resource could make them slower than just using one db.

deepGem2• 12 years ago

I first started with neo4j as a primary data store for our semantic graph but there are some limitations that are forcing us to look for alternatives.

1. Adding edges to a neo4j graph is a painfully slow process. For a large graph with a few million nodes - it'll take days. 2. Scaling neo4j on a cluster is either not possible or it's a painful process. I'm yet to discover this.

However, the greatest advantage that neo4j offers is the ability to query a path. So far, no other graph databases that I know have this ability (including Apache spark and giraph).

It's quite possible to build a directed graph database as an adjaceny list in redis. We tried this and it's super fast and scalable. However, querying is very painful.

philjohn4• 12 years ago

Openlink Virtuoso (and any other RDF store that supports them) does with the Property Paths feature of SPARQL 1.1.

kennybastani3• 12 years ago

1. Adding nodes and relationships in Neo4j does not have to be slow. It really depends on how you are loading that data in. Neo4j provides many options for data import and a transactional endpoint over HTTP for batching transactions and decreasing disk write overhead.

2. The reason Neo4j is the only database that allows you to query a path is the same reason that setting up clustering or sharding is difficult. If your graph is complex then the problem is "How do I split up these subgraphs into shards so that traversals don't have to traverse across shards?" -- Building a giant adjacency list and using that as a traversal index is a clever idea, I must admit. :)

philjohn4• 12 years ago

We use an RDF datastore (OpenLink Virtuoso, clustered edition) as our primary datastore. We use it in combination with Apache Solr to provide fulltext search over various resources that we extract and pass through an Indexing pipeline to go from RDF Graph -> Search Document.

TallTed• 12 years ago

It's worth noting that Virtuoso (produced by my employer, available in free Open Source and paid Commercial variants, http://virtuoso.openlinksw.com/features-comparison-matrix/) is a hybrid Relational/Graph/XML/FreeText storage and query engine, which natively supports SQL, SPARQL, XPath, XQuery, and many other open standards. It might satisfy the OP's needs on its own.

Virtuoso's support for open standards makes it easy to use it as a complete solution covering all the bases, or, as in @philjohn's case, to plug-and-play with best-in-breed solutions along any axis where our implementation proves not to serve your needs for any reason. (We do want to know how and why we don't measure up, so we can improve that aspect!)

AlisdairO3• 12 years ago

I've got a lot of background in RDF graph stores. It depends a lot on your usage, but I think for your typical web app, you'd be better off using a Postgres install, and making use of fancier features like WITH RECURSIVE as necessary. Graph stores often miss out on features like guaranteed relational integrity and guaranteed constraints, which I find invaluable for safe application development in the face of concurrent updates.

Graph stores are typically much slower for repetitive data that fits cleanly into a relational model. This isn't to say they're not useful - for more irregular data they're a fantastic fit - it's just that very irregularly structured data isn't the common case.

Of course, you can always use two different stores - much like many sites do with a separate lucene/elasticsearch index for text search - but your graphing needs must be relatively componentised for that to work well.

webmaven3• 12 years ago

Which ORMs support WITH RECURSIVE?

JPKab• 12 years ago

Curious: What RDF triple stores have you used, and in what kind of application?

I was looking into using Stardog for a metadata repository I was building, but we ended up (probably unwisely) bastardizing Postgres into a bunch of self-join heirarchies.

malkia• 12 years ago

SQLite added CTE (WITH RECURSIVE) recently - http://www.sqlite.org/lang_with.html

preillyme2• 12 years ago

Has anybody tried out Orly? https://github.com/orlyatomics/orly

JasonL90005• 12 years ago

I have. I'm on the team that's been developing it for the past four years. It's nice to see graph databases getting some popular traction at last. (I was into them before they were cool, of course.) I've only just started looking at Cayley but it looks like there are some significant differences between the projects. Orly is designed for high-speed, high-volume applications that need large-scale storage and consistent transactions. We're more OLTP than OLAP, which seems to be the way Cayley leans. Most graph systems tend toward analytic applications.

preillyme2• 12 years ago

Could Cayley use Orly for a storage engine?

slapresta• 12 years ago

I can't get over the logo being a 3-colored version of a 2-colorable graph.

JasonL90005• 12 years ago

Even Arthur Cayley might have smirked at that one. :)

msutherl• 12 years ago

It would probably look better without the grey nodes on the ends anyway.

nandemo• 12 years ago

That's intentional. It represents robustness and scalability: you can add an edge between a red and yellow node, without having to re-color.

(actually I've just made that up)

jperras• 12 years ago

leccine2• 12 years ago

Woo, this is probably the coolest project from Google in the last while. Is there anybody using a graph database here? What is your use case? Would you mind sharing?

jonathanmarvens• 12 years ago

At my company, we're using graph-based models for most of our NLP stuff (mainly entity extraction ATM) and our ANNs (mainly object classifiers).

- Jonathan

leccine2• 12 years ago

Thanks Jonathan. I am not terribly familiar with NLP would you mind resolving the 3 letter acronyms (ATM, ANN)?

dajohnson89• 12 years ago

Has anyone tried out Tinkerpop?

http://www.tinkerpop.com

I played with it, and it was kinda fun.

bobbriody• 12 years ago

Can anyone provide a working example for the visualization feature? The docs say that you can use the Tag functionality to label source/target nodes for sigma.js rendering, but bridging the gap between that suggestion and the actual query does not seem trivial.

tempodox• 12 years ago

  OS X:  Homebrew is the preferred method.

Over my dead body. Homebrew messes up my /usr/local, leaving its heaps of crap everywhere. I don't find that acceptable. MacPorts at least has the decency to put itself into /opt/local, which isn't normally used by anything else.

sfvisser• 12 years ago

Interesting 'triple' they use :)

    // Our triple struct, used throughout.
    type Triple struct {
        Sub        string `json:"subject"`
        Pred       string `json:"predicate"`
        Obj        string `json:"object"`
        Provenance string `json:"provenance,omitempty"`
    }

AlisdairO3• 12 years ago

This is weirdly ubiquitous in the RDF store world as well. I eventually gave up talking about 'triple stores' and called them 'RDF stores' instead :-)

jerven4• 12 years ago

It is basically required for SPARQL 1.1. (Named) graph support. Often people call them quad stores.

webmaven3• 12 years ago

Seems like provenance ought to just be it's own triple describing the relation of another triple to some source. Is this just a shortcut?

philjohn4• 12 years ago

Sad to see no SPARQL support as of yet, it looks like it's on their longer term goals though as they are query language agnostic.

Interesting to see more products entering this space.

brickcap• 12 years ago

Almost all graph databases use gremlin as querying language but I really love orient db's approach of using sql[1] for querying graphs. It feels more natural in my opinion plus it lowers the barrier to entry for people who already know about sql.

Some of the things I like about cayley

1. Switchable backends(I wonder if I can configure it to use couchdb as a store)

2. Documentation get right to the point. When I first tried my hand at graph databases I could not understand where to start but cayley's approach is pretty straight forward and it wins plus points from me for including a big dataset :)

A question: I see no mention in the docs about running it on multiple nodes (where does it stand with regards to CAP etc)

[1](https://github.com/orientechnologies/orientdb/wiki/SQL)

weihuan• 12 years ago

zxcv

Cayley – An open-source graph database

Related Stories