HN user
they say good technology makes "easy things easy, and hard things possible".
A lot of those java built-in libraries personally, I think they make easy things easy. Will it Just Work™ Out Of The Box™?
Hey we're engineers right? We know that the right answer to every question, bar none is "it depends on what you're trying to do" right? ;)
"just" is doing a lot of work in this construction. Regardless what a person's constellation of privileges is, it always takes an incredible amount of grinding and that's pretty damn cool / laudable / praiseworthy all by itself.
The secret sauce has never been secret
love graph or hate it, you can't really be a well-rounded dev without some knowledge of how graphs work. Forget even databases, it's one of the basic data structures all the way back from the beginnings of computer science and math courses. They're a pretty elegant way of thinking of and working with data.
“Easier” is completely subjective, no way you can demonstrate that.
I agree it's subjective. While there's no exact measurement for this sort of thing, the proxy measure people usually use is adoption; and if you look into for example Cypher vs. SPARQL adoption, Neo4j vs. RDF store adoption, people are basically voting with their feet.
From my personal experiences developing software with both, I've found property graphs much simpler and a better map for how people think of data.
It's true that RDF tries to solve data interchange on the web scale. That's what it was designed for. But the original design vision, in my view, hasn't come to fruition. There are bits and pieces that have been adopted to great effect (things like RDF microformats for tagging HTML docs) but nothing like what the vision was.
How do you see property graphs as distinct from RDF?
This is the full answer: https://stackoverflow.com/a/30167732/2920686
Can you name a property graph use case that cannot be solved with RDFS and SPARQL?
No - that's not the point. Of course you can do it with RDFS + SPARQL. For that matter you could do it with redis. Fully beside the point.
What's important is what the more fluent and easy way to do things is. People vote with their feet, and property graphs are demonstrably easier to work with for most use cases.
Linked Data tends to live in a semantic web world that has a lot of open world assumptions. While there are a few systems like this out there, there aren't many. More practically focused systems collapse this worldview down into a much simpler model, and property graphs suit just fine.
There's nothing wrong with enabling linked data use cases, but you don't need RDF+SPARQL+OWL and the like to do that.
The "semantic web stack" I think has been shown by time and implementation experience to be an elegant set of standards and solutions for problems that very few real world systems want to tackle. In the intervening 2 full generations of tech development that have happened since a lot of those standards were born, some of the underlying stuff too (most particularly XML and XML-NS) went from indispensable to just plain irritating.
Still a more grounded GQL will allow Neo4j competitors to gain on them.
Can you expand on this? What do you mean?
SPARQL is purpose built for the RDF world where you're mixing and matching a zillion different vocabularies, all of which have to be painstakingly declared and name spaced every time.
For most of us working not on the "semantic web", we typically only have 1-2 vocabularies, which is our data model, and SPARQL is super clunky to use.
Why would you want to imprint this stuff into your long term memory? Google is an extra brain with effectively infinite free space, why would I want to use mine?
Hi! I'm the original author of this post. Happy to field any questions if there are any.
This answer is not correct. MITRE is not a staffing agency, but a special purpose entity called a Federally Funded Research and Development Center (FFRDC). The other answer in this thread is better -- a key thing MITRE gets pushback from other entities about is whether or not it is ideally suited to fill some government need, or whether it's just an anticompetitive way of awarding a contract to one body instead of letting it out as the government would normally do.
Those pressures tend to focus and specialize MITRE on issues which require objectivity and high levels of expertise in systems engineering sub-areas. They are most definitely not a staffing agency, although the government can (and sometimes does) misuse them to this effect.
What's stopping you from having one table that contains A and what A points to? So you only have 2 tables?
So you can do that. Suppose you denormalize the graph into a single table. Either your denormalizing, and to connect A to many B's you have to duplicate A's data in the table, or you have the constraint that the A -> B link can only have a cardinality of 1.
This would be not a good choice for nodes with many relationships to other things, say for example "Person friended Person". It might work if the cardinality was somewhat capped, like say for example "Customer ordered Product" (a common denormalization).
For a real application, you're not going to have 1 of these. You're going to end up with 20+.
I'm yet to be convinced that a problem can't be reshaped and mapped on a traditional RDMS and yet remain performant.
Any problem can be re-shaped to any database formalism. For that matter we can re-shape everything we're discussing for a straight K/V store like Redis. The expressive power of a database isn't at issue, because any database can store any dataset. Period.
More relevant questions are about performance and about conceptual fit for the problem. First on performance, have a look at the performance growth graph here:
https://neo4j.com/blog/oracle-rdbms-neo4j-fully-sync-data/
If you reason from the computer science of how these systems work, this result makes sense.
From the conceptual simplicity standpoint, that's kind of a matter of personal taste and application. Can you do it all with an RDBMS? Sure. But all of these different tech niches exist because sometimes you want more than one kind of tool for the wide variety of jobs you need to accomplish.
I'd argue that it's conceptually simpler to think of your graph as nodes and relationships, rather than to remember each time which node/rel set was denormalized into one table, which node label was split out into its own separate table, what that join table was, how the key naming differed between tables, etc. etc. etc. (Because once you have a non-trivial sized graph, you'll have a lot of these, and maybe you made different decisions at different spots).
The prize if you do remember all of that is that you get to write quite complex SQL to join together the datastructures for non-trivial traversals, because substantial graph use cases that you can answer on the basis of a single denormalized table are going to be rare.
In terms of conceptual simplicity, compare a recursive join SQL query to a cypher snippet like "MATCH (user:User {login:'bob'})-[:KNOWS*..5]->(foaf)". The equivalent SQL is...difficult.
It's just a "use the right tool for the job" situation at its core.
If you want to self-host Neo4j in cloud environments, that's doable today in a variety of different ways:
https://neo4j.com/developer/guide-cloud-deployment/
As others pointed out in other threads, this can be done for free for startups of a certain size (https://neo4j.com/startup-program/), and eval licenses are available (https://neo4j.com/lp/enterprise-cloud/?utm_content=aws-marke...)
Many of us who have worked in the graph database space have been tempted to use the graph abstractions on top of a relational database. It's a reasonable first step. Tables are nodes, join tables are relationships and so forth.
It's also however a bit of a dead end once you go beyond the basics. The costs of joins get worse the deeper you go, and "hundreds of milliseconds" is at least an order of magnitude slower than what Neo4j would do for you.
Once you take that major performance penalty, and then layer it into more complex graph algorithms or analytics, it gets really, really painful quickly. Granted, you might not notice this if you never needed to go further than 2-3 hops in a graph. But once you start working with graphs you're not going to want to stick to such basics.
More technical detail on the difference between a graph abstraction on top of another database, and a native graph database, can be found here:
$100M is likely a placeholder amount. They're filing to go public and can adjust that figure later, it's not that strange to do so. I don't think $100M means much there, and relative to their overall valuation if $100M is all they wanted they'd likely have other options.
Can you expand on major points of how this will make the content different, (for example, Wikipedia is curated and non-notable people pages get thrown out, so if you're reading all of the web, presumably you'd know about non-notable people) -- and why it's better?
Wanna thread other known attempts at this? Maybe someone will jump in with extra detail about how these approaches are different, or what extra value we'd expect.
- OpenCyc http://www.cyc.com/opencyc/ - DBPedia https://wiki.dbpedia.org/
That anchor effect of being slow is what I meant by "installment payments". Financially, if you have enough debt you can make $300k a year and still live in a crappy apartment, because all of your income (in this weird metaphor, that's your available dev hours) are being spent on debt (fixing things that wouldn't have occurred had you not had the debt)
No, I wouldn't put it that way. It's not a problem to be embraced, it's just another variable to manage.
Just as there is the trilemma of "better, faster, cheaper: pick 2" there are others as well. Speed and tech debt are balanced against one another. It's not that I think we should embrace tech debt. I actually rather hate it. But sometimes I'm willing to trade to get to a business objective that is not related to technical elegance.
In terms of how to pay it off, it may sound like a cop-out but it just depends. The core problem is that things are always changing and we can't predict the future. So staying supple and flexible I think is the way to go. Pay off as much of it as you can when you can, but I think the main point for me is to keep the eyes on some business objective (where tech debt appears as only one variable in the equation) -- the main point is not to keep your eyes on the tech debt all the time.
I've been the CTO at two tech startups. Here's how I think about it.
"Debt" is a good metaphor because by accumulating it, you get advantages right now in speed and ability to focus on other things, in exchange for having to pay it off later. You can either pay it off by stopping what you're doing (uncommon) and fixing stuff, or you can do lifetime installment payments of more complexity, bugs, etc.
Tech debt is just one aspect a CTO has to manage. Time to market is pretty damn important. Features are important. Many other things too. This is not to say that tech debt should be ignored, but elegance of the system is not the paramount concern. Just as startups borrow money and take financing to move quickly, so too a responsible amount of tech debt can be a good thing.
In engineering we tend to over-emphasize how important the technical qualities of the system are because that's what we see every day. There are a great many things that are also important, and various business scenarios where they become MORE important than tech debt. But if you're in the stage of trying to find product-market fit, you probably have bigger fish to fry.
The biggest tech debt I've struggled with always comes from the same patterned source. You think a system is intended to have a certain feature set for a certain user base, and so you make a set of architectural decisions to match. The world moves on quickly, you adapt, pivot, whatever you want to call it, and suddenly your carefully made architectural decisions are no longer correct or wise for your new scenario, but there is no time to go back and have a "do-over".
So in essence, failure to be able to tell the future, paired with a constant need to move forward, is the source of the problem. Both of those things are going to keep happening to startups forever.
Why so much faith in non profits? If they have some specia sauce I’m not sure what it is. They still require the prisoners in order to exist and grow. And economically there’s this old joke...they’re not for profit, but they’re not for loss either.
Some differences from my perspective:
- SOA/ESB in the 2000s was driven more by business, less by available tech. As an example of what I mean here, SOA folks would usually talk about decomposing business process. E.g. Amazon sells something, there's a "shopping cart" service, a "payment" service, and a "delivery" service. (I'm simplifying, but you get the point) Modern microservies are way more granular than that, and it could be that you have 5 tech-oriented services that compose to a single sub-step of a business process. The original SOA idea had the concept right, but was (IMHO) still too coarse to make the idea work.
- Several enabling techs that came since then (containerization being an example key one) didn't exist then, which made doing the same thing 5x more painful. So the tech just grew up.
- On the data side, SOA/ESB was driven by XML and XML Schema. As someone who practiced a lot of that, it was really painful to use a document markup language to do structured data exchange. XML was popular because in the age of proprietary formats it was the first open, text-based, non-license encumbered format. So don't get me wrong, there wasn't anything better at the time, but that didn't make XML actually good for the job. Note this is not a comment that JSON is better. In 2018 we're in a world where open data standards are the default. So it's not XML vs. JSON. It's XML vs. the entire rest of the world you have so many options.
- Performance improved, how many years of Moore's law, storage & memory in between? This may not seem like a big deal, but it is. Since 2000 we gained so much in computing power, that we can afford to slather on another 10 layers of abstraction to make microservices easier on ourselves.
- Software support improved. Go back to 2001 and scaffold a java app that worked with WSDL and SOAP. Then go check out 2018's serverless framework and scaffold a node.js serverless function. Back in 2001 you may have been manually downloading JARs and putting them in a lib folder, then checking that into cvs. In 2018 it's yarn install whatever, saving dependency structure (but not binaries) in git. This amounts to hours of extra work the developer is no longer doing. Greg Lemond, the famous cyclist, was inadvertently talking about software when he said: "It never gets easier, you just go faster".
- Infrastructure improved. As with the previous point, there are hours of work you're not doing, that lets you focus on your microservice. For example, almost no one who writes a microservice admins the server it runs on. Why would you take such care to admin a single box? Just execute it and spin up another. The average developers of 2001 would be mind-blown.
...But some things stay the same....namely there's nothing free in this world, and everything's a tradeoff.
- Service decomposition to the right level is still really tricky and people regularly screw it up.
- Being able to write your system in 10 different languages is nice for flexibility of hiring, speed, and team independence. But then of course you have to maintain 10 different languages worth of software.
- We've traded debugging of simple stack traces in monoliths for ultra-complex network inspection setups where we debug the failure to pass a parameter to a remote function through a nightmare of complexity -- through the tech stacks of both services, through the networking layer, through the containerization layer, through the orchestration layer, etc. etc. Monoliths were not without their charms.
I see what you mean. And yet strangely, the need for that "Friendship" node can also be seen as a strength. How else would you assert metadata about that thing otherwise?
If A and B share an event via a friendship, you can keep track of things about that. Granted in an RDBMS if all you wanted was to draw the line then you could do it with an extra FK, but I think the conclusion you're drawing is going too far, specifically:
In SQL that wouldn't have been a remodel, because there's no difference between a relationship and an entity
There is a difference in SQL; relationships are EITHER more columns, OR a join to another table, both are possible. In graphs, "hyper relationships" (e.g. relating more than 2 things) require another node, but this is apples/oranges comparison.
While this is technically true, in the SQL world this requires wizard level skills that most SQL developers do not possess, and when you arrive at this spot, you end up with a query that performs really, really badly.
Look, between the database formalisms, they're all "complete" in the sense that you can choose any database and solve all the problems. But certain databases are going to be pathologically bad at solving certain types of problems, which is why there are so many sub-niches that persist over time.
For deep path traversals, you can do it with RDBMS, but a graph DB is going to win every time in part because the data structure is just set up for that purpose. There are other queries where RDBMS will be best too. So it goes.
Graph databases are built a lot of different ways; for example Neo4j's architecture is very, very different than something like an RDF triple store, or datastax on top of cassandra.
[Neo4j internals can be seen here](https://www.slideshare.net/thobe/an-overview-of-neo4j-intern...)...it's a bit old but I think mostly still accurate.
In graphs you have to persist nodes and edges, though you may partition nodes by label/category. In the case of neo4j there is a property store rather than a set of columns.
The media has reported on some fairly outrageous cultural excesses. Question -- did you experience these when Kalanick was there? Since Dara showed up, anything specific you can say about what was done and whether it "feels different" and how?
In some places the CEO is a distant figurehead, and whatever his or her values, the day-to-day experience is more dominated by your local crew of < 20 people. In other places, culture is truly pervasive. I feel like the reporting is saying that culture is pervasive at Uber, but it would be interesting to hear an inside perspective.
Conferences also exist to concentrate people so that vendors can market and sell. I don't think that will go out of style. Add person to person networking, people's desire to get a break from work...
I'm cringing so hard at the idea of a tech guy somewhere explaining to his boss that "we can't go serverless because we don't have enough servers for that". You know that conversation is happening in an office somewhere.
Godspeed, poor engineer.
Well sure. Consistency is great and for years most DBs people really used gave you all of the ACID guarantees. Then, huge scar came along. At extreme scale, sometimes you have to trade things for speed. But that extreme scale is a problem that frankly most people don't have.
So don't trade off a good attribute unless it's absolutely necessary. Seems sensible