Scaling to 1M active GraphQL subscriptions on Postgres

https://github.com/hasura/graphql-engine/blob/master/architecture/live-queries.md

by tango12 • 7 years ago

353 80 7 years ago

Hasura is such a promising application platform. I tried it a few months ago and was really pleased. In a way it kind of felt like RethinkDB, except GraphQL and Postgres. Was very easy to turn on with Docker Compose and play around. Looking forward to 1.0!

z3t4• 7 years ago

I guessed it would use the pub/sub capabilities of postgres. But it seems from the readme that it use polling, although using the same query which allows cache optimizations. The naive implementation can often be surprisingly effective.

tango129• 7 years ago

Yep. We experimented around a lot with pub/sub (listen/notify, wal-lr). But it was getting hard to scale and heavy/spiky write loads needed careful thought. This seemed like the right way to get started esp. given that Hasura can get the entire sql query including authz, before moving to tackling that problem more effectively[1].

For folks that want to play around with wal and listen/notify:

https://github.com/hasura/pgdeltastream https://github.com/hasura/skor

[1] https://github.com/hasura/graphql-engine/blob/master/archite...

Thaxll• 7 years ago

I was curious about the code so I went to check it, it's written in Haskell, never seen Haskell before, it's really hard to follow for someone not used to it: https://github.com/hasura/graphql-engine/blob/master/server/...

Not sure how people enjoy FP.

mrkurt• 7 years ago

It's worth writing something in an ML or Haskell sometime. Functional programming is great, and it makes my code in things like JavaScript much more robust.

Cyph0n• 7 years ago

Scala is probably as far as I’ll get when it comes to FP.

lpage• 7 years ago

Haskell and ML languages look like symbol soup when you're used to mainstream/C style languages. Once you understand some of the basic syntax they become quite readable and easy to reason about as a series of transformations.

Step one, which applies to most functional languages, is knowing how to read a Hindley-Milner type signature [1].

Armed with that you can often gain a surface level understanding of what some function does - and what the program does in aggregate - just by reading type signatures and seeing what calls what.

Hoogle [2] is a good resource for figuring out Haskell syntax, like what's up with all of those $ signs.

Converting OCaml to ReasonML syntax with sketch or reason-tools [3] makes OCaml less foreign.

With basic knowledge of Haskell and ML syntax you'll find many FP languages readable.

[1] https://drboolean.gitbooks.io/mostly-adequate-guide-old/cont...

[2] https://hoogle.haskell.org/

[3] https://reasonml.github.io/docs/en/extra-goodies

jakear2• 7 years ago

“Quite readable” is a stretch... I know of no other language that needs a hoogle equivalent.

That being said, my experience with Haskell is limited to working on some meta-meta-programming research in college, where every file used at minimum 3 different GHC extensions. So I may be biased.

seer• 7 years ago

Thank you so much for the reference to “mostly adequate guide to FP” book. It looks incredibly easy to read. As someone who is fluent in JS and dabbled in FP this is right down my alley, and the style of text itself is actually quite funny and conversational - I love that.

I’ve noticed that a _a lot of_ stuff written about FP tries to emulate this fun and easy going style - I wonder why that is. I still refer to clojure’s “re-frame” docs as the epitome of technical literature - a text so good you can read it in its own right just for the fun of it.

I don’t want to diminish other languages’ communities of course, as this is totally anecdotal. I’ve found awesome prose in other places as well (mostly ruby/python) but I just encounter those funny gems more often reading about FP.

coldtea• 7 years ago

it's really hard to follow for someone not used to it

So like anything else?

Not sure how people enjoy FP.

By virtue of actually having tried it?

bishala• 7 years ago

Not really. Some other functional languages (eg. F#) don't seem as arcane as Haskell.

robotmay• 7 years ago

I've said it a few times on HN in the past but Haskell is by far and away the hardest language I've tried to learn. I have no formal grounding in maths/CS and I think a lot of the people who enjoy it really struggle to communicate with those of us who don't come from that sort of background.

In contrast I picked up Rust and Elixir in a couple of weeks apiece, so I don't feel like I'm particularly dense; I just suspect that it needs a certain way of thinking to get the best from Haskell :)

andy_ppp• 7 years ago

I’m along the same opinions, if there was a language as complex as Haskell that gave more consideration to beginners above terseness [naming concepts better] I’d be able to accept more of it. The guy next to me is determined to hatchet in all of the functional paradigms into Typescript and he gets to decide this. I can’t help think you’re average JS dev is going to look at Maybe.may2 and be completely bemused...

hota_mazi• 7 years ago

And FP developers in general tend to push conciseness to an extreme, especially with this kind of shenanigan (from the very source you linked):

    import qualified Data.ByteString.Lazy as BL

It's ironic that after all this obfuscation, the author of the source still finds it useful to do ASCII art and align everything neatly. Talk about misplaced effort.

stoksc• 7 years ago

Pretty sure that’s done automatically like go-fmt or other similar tools. It’s zero effort and makes it easily to read. Also lots of languages have that import syntax (python import pandas as pd).

tirumaraiselvan• 7 years ago

Not sure how people enjoy FP.

See The Blub Paradox [0]

0 : http://www.paulgraham.com/avg.html

jose_zap• 7 years ago

Anything you are unfamiliar with will obviously look alien and difficult.

I actually find their code quite readable, but that’s because I have experience writing Haskell.

thegoleffect• 7 years ago

When I first saw Hasura, it was AGPL licensed which makes it unusable in a lot of use cases so we passed on it. Looks like they converted it to Apache license two months ago. I suppose it is worth another look, I'm glad they made the change.

StreamBright• 7 years ago

Nice scaling numbers. Can you use Hasura without Kubernetes or Docker?

tango129• 7 years ago

You can compile the binary yourself, but otherwise docker is the way to go.

What’re you looking at?

agrippanux• 7 years ago

Hasura is fantastic. I am using it for several projects atm. Coming from Prisma it's refreshingly easy to use and well thought out. The security/ACL model is esp. nice and easy to grok.

jazoom3• 7 years ago

I know there's an issue about this but it doesn't really indicate Hasura's intentions regarding it:

Is supporting CockroachDB something Hasura is actively working on?

tango129• 7 years ago

It’s on the cards!

Don’t want to spread ourselves too thin too early. Also working with the yugabyte folks closely to have Hasura supported with their 2.0 release.

https://twitter.com/karthikr/status/1128003465370693636?s=21

jazoom3• 7 years ago

Don’t want to spread ourselves too thin too early.

I certainly understand that, but I wonder since CockroachDB is essentially Postgres, if it wouldn't be easier to ensure compatibility from the start. It seems like it'd be easier to just avoid using special Postgres features that aren't in CockroachDB than to go back later and re-write all your code that relies on those features.

londons_explore• 7 years ago

we’ve currently fallen back to interval based polling to refetch queries.

This fancy "subscribe to events when the results of this query change" system boils down to just polling the database...

aidos6• 7 years ago

Sometimes the simple option is the best. Having said that, it’s not that simple, for them. I believe it relies on being able to fetch and dissect different results for different clients in the same query (all while applying their own form of RLS).

GorgeRonde2• 7 years ago

    Future work:
    Reduce load on Postgres by:
    - Mapping events to active live queries
    - Incremental computation of result set

Fake it till you make it.

Meanwhile: https://opensource.janestreet.com/incremental/

aidos6• 7 years ago

I’m not entirely sure how the library you posted is related. Because it does incremental computation? It doesn’t seem to have anything to do with Postgres. As far as I can tell it looks more like mobx.

josephg• 7 years ago

Yep. I’d love to see an implementation of this in the wild built on top of logical decoding:

https://www.postgresql.org/docs/9.6/logicaldecoding-explanat...

ZitchDog2• 7 years ago

Interesting idea. Logical encoding would give you the updated row, but how would you check it against each query to see if the query needed to be rerun? Sounds tricky.

Liron• 7 years ago

Me too. Here are my thoughts on the current state of "realtime databases": https://hackernoon.com/why-eve-will-be-perfect-for-realtime-...

aidos6• 7 years ago

Just another little shout out for Hasura here. It’s a really lovely system to use and the team behind it are awesome to deal with. Super friendly and helpful.

We evaluated a lot of stuff recently before settling on Hasura. Prisma, aws appsync, postgraphile, roll our own socketio backend etc. I had a few hesitations about Hasura. In particular I was hoping to lean on Postgres RLS for the authorisation, and to be able to customise the graphql endpoints a bit more. After hearing the rational from the guys about how their system is better than RLS and seeing the patterns they have for implementing the logic I’m really happy we made the leap.

We’re still in early days, but it’s been really solid and a real pleasure to use.

(No affiliation, but happily paying for their support licence)

ndarilek5• 7 years ago

Out of curiosity, what is the rationalle for abandoning RLS?

In a previous project, I used Postgraphile and enjoyed it. Sure, RLS and PLPGSQL were a bit awkward, but I knew they were well-tested and had lots of eyes on them, so I felt comfortable that if an RLS policy blocked access to a row, then that particular role wasn't getting access. I also enjoyed working with a text-based format that more clearly fit within my other workflows of, well, working with other text-based formats. :)

Another part of it is that I'm blind, and Hasura seems to make doing things outside of its web interface somewhat painful. Sure I can write a YAML migration by hand, but the YAML migration format seems more machine-friendly than user-friendly. Yesterday I needed to create a function, and the SQL web interface wasn't showing me the line number on which my function had an error. Ultimately I dropped to the psql command line, cut-and-pasted the function in manually, and right away got back the line number and fixed the issue.

Please don't get me wrong. I'm not trying to hate on Hasura. But the fact that I can't just drop to a non-YAML text-based format and throw down a few checks to secure my tables has been an endless source of frustration for me. So if there is a non-NIH reason for abandoning RLS in favor of a separate security system, then maybe knowing that might help me be a bit less annoyed. :)

aidos6• 7 years ago

To be totally honest, I haven’t implemented anything using RLS, it just seemed the way to go since it’s native Postgres. The Hasura team gave me the same explanation as they give in this article - when you use RLS, you have to rerun every query for every client to get your results. By doing it in the application layer, you can run a single query instead.

In terms of the setup and migrations, I also have the same reservation about it being harder to do with text. So far I’ve left it to one of the other guys to implement so it hasn’t been painful for me, yet!

Edit to add: I believe they built their version of RLS before Postgres had it. So it was more a case of Not Invented Yet

tango129• 7 years ago

(I work at Hasura) Thank you for the note! This can definitely be made easier, and we’re working on this.

You can also add sql migrations as a sql file and that’s something we should document better! https://docs.hasura.io/1.0/graphql/manual/migrations/referen...

Coming back to RLS, doing what we’re doing to fetch multiple results for different clients is not easy. Connection variable vs being in the same sql statement essentially.

https://github.com/hasura/graphql-engine/blob/master/archite...

Further, owning RLS helps us target more hosted Postgres vendors, and other Postgres flavours that don’t support RLS well. RLS on Heroku doesn’t work: https://devcenter.heroku.com/articles/heroku-postgresql#conn...

Owning RLS does have a few other advantages. Having a unified experience in bringing that authz experience to “remote schemas”, for example.

(Typing on my phone, apologies for typos etc)

ZitchDog2• 7 years ago

I can't speak for the author, but having used RLS for authentication I would not recommend it. It complicates nearly every EXPLAIN plain, and it can cause performance headaches in even the simplest of queries.

We had to mark many internal functions as LEAKPROOF in order to get them to perform. It was not possible to move our database into RDS, so in the end we had to abandon the RLS authentication layer.

k__2• 7 years ago

Does AWS AppSync even play the same game as Hasura?

aidos6• 7 years ago

Not entirely but it crosses over. Appsync (from what I could tell) has a hacked version of Apollo for the front end and then it’s a configuration based system to provide graphql on top of dynamodb (someone can correct me in all of the above). But if you think of the backend of appsync as being a graphql engine with subscriptions and and a permissions system then there are a lot of similarities.

marton782• 7 years ago

Hasura sure looks interesting, but two things make e hesitant:

1. The overly complicated schema it generates. For example, given an integer id, one can (and must) use various operators to query that id, e.g. `user(id: { _eq(1) })` rather than just `user(id: 1)`, where other operators are `_gt`, `_lt`, etc. To make this possible, hasura defines a comparison object, which results in a very long schema. However, often the only comparison that makes sense is equality, e.g. for IDs, so this should be optional.

2. More importantly, using Hasura establishes coupling between the DB and the GraphQL schemas. What if at a later stage one decides to remove information from the DB and move it to an external source? GraphQL was designed to hide such implementation details. With Hasura this is no longer possible.

pyankoff• 7 years ago

Re 1: You can do `users_by_pk(id: 1)` for a while now.

sbecker• 7 years ago

Regarding your second point, I believe it is still possible to query an external source. Read the docs on “Remote Schemas” - https://docs.hasura.io/1.0/graphql/manual/remote-schemas/ind...

lubesGordi2• 7 years ago

I'm curious about the underlying push tech. Is Hasura using firebase to handle that? Then firebase relies on different push services depending on the client, like Apple's push service for iOS and http/2's push for web (which is supported by what percentage of browsers)?

pier252• 7 years ago

Firebase? No, not at all. It's all Hasura and Postgres.

lubesGordi2• 7 years ago

What I'm getting at is what is the underlying tech handling push to clients? Reading further I see that the graphql libraries handle subscriptions via websockets. But elsewhere in this github I was seeing firebase. So it's a little confusing. Websockets are pretty ubiquitous now but but from what I understand maintain a tcp connection to the server. That has scalability implications.

machbio2• 7 years ago

Write Ahead Logs would have been amazing - hate that AWS RDS does not support WAL - could have saved so much time for some of the applications we build.

gunnarmorling2• 7 years ago

WDYM by "RDS does not support WAL"? I wouldn't know how Postgres works without them, and they are accessible using logical decoding. wal2json is supported on RDS and used by tools such as Debezium for streaming data changes into Apache Kafka (Disclaimer: I work on Debezium).

machbio2• 7 years ago

Thank you, had missed this announcement.. I did check Debezium a while back, found it appealing

pier252• 7 years ago

We evaluated Hasura and the only problem we found is the integration with our own authorization system. Otherwise Hasura is quite frankly amazing.

Our auth system has many roles per user, then each role has its permissions, pretty common setup. The problem is that Hasura expects a single default role per request to then evaluate against its permissions. The Hasura team has been looking into accepting an array of roles and merging permissions and whatnot, but AFAIK this hasn't been solved.

If you start with Hasura from scratch this is not really a problem, it happened to us because we had to figure out how to integrate Hasura with our current permissions.

contrasti• 7 years ago

the only problem we found is the integration with our own authorization system

But integration with external authorization systems is an extremely basic requirement for every piece of software, certainly on the top 5 on the requirements list for every serious solution - I can not understand how did you forget about this?

sk5t• 7 years ago

I haven't found this ("integration with external authorization systems is an extremely basic requirement") to be the case at all; most solutions ship with tightly-coupled access control mechanisms, and it is an exercise for the user to figure out how to wield them effectively.

In Hasura's case, it looks like the flow of data to the authorization bits shows a simple/naive approach, although it also appears this is something Hasura are working to correct. Even as it stands, moderately complex authz logic could be implemented via x-hasura headers or in-database associations.

themccallister• 7 years ago

I’ve been using Hasura for around 3 months now. It has changed the velocity and way I approach our projects. I have a small team that works as a startup inside of a larger organization and this was crucial to delivering some key projects... plus very little code. The event trigger really help reduce our costs as well, using the Serverless.com framework with Lambda Go has been a huge success in our organization!

Thank you to the Hasura team for all you do and in turn allow me and my team to do!

rishav_sharan• 7 years ago

As an Indian engineer, Hasura is one the few indian startups that i really am proud of. They are innovative, distinct, product focused and super young. The only other company which made me feel like this was Notion Ink, but those guys seem to have faded away.

I wish Hasura the best and would be cheering for them from the sidelines

gunnarmorling2• 7 years ago

I'm impressed by the numbers, but I'm having some doubts about the general idea of exposing your database via GraphQL.

It seems to expose your internal data structures, and any change to those will immediatly impact that public GraphQL API. Also it seems that this approach would only work well for applications without any business logic, or with business logic solely implemented in the database. At least the applications I've been working on wouldn't fall into this category.

But in any case, impressive work and demos!

aleksei• 7 years ago

You aren't limited to a single database though, so you could stream your main database to a public facing one with a more stable structure.

But you'd have to handle changes through another API, or at least route them differently, and then worry about consistency so probably not ideal for all business cases.

tango129• 7 years ago

True!

We’ve added event triggers to Hasura to kind of support this pattern via Hasura itself. So you can create “action” tables that basically have a log of request data (a mutation inserts an action). Hasura will then call an event handler which can run with the action data, user session information, related data in case there are any relationships etc. This handler can then go update the tables that can be queried from.

Ofcourse, if the eventing system is in-order / exactly once quite a few use-cases become feasible ;)

he0001• 7 years ago

How does it handle transactions and locking? It seem to support multiple databases, does it handle distributed transactions?

rj5• 7 years ago

I can’t wait to try this out!

Scaling to 1M active GraphQL subscriptions on Postgres

Related Stories