Quine – streaming graph with materialized views

by purplerabbit • 4 years ago

8 2 4 years ago

No affiliation. Haven't investigated much yet, but I'm blown away by the prospect of "standing queries" that you can subscribe to.

It looks like this could provide many of the benefits of Flink/Spark/ksqlDB with much less complexity.

rrwright• 4 years ago

I'm the original creator of this tool. In case it helps, the usage model is:

1. Plug Quine into Kafka or some streaming source of data (blockchain, SSE events, etc. or stream in batch data from CSV/JSON) and write one query to build those into a graph.

2. Set a "standing query" on that graph, which monitors the ever-changing graph for matches to your standing query. Each match triggers a custom action to publish it out to another system, or call back into the graph to make another update.

3. If you need a consistent unchanging view of the graph, you can issue a normal query any time and include a timestamp. You will get results to your query from the graph as it was at that historical moment in time.

The system scales horizontally. We've tested it on a resilient cluster ingesting over 1 million events per second.

Quine – streaming graph with materialized views

Related Stories