mjdrogalis

Nice to see different example of how to process historical and streaming data that doesn't use a complicated lambda architecture.

Author here. It's scary for everyone at first!

I have a couple of newsletters about selling, of which this is probably my favorite: https://michaeldrogalis.substack.com/p/week-13-what-i-learne...

Happy to answer any other questions that come to mind.

Hey there, thanks so much for the kind words. :)

Woah, woke up surprised to see my Substack posted here. Happy to answer any questions!

Hey, author here. I write a short weekly blurb (like this one) about how my startup is going. The origin story starts here: https://michaeldrogalis.substack.com/p/im-launching-4-startu...

Let me know if you're wondering about any topics in particular and I can dig up the right week's links.

It's all good—I understand the need to protect the legitimacy of what's trending here.

I think in this case, a lot of people who've been following me on other channels just wanted to show some support and weren't familiar with the etiquette here. No harm intended.

Hey, author here. No foul activity was intended.

I've been building in public on Twitter and LinkedIn over the last few months and pre-announced the launch a few weeks in advance. I think a lot of the comments here were just folks flowing through there.

Thanks Chuck! It's been cool getting feedback from you and the team the last few months. Customer-guided feedback FTW

Thanks! Yeah, I didn't want to seem cheeky throwing out a big round number, but it feels ballpark right based on all the situations I've been in.

Many thanks! Indeed, I'm looking forward to offering this as a cloud service someday.

Thanks! I had initially built ShadowTraffic to help with simple demos, but I'm realizing over time that to run a good load test, you need the same kind of statistically accurate data.

Hey everyone, founder of ShadowTraffic here. I built this product in 90 days bootstrapped on my own dollar, which was a wild experience all on its own. Happy to answer any questions :)

Is it an option to model it as a table/table join? t/t triggers when there's a change on either side of the expression.

If that doesn't work, feel free to swing by our community Slack room and we can get into the weeds. :)

Looks like a blip - we're working on fixing this right now.

Hey, ksqlDB contributor from Confluent here. In addition to the migration tooling that you mentioned (https://www.confluent.io/blog/easily-manage-database-migrati...), ksqlDB also supports in-place schema evolution (https://docs.ksqldb.io/en/latest/how-to-guides/update-a-runn...). There are some constraints on what kinds of evolution are supported, but it's something we're constantly chipping away at.

ksqlDB (http://ksqldb.io/) does exactly this out of the box. You write SQL queries whose results you can subscribe to with a client connection. Today these subscriptions are somewhat expensive, but we're actively working on making them lightweight and scalable. I'm really looking forward to seeing the end result, I think it opens up whole new use cases.

As someone who’s spent a lot of time working on data pipelines, I think this is a great breakdown of the complexity most data engineers are facing. However, I think there’s two more keys to tidying up messy pipelines in practice:

1. You need to colocate both stream processing for the data pipeline and real-time materialized view serving for the results.

2. You need one paradigm for expressing both of these things.

Let me try to describe a bit why that is.

1. You virtually always need both stream processing and view serving in practice. In the real-world, you ingest data streams from across the company and generally don’t have a say about how the data arrives. Before you can do the sort of materialization the author describes, you need to rearrange things a bit.

2. Building off of (1), if these two aren’t conceptually close, it becomes hard to make the whole system hang together. You still effectively have the same mess—it’s just spread over more components.

This is something we’re working really hard on solving at Confluent. We build ksqlDB (https://ksqldb.io/), an event streaming database over Kafka that:

1. Let’s you write programs that do stream processing and real-time materialized views in one place.

2. Let’s you write all of it in SQL. I see a lot of people on this post longing for bash scripting, and I get it. These frameworks are way too complicated today. But to me, SQL is the ideal medium. It’s both concise and deeply expressive. Way more people are competent with SQL, too.

3. Has built-in support for connecting to external systems. One other, more mundane part of the puzzle is just integrating with other systems. ksqlDB leverages the Kafka Connect ecosystem to plug into 120+ data systems.

You can read more about how the materialization pieces works in a recent blog I did. https://www.confluent.io/blog/how-real-time-materialized-vie...