dsies

Thanks for reading the post/article :)

I did not see any point to listing out the cons as others have already done it a bunch of times. I was trying to focus on what moving to a monorepo solved for us (which was optics and poor ux).

I was answering your request for a source.

The linked article talks about identification numbers that can be used to link a person. I am not a lawyer but the article specifically refers to one person.

By that logic, if the hash you generate cannot be linked to exactly one, specific person/request - you’re in the clear. I think ;)

https://gdpr-info.eu/art-4-gdpr/ paragraph 1:

‘personal data’ means any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person;

Thanks Ivan - really appreciate the kind words and support!

Re: docs - oops. We were frantically putting stuff together and linking before the docs were in that location - the link is supposed to be https://docs.streamdal.com/en/core-components/sdk/ . Fixed it in the readme.

re: libs vs sdk - we named it that in anticipation of exactly having to do some funky stuff. As it stands, we are already doing grpc, Protobuf, wasm and having it all interop across all languages is not easy - so having to introduce some sort of a “helper” binding/lib is not at all unlikely.

Besides that, the “tail” part is really a small part of the functionality - the overall idea is that the sdk/lib has access to most/all I/O of the app and is able to interact with the payload that the caller provides before it is sent on its way.

Traditional pipes aren’t really in the equation.

We went with calling it “tail” because it’s easier to explain instead of “it’s a lib that an app owner can wrap their i/o calls with to enable calling dynamic wasm”… and that’s still not the whole thing haha

Last, here’s a diagram depicting the flow: https://docs.streamdal.com/_astro/StreamdalSdk.cd7c8d45.png

Hmm good idea about the spark integration - integration is possible with basically anything that you’ve got code-level access to. I don’t know about messaging though - runtime data transformations for spark? I guess data folks would have no problem with that hmmm.

And re: emojis - we’ll tone it down - we were all working hard on docs late into the night and may have gotten a little wild with emojis haha :)

This is solid - thank you very much. We will do some more research but basically sounds like - go as low as possible, as long as the underlying libs support it.

And re: kotlin - I last worked/played with it in 2016 and recall that it was MUCH nicer to work in compared to Java.

I just did a quick cursory look and it seems like Kotlin only has slightly slower builds compared to Java and rest of perf is basically the same due to generating similar bytecode. Neat!

I know you kid - but the _data_ in this context is the data that the app is processing at runtime. Ie. If the app is reading from a DB - that’s what we are tailing.

And if that’s already clear - sorry :)

I wonder if we screwed up by calling out “tail” - it is so much more then that - it executes wasm rules on the client that are pushed to it by the server AND because we have access to the data - we can expose a UI to see it flowing… like a “tail -f” - but that doesn’t quite flow off the tongue :)

I’d urge you to check out the live demo and “tail” an app at runtime - it might be able to explain what we we are doing better than I can.

Hey there - we have documented the tech stack here: https://docs.streamdal.com/en/resources-support/open-source/

Tldr: go, grpc, Protobuf, wasm, deno, reactflow, ts

And yep, you’re right - we are using protobuf to have a common schema between all SDKs, the server and UI.

Re: sdk implementation - it’s basically implementing grpc methods, knowing how to exec wasm and doing a couple of extra things at instantiation. In real terms - it took us about a week to implement the python SDK - that’s with learning how to do wasm, Protobuf and grpc in python + 1 week afterwards to iron out edge cases.

Re: Java - that was going to be the next sdk we do but we have no idea if it needs to be a specific Java version? Should we target lowest possible Java version? We need to have a solid wasm runtime support - so maybe that limits us to newer versions of Java. Is that a problem?

I did Java a looong time ago - so need some outside input at this point haha

Good news then :) Everything stays on your network. Actually, in most situations - everything stays completely client-side. Because the rules that the client executes are Wasm modules, all data inspections and transformations occur in the client itself.

There is a server component (that you host) - but it is only used for pushing rules/Wasm down to the SDKs and for facilitating tail - that's it.

Hey peeps - Dan here - ready to answer any questions you've got!

Not being pedactic, I swear :)

Can you elaborate on the Sun comparison? I am a huge fan of Sun and what they did for computing at large - designing hardware, creating specs, their contributions to evolving unix and so on. I'm not sure how Oxide compares. Unless you're talking about "in the spirit of Sun".

What a great idea. He was part of the hacker subculture - regardless of what your opinion of him or his abilities is. This made me think of the takedown.com telnet transcripts - what a blast from the past.

Really sad day, RIP - will definitely have a drink for him tonight.

`protoc` is gnarly - `ffmpeg` unix graybeard vibes ;)

If you have a gRPC service, you'd use `grpcurl`. This one is for RESTish HTTP 1 API's where the req/resp body is protobuf - something that `grpcurl` can't handle. In other words, you'd use this if your API uses traditional HTTP methods and responds with binary encoded protobuf blobs. I would imagine this is extremely niche.

The "reimplementing" part resonates with me 100%. I've also reimplemented this solution many times and every time it has been a pain in the ass.

I built https://batch.sh specifically to address this - not having to reinvent the wheel for storage, search and replay.

For some cases, storing your events in a pg database is probably good enough - but if you're planning on storing billions of complex records AND fetching a particular group of them every now and then - it'll get rough and you need a more sophisticated storage system.

What storage mechanism did you use? And how did you choose which events to replay?

^ 100% this. Do not delete events. They are your source of truth - you shouldn't really even modify them but stripping out PII is "alright".

Re 24M+ records: create a batch runner that goes through "jobs" to perform stripping/cleaning tasks. To store state (and to organize cleaners), use a distributed store such as etcd - that way you can bookmark where you were at in the cleaning process.

Hi there, I started a company specifically centered around event sourcing - it's a saas platform for capturing all of your events, gaining the ability to granularly search them and then replay them to whatever destination.

I'd be happy to walk you through the platform - there's no lock-in, since we don't require the use of any SDK's, just run an event-relaying container which will pipe all of your events to our stores.

One big piece is that our platform is message bus agnostic meaning, we are able to hook into any message bus, be it kafka, rabbit, nats, etc.; same goes for replays - we can replay into any destination.

Check it out: https://batch.sh

The relayer is open source - https://github.com/batchcorp/plumber - if anything, the relayer can also be used for working with message busses which could improve your dev workflow for reading/writing messages, etc.

I'm late to this party but I'll chime in anyway - I love event sourcing. So much so that I actually built a company around it (and got into YC S20) - https://batch.sh

Event sourcing is not easy but the benefits are huge - not only from a software architecture perspective but from a systems perspective - you gain a whole lot of additional platform reliability.

There are roughly 3 pillars to event sourcing that have to be built (most of the time) from scratch - storing events (forever), reading stored events, replaying stored events.

Those are the 3 things I've built at several companies and it is always a huge barrier to someone taking up the pattern. Of course there are more things under the hood, but those are foundational pieces that will make the whole experience much better.

Would love to chat with folks who want to nerd out over this stuff :-)

Elixir! I've got such a mixed bag of feelings for that language.

  It's elegant but it's build-times are pretty terrible.

  It has the RPC stuff built-in but I've never seen anyone use it in production and instead folks are doing redis or 
 something else traditional.

  It has elegant process management but it's cloaked in magic (that you have to carefully learn).

This is cool - I love this space and it's indeed pretty complicated. And not that you asked for it but I'll try to add some additional insight.

It depends on what you're trying to achieve - are you trying to increase reliability by going fully event driven? Are you trying to improve just one particular flow? Are you wanting to expose certain data for data science?

Unfortunately I don't think there is a good prescriptive answer to this - it all "depends" on your situation. If this is a publicly exposed service - you probably would want to commit/record that a user signed up or posted something and THEN emit an event which will start the chain reaction of other services reacting to the event.

If the user created a post - it probably has an ID and you would probably want to communicate that in your event - but if you did not commit anything to the DB, then what are you exactly communicating?

Your goal should be to try to be as async as possible - but in some cases, it just doesn't make sense. For example - receive an HTTP request to do something is a 100% synchronous operation. But _after_ you've handled that event - most, if not all things, can be async.

Contrived example:

  1. User signed up -> UI hits public API which is backed by "Main API service" which commits to DB -> emit USER_SIGNUP event

  2. Billing service consumes USER_SIGNUP event and creates a subscription in a 3rd party billing system (and maybe emits another event)

  3. Metrics service consumes USER_SIGNUP event and starts recording metrics for this user

  4. Audit service consumers USER_SIGNUP event and creates an audit log trace

  5. --- Finally, the "Main API" service listens to all kinds of other events that causes it to update its "view" of the user.

Finally, some non-abstract advice - if you're building an event driven system from scratch, I would probably avoid doing CDC as the primary event emission mechanism and instead have your application emit events instead. You'll have a lot more granular control over what is in an event and not be tied directly to the DB schema. Your event schema will evolve and maybe it won't fit what's in the DB.

I would lean towards CDC if you are trying to retrofit an existing system.

Anyway, hopefully that's a bit helpful! I've been working on this sort of stuff for a while (and actually have a YC-backed startup that focuses on event driven: https://batch.sh).

I recently presented on event driven design at a few local meetups and made some slides, it might be useful: https://docs.google.com/presentation/d/1j6Cyid88Ca1shwEN6uyI...

Love this tool - use it to populate a bunch of "real-looking" data to test CDC functionality for Mongo.

Pretty quick too - generated 100,000 records in <3 sec.

Ahhh gotcha! Yes, that's actually one, decent sized pro for doing an event driven monolith haha

With that said, in a monolith - there is still a chance of missed events - deploy, bugs, etc. and at that point, you'd still want to employ replay to get your system back into a good state.

Your initial question was about missed events - replay is still the answer and if you design your system with idempotency - you will be able to reingest events (even if they're dupes).

Your sense is totally right and to boil it down further - you have to be OK with your system being eventually consistent.

I might be missing something - what is the issue with being able to handle a replay event stream in a monolithic service?

As the previous poster mentioned, if you are are able to handle events idempotently, you should be able to ingest events, regardless of the destination - monolithic or distributed.

In my experience, lots of banking, some trading. In both cases, it is used for history, audit and precision.

The idea is that for banking, it is not enough to just get the _current_ state - the more important thing is how someone _reached_ that state.

Adding history to transactions is not new - so rather than bolting on a history/audit mechanism, you knock out both - a higher resilience, distributed system + built-in audit/history mechanism.

Full disclosure: I co-founded a startup (https://batch.sh) which focuses specifically on solving problems in the event driven space - we do event archive, search and replay. So this space is super interesting to me :)

You are totally right - the CQRS pattern alone is not enough. In my experience, most shops that go down the CQRS path, also tend to make use of other event driven patterns like event sourcing.

As with most things, there's pro's and con's and the most obvious con here is increased complexity in exchange for higher reliability and scale.

Your advice is great, especially deciding on a structured message format such as protobuf. I would 100% avoid JSON schema as it's possible folks will forget to fill out fields. Protobuf has excellent support by now and there is a ton of support tooling for it. I'd be hard pressed to choose Avro at this point (unless you're a Java or Kafka shop that has it already in their ecosystem).

In addition, if you are already on CQRS, I would advise looking into fully embracing event driven - by utilizing an event bus of some sort (rabbit, kafka, eventbridge), you could make message passing completely async (and avoid having to use gRPC, which is another layer of complexity, as you mentioned).

A good friend of mine once said that in order to successfully do event driven arch, you must be OK with eventual consistency and I think that lies at the heart of this. If you are OK with eventual consistency, you understand the burden of complexity you're bringing on.

Thank you very much!

Ha, great observation!

Yes and no :)

We've been exceptionally lucky to have several of our close friends help us out with building an MVP (also helps that our friends have serious experience!). There's a total of 6 of us - 3 people focusing on infra, frontend and Java connector bits, which allowed myself, Ustin and another dev to put 100% of our attention on backend services + arch.

That enabled us to knock this out in a few months. Without the assist, it would probably be closer to your estimate.

Something that may be of interest to some folks: we saved a significant amount of time by not having to run our own k8s - we use EKS, it's very nice. Also, MSK - not having to run/manage ZK clusters and kafka nodes is a (costly) privilege haha