Eventual Business Consistency

https://tidyfirst.substack.com/p/eventual-business-consistency

by cratermoon • 3 years ago

134 41 3 years ago

tidyfirst.substack.com

Terr_2• 3 years ago

Another approach to this kind of thing would be storing most of your data as a log of events ("event sourcing") and having "Retroactive Address Change" as its own business-process, one which can be linked to the other steps like sending a refund check or crediting an account.

In that architecture, the database-row will often remain plain and boring with just one "current address." That "projection" (a result of events-so-far) would simply switch whenever the "Retroactive Address Change" event comes down the pipeline, which would be good enough for, say, the system that is sending out monthly statements.

Anybody needing more, like the exact history of changes and realizations, would be running their own processor and projection, scanning the event-stream in a way suitable to their specific use-case.

gtramont• 3 years ago

Came here looking for this comment. It seems that Kent's re-discovering (re-inventing?) event sourcing…?

svilen_dobrev2• 3 years ago

it's event-sourcing re-discovering accounting, back to ~1500 Luca Pacioly maybe..

skrebbel• 3 years ago

I have never seen a productive application of event sourcing at scale. The idea is super elegant but there’s so much extra overhead. Your data size is easily 10x bigger, unless things nearly never change anyway (and then why do it, right?). Even simple things such as fixing a typo require a Command to be stored as an Event etc etc. Fixing fuckups by manual database edits / script runs gets harder because it means you’ll create a mismatch between the events and the main db, so you ought to inject events instead, but you’re under time pressure so what to do, and the list goes on. I wish these things weren’t issues because in a very real way event sourcing feels like the “obviously best way” to deal with pretty much any kind of structured data storage. I want it to succeed.

Does anyone know of a system that successfully did/does event sourcing at scale? How did you deal with these downsides?

bob1029• 3 years ago

Even simple things such as fixing a typo require a Command to be stored as an Event etc etc. Fixing fuckups ...

Event sourcing is a completely different way of thinking about business information. Of course you are going to run into things that look like "downsides". Consider the desired semantics of event sourcing and how your concerns run directly in the face of these semantics.

If you aren't comfortable with what qualitatively seem like "trivial" updates, then I don't think a large enough leap of faith has been made. We like to empathize with the machine but sometimes we take it a bit too far.

Event sourcing absolutely works and it works at the most massive scales the best. If you can dedicate an entire wing of your 15th floor office to "enterprise messaging and events", this stuff 5000% works. It seems to be at the small scale that it fails most frequently. At small scale, it's usually just one developer who has enough motivation to see it through, and the moment they shift their focus it falls apart.

I saw it work in manufacturing in ways I cannot do justice in HN comment size limits.

cratermoon4• 3 years ago

Does anyone know of a system that successfully did/does event sourcing at scale?

The git version control system.

Even simple things such as fixing a typo require a Command to be stored as an Event etc etc

Yup. It's a commit in git, but yes, there's a command and it's store in the git history as an event.

rswail• 3 years ago

I've implemented a system that uses event sourcing at "scale" (YMMV at what is "scale").

1. Data storage is not "10x bigger", it depends on how long you maintain the event stream in the past. Think about a bank account. The "monthly opening balance" is effectively a snapshot in time of all of the previous events for that account. So you can archive events to secondary storage as needed.

2. You don't "fix fuckups by manual database edits/scripts". You make sure that for every business level event, you have a reversal event defined as well. A fuckup is "reverse the affected events, then replay them with corrections". It requires the design at the start to understand that this needs to be dealt with.

The benefits:

* Each entity has its own state and events, it's self contained and easily mockable and testable on its own. The events are idempotent.

* Dependencies between entities is entirely driven by the events, so they are fully decoupled and can be deployed, blue/green, sharded, scaled etc independently.

* The events are all that is needed to recreate a scenario or failure, so can be saved and replayed as needed.

* Deployment of new entities and/or blue/green and/or scaling becomes trivial.

Cons:

* If there's an error in production, you have to apply reversal events and then replay the previous events to "fix" a problem. There isn't a "main DB" as such, so there's nothing to change there, but anything that takes the stream of events and turns it into RDBMS (for reporting etc) needs to also understand the reversals and replacement events. This can lead to confusion when a report generated on day X is reversed/redone and now it is "different" to what the report showed at the time. However, this can be covered by actually dealing with the reversals as first-level events in their own right.

* When you reverse/replay events for an entity, you need to be aware of side effects on other entities. For example, if you reverse/replay an order, you need to ensure that you don't trigger another round of payment processing. That means adding logic to the payment entity to ensure that it "knows" what order it got payment for so that it doesn't repeat.

* Event sourcing is not about "structured data storage", they're entirely about business logic. The events are business events. Programmers seem to have trouble dealing with the fact that the low level storage and processing are independent of the business events.

* You need to have versioning of event schemas so that you can add/remove attributes of the event type over time. Your processing has to accomodate the versioning (semver is your friend).

* If you have your event processing separate from your query processing, you cannot rely on the query when processing events for the entity itself. If you have multi-threaded and or scaled processing that deals with the entity at the same time, you need to have some form of internal caching/locking/vector-clocks to ensure that you don't have two processes updating the entity from two different events simultaneously.

So it has distinct advantages when in production and in the CI/CD deployment. It has additional complexity at the architecture and high level design phase and it needs some standardized libraries and infrastructure to support the necessary features (primarily the internal synchronization of entity/event processing between threads/processes to ensure monotonic updates of the state of the entity).

n0w• 3 years ago

The more I learn about the relational model from a first order logic perspective, the more I wonder what we would have ended up with if more people thought of data/databases in this way (as opposed to the more record/object school of thought).

Event sourcing and CRDT like distributed data using something like the Event Calculus?

ckastner• 3 years ago

Galanwe• 3 years ago

This kind data modelling happens all the time in quantitative finance, where you need to run historical scenarios (backtests).

We just call that "point in time" (PiT) data. There can be multiple dimensions of timestamping, but you often end up with:

- published date: when was this information publicly available

- received date: when was this information actually received in our systems

- effective date: when does this information starts applying

LAC-Tech2• 3 years ago

so bi-temporal extends to n-temporal. interesting...

cratermoon4• 3 years ago

There's an old saying "two is an impossible number". In programming, we either deal with one of a thing, or a collection of things. The cases where there are exactly two things are uncommon (cue HN coming up with a bunch of exceptions). In the case of points in time, even bi-temporal leaves out a third, implicit point: now.

nerdponx• 3 years ago

This is similar to what I deal with at work, where we have a large number of devices (dare I say "IoT") that all send data to a cloud system. There's the time that each record was recorded, the time it was sent, the time it was actually picked up from the queue for processing, and so on.

It also arises in ETL systems like Apache Airflow, where the "logical date" of a task run might or might not correspond to the date it was actually run.

The common element here is that the time relationships are not monotonic, and you occasionally have to make business decisions around that non-monotonicity.

jgalt212• 3 years ago

Yes, I remember someone complaining about datomic not being able to handle multiple meanings for the "as of" column.

refset7• 3 years ago

Probably this post: https://vvvvalvalval.github.io/posts/2017-07-08-Datomic-this...

If you're new to Datomic, you probably have the same misconceptions as I did regarding the use of Datomic's historical features.
bad news: you've probably over-estimated the usefulness of these features for implementing your own specific time travel. Unless you really know what you're doing, I recommend you don't use db.asOf(), db.history(), and :db/txInstant in your business logic code.
good news: you've probably under-estimated the usefulness of these features for managing your entire system as a programmer.
I believe the key to getting past this confusion is the distinction between event time (when things happened) and recording time (when your system learns they happened).

sails• 3 years ago

In case anyone is wondering, if this isn't dealt with by the system designer, then what ends up happening far too often is the data team is tasked with inferring the sequence of events based on a series of clues across this and other business systems to come up with some kind of pre-eventual-consistency guess.

LAC-Tech2• 3 years ago

Very cool article. It's a problem I knew kind of existed but never thought about formulating. I'm glad it's been at least called something since the 90s - this means maybe there's good stuff to read!

Bit annoying as I'm writing an event store in rust and now I want to make it bitemporal. But annoying in a good way.

cratermoon4• 3 years ago

now I want to make it bitemporal

Hint: make it n-temporal. Bi-temporality is then a special case easily handled.

refset7• 3 years ago

Except making n-dimensional indexes both fast and scalable is hard. Often its safest to build your way up, one index at a time, until you fully understand the workload requirements.

svilen_dobrev2• 3 years ago

heh. when i read the title it suggested completely different tangent.. - that pure technical stuff is always eventualy consistent with business needs/goals. Sometimes very close (success). Sometimes very perpendicular (failure).

Nevermind, bi(tri)-temporality is amazing concept (together with "relations Are Objects too" someone mentioned long time ago... ; Back in the days ~2006-7 i wrote a library [1] to allow for it over plain SQLalchemy.. in a bigger project that did not see light. Heh, recently i made a wrapper for xtdb for python [2].. for another project which may or may not happen too).

But it is very difficult to grasp for common programmers, let alone common people. (Actualy, similar to time jitter, and other dimensional inconsistencies. wow-and-flutter). People like to think proceduraly i.e. in sequence..

Now, 20y later, temporality of things should be default way of handling data - not a quirky write-your-own-wrongly or use whatever weird extension..

have fun

[1] https://github.com/svilendobrev/timed + https://github.com/svilendobrev/dbcook

[2] https://github.com/svilendobrev/py-xtdb-datomic

refset7• 3 years ago

[Re-posting my comment from yesterday https://news.ycombinator.com/item?id=37000572]

Part of the reason it hasn’t taken off is because of the additional complexity it imposes on programmers.

My take is that it's specifically the additional complexity it imposes on _database_ programmers - the impacts across storage design, foreign keys and schema evolution when all data is bitemporal are profound. SQL:2011 introduced "temporal tables" but the design was arguably incomplete and implementing it ubiquitously requires a real re-think of database internals, so we're now over a decade on and these capabilities are still far from commonplace.

However, I think part of the reason it hasn’t become more popular, given the benefits it brings, is just the name.

No disagreement from me there!

Disclosure: I work on xtdb.com, building a new database engine where bitemporality is the default. I also spoke with Kent about this last week and am running a webinar about bitemporality more generally on Thursday: https://attendee.gotowebinar.com/register/296060701290006793...

refset7• 3 years ago

People here may also be interested to see this analysis of the state of SQL:2011 "temporal table" feature adoption: https://illuminatedcomputing.com/posts/2019/08/sql2011-surve...

I don't think much has really changed since, and I'm not sure Postgres is any closer to addressing this natively (although there have been extensions, e.g. https://github.com/scalegenius/pg_bitemporal).

piyh• 3 years ago

This is second nature to me with payroll processing.

refset7• 3 years ago

Makes sense, payroll are usually the last people to find out when somebody's salary has changed :) Which databases have you used for it? Any tips?

pphysch3• 3 years ago

It's tempting to go "wow bitemporality is a jolly great idea! Let's make a bitemporal database where every field is bitemporal" but this elides the significant jump in complexity/overhead of such systems.

Bitemporality should be explicitly enabled on data where required, part of the database schema rather than default behavior. This could be a big chunk of your data, but unlikely to be 100%.

refset7• 3 years ago

I mostly agree, but it's the same as the argument for immutability in systems more generally: if you can afford the RAM / GC / storage costs then great. Eventually though costs will decrease and new systems will routinely capitalize on the productivity benefits in spite of reduced peak efficiency. HTAP database systems illustrate this trend.

But it's not clear to me that existing, popular database engines will ever be able to evolve to make bitemporality an efficient or easy default choice. And in the meantime countless person-years of effort are wasted rediscovering and solving the same time-related problems over and over.

A world where bitemporality was (somehow) a realistic default choice would be a simpler world, I reckon.

pphysch3• 3 years ago

A world where bitemporality was (somehow) a realistic default choice would be a simpler world, I reckon.

I think MongoDB sorta proves this theory false: "let's avoid RDBMS schema/migration hell by making everything a flexible JSON document".

As it turns out, JSON columns are a jolly good idea, but making that the dominant default i.e. MongoDB led to a bunch of other (perhaps unanticipated) issues. I believe it would be a similar situation with bitemporality.

So yeah, it would be cool if e.g. Postgres had fast native support for bitemporal fields in a way that makes them accessible without changing the whole paradigm.

awinter-py• 3 years ago

have always wanted to see time / versioning / history exposed as a first class citizen in sql schemas

and this invokes my eternal pining for datomic's approach to temporal history

proteal• 3 years ago

I built a similar system for one of my old insurance clients. Every 3 months we would send an updated list of what we wanted the insurance company to cover. This batching worked great because changes came in daily and without it we would have to amend the policy for each individual change. We tracked the date the client requested a change (posting date in the article) alongside an effective date (the date we would base cost calculations off of).

Often the client would forget to report changes to us, so the reporting delay worked in our favor. For example, we would find out out on Friday that a client needed coverage starting the previous Monday. We would make sure there were no known losses and backdate coverage in our database. The insurance carriers would receive the report in a few months and be none the wiser. Carriers will backdate coverage like this, but it can be messy with the paperwork. Tracking posting dates, effective dates, and reporting dates really cut down on the complexity of managing the day-to-day insurance needs of my client.

Eventual Business Consistency

Related Stories