HN user
I'm building Polynya (https://polynya.dev/) and pg2iceberg (https://pg2iceberg.dev/). LinkedIn: https://www.linkedin.com/in/hasyimi-bahrudin/
If your domain is analytics-heavy, don't try to optimize your Postgres for analytics. Follow the standard pattern of mirroring your data to a data warehouse and go to town there instead. There will be upfront cost of having to pay for a warehouse and the ETL, but it will be worth it.
As a parent with 3 kids and both my wife and I working, folding clothes is the most tedious household chore. It's physically and mentally exhausting (unlike doing the dishes, which is only physically exhausting as I can wash piles of dishes while zoning out). And not to mention take so much time, a week's worth of unfolded pile of clothes for our entire household can take me 1-2 hours to fold uninterrupted (2-3 hours if I do it while watching Netflix or something).
I'm surprised no one has mentioned Multigres yet, they are the competitor of Neki. I'm a big fan of both and has been following them since they were announced last year. I think this blog post is the first time they are talking about the internals of Neki. In contrast, Multigres is being built in public since day 1, you can see their high-level architecture here [1], though I'm still waiting for more details on their sharding model.
At huge companies, it's hard to prioritize this because it's hard to pin dollar values to removing legacy code, while it's easier to show how building feature X will earn the company $Y amount of revenue. And because of that, there is also no incentive to do it, you don't get promoted by deleting old code, you get promoted by showing how your effort helped contribute to company revenue. At my previous company (100+ engineers, hundreds of microservices), teams that regularly clean legacy codebase tend to be platform teams (cost centers), while teams that struggle to get these prioritized are product vertical teams (revenue centers).
How does LTAP architecture deals with major Postgres upgrade? Is it truly zero-downtime for both upstream and downstream?
If you have something like dolt (not affiliated), a version controlled database, you wouldn’t have to slap change dates on anything OR create your historical table. The changes would be implicit in the version history.
Nope, even if I have the ability to see the exact changes of each row, I would still add timestamps everywhere, because timestamp of row change does not equal event timestamp. For example, if I have an order table with status column, and I see a CDC event where status changed from in_progress to completed, I cannot simply assume that the CDC timestamp is the timestamp when order was completed. It's possible that the source database received the event late a few minutes late due to delay upstream, or it's backfilling some missed orders a few days ago. Having a completed_at timestamp (and a bunch of other timestamps for each order lifecycle) would eliminate any ambiguities, and your data analyst will thank you for it.
It's the same thing with row history. You cannot simply assume that your row changes are aligned with the logical history of your entity.
It wouldn't be possible to do this with LTAP architecture since (I'm assuming) the individual logical changes are not visible. But honestly I've always seen SCD type 2 table as a workaround due to lack of data modeling experience in the source database. If you design your tables correctly, you shouldn't need SCD type 2 downstream.
For example, if you know your user can change emails, and there might be events from another source that is keyed by user email (e.g. marketing-related events), then naturally you will need some sort of email_history table that has historical mapping of user id to email (you probably need it for audit purposes too). Then in this case there is no need to build SCD type 2 table of user from CDC, it's already there.
How does it compare to a full-fledged durable execution platform like DBOS[0], which follows the same philosophy? Looks like River does have workflows, but it's locked behind Pro [1].
[0] https://dbos.dev
What about Multigres[0]? It builds on top of Postgres and adds HA (based on Flexible Paxos[1]), sharding, etc. They're still not production-ready, but I'm highly optimistic they will solve a lot of the problems Postgres have.
For example, with Multigres, you should be able to achieve true zero downtime major version upgrade by simply resharding [2]. With vanilla Postgres + pgBouncer, you can only achieve near-zero downtime (few seconds at most), though it's probably good enough for most use cases.
[2] https://multigres.com/docs#migrate-across-postgres-versions
It is also easier to make your code deterministic with Rust vs with Go, which is incredibly useful if you need to perform deterministic simulation testing + property-based testing. I recently wrote a Postgres-to-Iceberg data mirroring tool [1] in Go, but I ported it to Rust because I wanted the ability perform DST without fighting Go's runtime [2]. But if the domain is not critical that warrants DST, I would still pick Go over Rust any day.
[1] https://github.com/polynya-dev/pg2iceberg
[2] https://www.polarsignals.com/blog/posts/2024/05/28/mostly-ds...
The problem with the us-east-1 outage is that a lot of big companies are there, so even if you try your best not to depend on us-east-1, your third party providers are most likely there. In my previous company, we were completely down during us-east-1 outage because of other dependencies that are beyond our control.
In some cases you have no choice but to retain the data, e.g. due to compliance. But the good thing is it doesn't have to be in Postgres. You can periodically offload data to a lakehouse, then delete it from Postgres. If the table is partitioned, delete should be cheap.
I'm guessing with Neon, since their storage is a lakehouse, you get this for free.
How does Lakebase compare to OrioleDB[0]?
You should check them out, their interface pretty much looks like chat nowadays.
How does this compare to open source Deepnote[0]? We use the cloud version (BYOC) at my previous company to replace self-hosted Jupyter notebooks, and it's pretty great.
I mean that's what the Vega team is doing no? They are building the standard grammar (Vega-Lite), along with an implementation (Vega). And they are already quite established with rich ecosystem, and supports a ton of components[0]. The only thing missing is that it expects a CSV or inline data source. But it's probably not too hard to build an extension that connects to a data warehouse with an SQL query.
Why not use Vega-Lite[0]? It’s my go-to data viz DSL with Claude.
You can skip by running git commit --no-verify. I know this because I also hate pre-commit checks, and I will automatically use it when working with any codebase that has one.
Having worked at enterprise with 100+ engineers across multiple teams with complex reporting lines, I definitely agree with this. There's a lot of nuance behind decisions that docs simply can't capture. I mean in theory you can probably write every considerations that led you to make a certain decision, e.g.:
- I'm writing this service even though team X has built the same thing, because my team lead doesn't trust team X since the last time we depended on their service 3 years ago, they had a major downtime that screwed us up big time
- This service is using AWS Lambda simply because I think it's cool, despite the fact that the company has a dedicated team running k8s stack with argocd, argo rollouts, KEDA, etc for the entire company
- Service Y is written in this particular way because it's a service that is shared with another team that came from a company that was acquired, and they wouldn't use it unless we write it this particular way, and making the top execs happy is more important than dealing with a small tech debt (this is probably true)
But no one is going to write these in their RFC. But Sarah knows.
AFAIK that's what Multigres[0] and Neki[1] are trying to solve.
Looks cool, I've been waiting for someone to build this since dbt and SQLMesh acquisition. It would be great to have model versioning and support for ClickHouse SQL.
Slightly off-topic but Datalog related: is there a way for me to query data in ClickHouse using Datalog without any ETL? I want to do fraud analysis, and I've been reading about how Datalog is a lot better at these kind of use cases than SQL.
Ok then it's definitely the author's fault for clicking "Always Allow". I don't even trust my agent to run grep without approval, let alone curl.
I'm not familiar with Cursor, does it allow the agent to have access to run "curl -X POST" with no approval, i.e. a popup will show up asking you to approve/deny/always approve? AFAIK with Claude Code, this can only happen if you use something like "--dangerously-skip-permissions". I have never used this, I manually approve all commands my agent runs. Pretty insane that people are giving agents to do whatever it wants and trusting the guardrails will work 100% of the time.
I've experienced something like this at work but with data warehouse instead, and it happened multiple times (to be fair, data engineering is still fairly new where I'm from).
One example was an engineer wanted to build an API that accepts large CSV (GBs of credit reports) to extract some data and perform some aggregations. He was in the process of discussing with SREs on the best way to process the huge CSV file without using k8s stateful set, and the solution he was about to build was basically writing to S3 and having a worker asynchronously load and process the CSV in chunks, then finally writing the aggregation to db.
I stepped in and told him he was about to build a data warehouse. :P
I'm not sure why you are giving your agents write access to query your OLTP database, let alone write to it. The pattern that I use at work is:
- Read access through OLAP, not OLTP. You just need to setup a near real-time replication between your OLTP and OLAP.
- Write access through API, just like your application. You can add fancy things like approval layer, e.g. you agent cannot "ban_user(id)", but it can "request_to_ban_user(id)", and the action only happens once you approve it.
Why
Based on my experience with Claude, it's pretty damn good at doing data analysis, if given the right curated data models. You still need to eyeball the generated SQL to make sure it makes sense.
and how?
1. Replicate your Postgres into Snowflake/Databricks/ClickHouse/etc, or directly to Iceberg and hook it up to Snowflake/Databricks/ClickHouse/etc.
2. Give your agent read access to query it.
3. Build dimensional models (facts and dimensions tables) from the raw data. You can ask LLM for help here, Claude is pretty good at designing data models in my experience.
4. Start asking your agent questions about your data.
Keep steps 3-4 as a tight feedback loop. Every time your agent hallucinates or struggle to answer your questions, improve the model.
Side note: I'm currently building a platform that does all 3 (though you still need to do 2 yourself), you just need Postgres + 1 command to set it up: https://polynya.dev/
It's displayed Cloudflare Workflows dashboard: https://developers.cloudflare.com/workflows/build/visualizer...
Typically it's not pure random, instead they randomize the ordering of the 7 pieces, otherwise called the 7-bag system. Some games use different bag size, e.g. 35-bag system used in TGM3. AFAIK with 7-bag, a good Tetris player can play pretty much indefinitely, assuming that the speed of the falling pieces never increase, and that they can pause the game to take rests.
I used to play Nullpomino (an open source tetris game) for hours. It has a mode called marathon, where the goal is to play as long as possible. It's normal to see good players surviving for hours, even with increasing level. Here's an example of someone playing for 1h, 90% at level 20 (at level 20, pieces appear immediately at the bottom): https://www.youtube.com/watch?v=rHXxKywAQME
I wonder if it's possible to combine statecharts with durable execution engines like Temporal, DBOS, Restate, etc. At work we use Cloudflare Workflows for managing onboarding and payment workflows. It generates flowchart diagram that is useful for quickly reasoning about what the workflow does, which I guess is what statecharts is trying to achieve.