lsuresh

Over at Feldera, we focus on IVM for SQL, but incremental computing problems show up far and wide: UIs, spreadsheets, control planes, compilers and more.

When I last used it for such use cases, it was better to decompose the problem into something incremental (so fixed placements become constants). Most of the latencies we saw were spent in the presolve phase which scaled with overall input size.

I'm a big fan of the CP-SAT solver. It was a remarkable piece of tech to learn about (especially Peter Stuckey's talks on lazy clause generation [1]).

I'd used it in a past life to build a Kubernetes scheduler [2] and tackle some cluster management problems.

[1] https://www.youtube.com/watch?v=lxiCHRFNgno [2] https://www.usenix.org/system/files/osdi20-suresh.pdf

We built Feldera's engine in Rust: https://github.com/feldera/feldera

There are some solid ideas here and would definitely apply to the IVM engine we're building. I'm curious if some of these effects could play a role in faster rust compilation times (e.g. nopanic..)?

We use an in-product profiler (discussed here: https://www.feldera.com/blog/introducing-feldera's-visual-pr...), along with CPU profiles to identify where in the code we're spending time.

Good questions!

Feldera tries to be row- and column-oriented in the respective parts that matter. E.g. our LSM trees only store the set of columns that are needed, and we need to be able to pick up individual rows from within those columns for the different operators.

I don't think we've converged on the best design yet here though. We're constantly experimenting with different layouts to see what performs best based on customer workloads.

Feldera co-founder here. Great discussions here.

Some folks pointed out that no one should design a SQL schema like this and I agree. We deal with large enterprise customers, and don't control the schemas that come our way. Trust me, we often ask customers if they have any leeway with changing their SQL and their hands are often tied. We're a query engine, so have to be able to ingest data from existing data sources (warehouse, lakehouse, kafka, etc.), so we have to be able to work with existing schemas.

So what then follows is a big part of the value we add: which is, take your hideous SQL schema and queries, warts and all, run it on Feldera, and you'll get fully incremental execution at low latency and low cost.

700 isn't even the worst number that's come our way. A hyperscale prospect asked about supporting 4000 column schemas. I don't know what's in that table either. :)

Thanks for the Feldera shoutout Jim.

For anyone else, if you want to try out Feldera and IVM for feature-engineering (it gives you perfect offline-online parity), you can start here: https://docs.feldera.com/use_cases/fraud_detection/

Start with Postgres and scale later once you have a better idea of your access patterns. You will likely model your graph as entities and recursively walk the graph (most likely through your application).

If the goal is to maintain views over graphs and performance/scale matters, consider Feldera. We see folks use it for its ability to incrementally maintain recursive SQL views (disclaimer: I work there).

I currently run Firefox nightly with cross-site cookies disabled and all the trackers/scripts blocked. I also run uBlock Origin. Any idea if privacy badger is redundant with this set up?

Thanks for the kind words (Feldera co-founder here). I'll pass it on to the design team. :)

Do give us at Feldera a shot -- full IVM for arbitrary SQL + UDFs: https://github.com/feldera/feldera/

Would love to have our team try this out (we have some ridiculous rust builds).

Yeah that's been our biggest issue in this ecosystem (the non-JVM clients). They can't do writes and are often far behind on feature parity with the blessed JVM clients.

Not the one you asked, but here's what I would have told my 17 year old self.

* "Slope beats y-intercept." The best computer scientists and engineers I've ever worked with and/or mentored embodied this principle more than anything else.

* It can be tempting to over-optimize for short-term milestones (e.g., an important admissions exam, the next job or promotion), but there is a significant compounding value to knowledge accumulation and truly learning your craft well. Read and learn as much as you can, all the time, even if it isn't immediately necessary or useful.

It's the first time I'm hearing about the Berlin Declaration. :)

This made my day. Thank you for the kind words! What were you working on?

USENIX and their conferences were the absolute best to publish with. You as a researcher focus on submitting papers and/or being part of the PC. They help organize the whole conference instead of depending on an army of volunteers (you won't see "general chairs" and "local chairs" unlike with ACM). And all papers were open access without even needing a login: you literally just click the PDF from the conference website.

I first went to sqlx thinking it would be like JOOQ for Rust, but that wasn't the case. It's a pretty low-level library and didn't really abstract away the underlying DBs much, not to mention issues with type conversions. We've since just used rust-postgres.

Not sure either. We added recursion to SQL in Feldera and it's Turing-complete: https://www.feldera.com/blog/recursive-sql-queries-in-felder...

Thanks for the kind words. :) We hear you on the dialect differences.

An interesting case of a user dealing with this problem: they use LLMs to mass migrate SparkSQL code over to Feldera (it's often json-related constructs as you also ran into). They then verify that both their original warehouse and Feldera compute the same results for the same inputs to ensure correctness.

That's a cool demo you're building using ddlog! FWIW, we, the ddlog team have moved on to found Feldera (https://github.com/feldera/feldera). You could consider using DBSP directly through Rust.

Was this for a release build or a debug build?

Hard agree. Practically all the bottlenecks we run into with Rust compilation have to do with the LLVM passes. The frontend doesn't even come close. (e.g. https://www.feldera.com/blog/cutting-down-rust-compile-times...)

LLVM optimizations are the overwhelming majority of the compilation bottleneck for us over at Feldera. We blogged about some of the challenges we faced here: https://www.feldera.com/blog/cutting-down-rust-compile-times...

We almost definitely need to build a JIT in the future to avoid this problem.

(lalith here) -- Whatever ryzyk and gz09 said above. :)

Makes sense. We'd appreciate some more eyeballs here for sure. Between HN and a Reddit thread, there are a few hypotheses floating around. I've shared a repro here for anyone interested: https://github.com/feldera/feldera/issues/3882

Now it still seems strange (as in it looks like a performance bug) that most times rust was stuck in just one threat (instead of e.g. 8).

Agreed, seems like there are some rustc performance bugs at play here.

Yes. We found both cold and incremental builds sped up. The incremental builds were the main win -- small changes to the SQL can sometimes complete in seconds for what used to be a full recompilation.