sdairs

clickhouse is not bad at joins these days. But its true maybe clickbench needs some joins & larger data sets these days

Yes https://clickhouse.com/blog/clickhouse-full-text-search-obje...

It's using Vercel's skills CLI, so its not custom to these particular skills. You can just place the files yourself, too.

https://clickhouse.com/blog?category=engineering

Least offensive Bending Spoons acquisition to date. I don't really mind if they kill this one?

Everything from the astral team has been superb, I don't want to use Python without ruff & uv. Yet to try "ty", anyone used it?

Thanks for clarifying; I'm not trying to take anything away from you, I work in the OLAP space too so it's always good to see people pushing it forwards. It would be interesting to see a comparison of totally cold Vs hot caches.

Are you looking at distributed queries directly over S3? We did this in ClickHouse and can do instant virtual sharding over large data sets S3. We call it parallel replicas https://clickhouse.com/blog/clickhouse-parallel-replicas

Chrome does suck ass, hence why I use Firefox and said I don't want to use Chrome, lol. But I want Firefox to be a good browser in its own right, not just "not Chrome". Firefox is just about over the "acceptable" line for me, as a power user for 15+ years (and under that line for most normal users) so I continue to use it, but they're neglecting it in favor of these useless AI features.

ClickHouse is the market leader in real-time analytics so it's an interesting take that you don't think it qualifies.

Yeah I'm not at all interested in Brave, that's a dumpster fire of it's own. And that still gives control to Google by owning the defacto implementation of browsing the internet. There needs to be an actual alternative to Chrome.

I've used Firefox for 15 years and I really don't want to use Chrome. Can Mozilla just, like, make a good browser?

Pretty big caveat; 5 seconds AFTER all data has been loaded into memory - over 2 minutes if you also factor reading the files from S3 and loading memory. So to get this performance you will need to run hot: 4000 CPUs and and 30TB of memory going 24/7.

Yeah ClickHouse is written in C++, though rust is also possible https://clickhouse.com/blog/rust - would be cool to see zig!

Yeah metrics, logs, traces and can do session replay

ClickHouse doesn't use ZooKeeper anymore, and if you're just using a single server you don't need to worry about coordination :)

ClickStack/HyperDX is a polished OOTB stack that has an all in one image you can deploy to get started, so you don't need to worry about the ClickHouse side until you need to really scale (which is where ClickHouse really shines).

Sounds like you should take a look at ClickStack (HyperDX) to me

Yeah "Optimizing ClickHouse for Intel's ultra-high core count processors" is the original, unchanged article title, it's just been submitted slightly differently on HN (surprised mods haven't changed it here actually)

We do! (and we're hiring!)

how long until I have 288 cores under my desk I wonder?

ClickHouse works in-process and on the CLI just like DuckDB, but also scales to hundreds of nodes - so it's really not limited to just large scale. Handling those smaller cases with a great experience is still a big focus for us

Yep that's pretty much the case here!

yeah its legacy tech at this point

this has been my experience too

tbh I see just as much notebook-hell outside of dbx, it's certainly not contained to just them. There's some folks doing good SDLC with Spark jobs in java/scala, but I've never found it to be overly common, I see "dump it on the shared drive" equally as much lol. IME data has always been a bit behind in this area

personally you couldn't pay me to run Spark myself these days (and I used to work for the biggest Hadoop vendor in the mid 2010s doing a lot of Spark!)

In your experience, how are folks doing (1)? The post is talking about a framework to add e.g. type safety, schema-as-code, etc. over assets in data infra in a familiar way as to what is common with Postgres; I'm not familiar with much else out there for that?

I think even before dbt turned DE into "just write sql & yaml", there was an appreciable difference in DE vs SE. There was defo some DEs writing a lot of java/scala if they were in Spark heavy co's, but my experience is that DEs were doing a lot more platform engineering (similar to what you suggest), SQL and point-and-click (just because that was the nature of the tooling). I wasn't really seeing many DEs spending a lot of time in an IDE.

But I think whats interesting from the post is looking at SEs adopting data infra into their workflow, as opposed to DEs writing more software.

This is really nice

Yeah that's right. We create "patch parts" that contain the updates, and merge them along with the usual merge process.

To be honest, I just didn't think of it. But thanks for the suggestion, we'll give it a go!

I think we do allude this in the post, if my understanding is correct. AFAIK both ClickHouse and Postgres are using MVCC, so when an UPDATE is run, a new version of the data is written and the original data is not changed. In ClickHouse, we write a patch part, it's merged automatically by a background process or by FINAL. In Postgres, you write a new tuple and mark the old one as dead, which will be cleared up by a VACUUM.

I'd wager the overhead in Postgres is probably lighter (though it's also light in ClickHouse), so you're right, this would be an interesting test. We actually had something like that planned, to run concurrent UPDATEs and SELECTs at different volumes for a period of time, to see how they each cope. Will definitely do it!