ople

"Point at any ClickHouse table – no schema migrations/no OTEL requirement"

I haven't tried it yet but I definitely will because of this. Pretty much every other CH observability product is opinionated about their schemas, which may not be optimal for all the use cases and do not take into account that folks may already have a massive CH datastore. This creates a big hurdle for adoption.

This is a cool and underrated new feature: It basically enables simple joins of large and dynamically updated tables efficiently. This is something that has been a bit of an Achilles heel of ClickHouse, at least for me so it's exciting to see.

That said, it's not a solution for all kinds of joins, but probably covers the most common use case for most people.

Why not just use SQL? With LLMs evolving to do sophisticated text-to-SQL, the case for a custom language for the sake of simplicity is diminishing.

I think that expressiveness, performance and level of fluency by base language models (i.e. the amount of examples in training set) are the key differentiators for query languages in the future. SQL ticks all those boxes.

This is one of the things I appreciate about ClickHouse: Contributors willing and able to dive in to performance tune even low-level core stuff like this and getting merged quickly and efficiently, even if they are not affiliated with ClickHouse Inc.

Lovely write-up as well!

True. My bet is that arbitrary text queries converted to SQL via LLM will become an increasingly popular alternative. Not sure if it will completely replace the custom query languages though..

Recently there seems to be an bunch of SPL (Splunk) -like query languages popping up: PQL, PRQL, Grafana Explore Logs syntax, Kusto.. Probably others as well. Does yet another similar but slightly different language make sense? Why not leverage an existing one?

Very interesting observations! Merge performance tuning seems often overlooked even though it's a key aspect of sustained ClickHouse performance.

I also like that the blog is quite compact and gets the points across without getting too much into the weeds.

One thing I've noticed also that bloom filter index types can be quite costly to merge. In many cases that's acceptable though due to the massive benefit they provide for text queries. One just has to be mindful of the overhead when adding them.

I have had the same experience. I was constantly bumping into unexpected limitations. Moving to CH felt like the opposite, with many more ”Wow, I didn’t expect this to be possible but it is” experiences.

There is a place for BQ but it is good to set expectations correctly and also look at the constraints. They are sometimes not obvious. The docs do helpfully outline the limitations, for example:

- Materialized views: https://cloud.google.com/bigquery/docs/materialized-views-in... - Indexes: https://cloud.google.com/bigquery/docs/search-intro#limitati...

I have pretty much exactly the same experience.

However, I do feel that they are trying to really do the right thing with the new 3.0 architecture, addressing the deficiencies (most importantly performance and full-fledged SQL) while keeping the stuff that works (InfluxQL for simple and legacy queries). Also leveraging open-source projects and contributing to their upstream is a plus. Thus I’m hoping for them to succeed delivering on that promise.

Unsurprisingly IBM pioneered this with "Capacity on Demand" https://www.ibm.com/docs/en/power9?topic=environment-capacit...

I'm guessing it has had some popularity among customers as it has been around at least for a decade.

Hehe.. In retrospect the whole team was in fairly good spirit although the situation was stressful. A lot of this was due to the top management giving the time and space for the specialists to do their thing and the very understanding response from the customers once we explained the situation.

The reading of the metadata required quite a lot of random acces. We were fairly sure that if a high-end array and controller with fast disks is struggling with it, then a traditional clustered solution with slower node local disks would not fare much better. Thus we tried to find the solution which yields the highest possible IOPS.

We copied the raw image file of the corrupted metadata filesystem (MDT in Lustre lingo) to the ramdisk.

Then we mounted it via loopback and copied the files to tarballs. The bit that was really slow on the spinning disk was reading the millions of files from the metadata FS.

The basic process of the file-level backup is documented here: https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfu...

The metadata target (MDT) in the MDS is actually "ldiskfs" which is an enhanced version of ext4. One possibility may be to use ZFS in the future as the support in Lustre seems to be quite stable now.

It seems pretty impossible to find out the exact root cause in retrospect as the system was running for a long time without apparent issue. Any ideas are welcome though.

It was filesystem-level corruption in Lustre. The underlying disk arrays and other hardware have comprehensive redundancy.

Author here. We have had an automated deletion policy on our previous filesystems but opted out this time: There are users that have temporary files that they want to persist on the /wrk and we have plenty of capacity. We definitely learned our lesson, though. :)

Author here: We considered that but as the access pattern was likely pretty much random, the performance would have been terrible. Due to the break we had nearly a 1000 clustered servers sitting idle so it was reasonably quick to do the ramdisk trick.