tbragin

Yes, we are working on it! :) Taking some of the learnings from current experimental JSON Object datatype, we are now working on what will become the production-ready implementation. Details here: https://github.com/ClickHouse/ClickHouse/issues/54864

Variant datatype is already available as experimental in 24.1, Dynamic datatype is WIP (PR almost ready), and JSON datatype is next up. Check out the latest comment on that issue with how the Dynamic datatype will work: https://github.com/ClickHouse/ClickHouse/issues/54864#issuec...

One exists https://blog.coiled.io/blog/1trc.html

Cloud network egress costs is one of the reasons why increasingly infrastructure vendors likely to incur high data transfer costs (database / data warehouse services, message buses, machine learning, etc..) are moving to introduce BYOC or "Bring Your Own Cloud" deployment model (Data Plane runs in the customer VPC - original Databricks deployment model), instead of only pure SaaS (Data Plane runs in the vendor VPC - e.g. Snowflake) in their cloud offerings.

Databricks now has fully hosted too, but Snowflake still did not go the other way to introduce BYOC. Newer entrants, however - RedPanda, AnyScale, ClickHouse... - are going the way of providing both BYOC and pure SaaS options in their cloud. There are many other reasons to prefer BYOC, legal, security, data privacy, but network egress costs is one of them.

Disclaimer: I work at ClickHouse

Thank you! Glad it helped you.

ClickHouse is in particular well-optimized to take on real-time analytical workloads from BigQuery - both functionally and from a cost perspective.

For anyone wanting to try it yourself, there guide to follow when syncing or migrating workloads from BigQuery to ClickHouse, with some benchmarks and back-of-the-envelope cost comparisons: https://clickhouse.com/blog/clickhouse-bigquery-migrating-da...

Indeed, ClickHouse results were run on an older instance type of the same family and size (c5.4xlarge for ClickHouse and c6a.4xlarge for Timescale), so if anything ClickHouse results are at a slight disadvantage.

This is an open source benchmark - we'd love contributions from Timescale enthusiasts if we missed something: https://github.com/ClickHouse/ClickBench/

Compression helped, but not enough in our experiments https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQi...

However if you really want to optimize data currently residing in Postgres for analytical workloads, as the original comment suggests - consider moving to a dedicated OLAP DB like ClickHouse.

See results from Gitlab benchmarking ClickHouse vs TimescaleDB: https://gitlab.com/gitlab-org/incubation-engineering/apm/apm...

Key findings:

* ClickHouse has a much smaller data volume footprint in all cases by almost a factor of 10.

* There are very few ClickHouse queries that have >1s latency at q95. TimescaleDB has multiple >1s latencies, including a few in the range of 15-25s.

Disclaimer: I work at ClickHouse

Interesting project! Thank you for open sourcing and sharing. Agree that local and embedded analytics are an increasing trend, I see it too.

A couple of questions:

* I’m curious what the difficulties were in the implementation. I suspect it is quite a challenge to implement this support in the current SQLite architecture, and would curious to know which parts were tricky and any design trade-off you were faced with.

* Aside from ease-of-use (install extension, no need for a separate analytical database system), I wonder if there are additional benefits users can anticipate resulting from a single system architecture vs running an embedded OLAP store like DuckDB or clickhouse-local / chdb side-by-side with SQLite? Do you anticipate performance or resource efficiency gains, for instance?

* I am also curious, what the main difficulty with bringing in a separate analytical database is, assuming it natively integrates with SQLite. I may be biased, but I doubt anything can approach the performance of native column-oriented systems, so I'm curious what the tipping point might be for using this extension vs using an embedded OLAP store in practice.

Btw, would love for you or someone in the community to benchmark Stanchion in ClickBench and submit results! (https://github.com/ClickHouse/ClickBench/)

Disclaimer: I work on ClickHouse.

Cool article! I like the practical advice on managing schema changes. I hope you don't mind me sharing on other platforms. Thank you!

Disclaimer: I work at ClickHouse.

I'm a product manager, and I'm glad that many commenters in this thread agree that we are not all useless :) Of course, just like with any role, there are those that are good and bad at it, but I am glad most of you worked with good PMs and know the value such a person can bring.

I also saw astute comments from folks that recognize that if the product manager role does not exist as a distinct role, someone else in the organization has to take on the work of determining what to build and when, based on market and customers needs - usually engineering or product marketing. There is nothing wrong with that, btw, just different way to slice the same pie of responsibilities. Finally, several folks rightfully pointed out that a product manager (person accountable for building the right product for the market) is very different from a project or program manager (a person coordinating internal activities).

The only thing I'd add here is that product management roles can also be very different, depending on the stage of the company (early stage vs late stage), the type of product it builds (e.g. B2B vs B2C), and the industry vertical. Looking at the author's LI profile, he seems to work for larger companies, in the financial sector, in the UK. Perhaps in that world (which I perceive to be a fairly rigid, regulated industry), the role of the product manager is quite different from what I have personally experienced working as a PM in the Bay Area for early and mid stage startups.

Philosophically, I think there is a lot of room for open-source software that also has an amazing UI and developer experience.

+1 to that! Welcome to the ClickHouse community!

Disclaimer: I work at ClickHouse.

Thank you! Looks really interesting!

I personally agree that real-time OLAP databases have potential to better serve workloads currently in Postgres or cloud data warehouses that need real-time ingest and analytical queries. And simplifying developer experience on top of that, so you don't have to learn about all the details of a powerful database, really speeds up developer velocity.

I'm curious, how you see your project differs from GraphJSON (https://www.graphjson.com/) and Tinybird (https://www.tinybird.co/)?

Congratulations again on the launch!

Surprised to hear it. I still get quite a bit of engagement on X, certainly have not seen a decline in the past year. Also, a few folks I follow on X, even if they claimed to "have moved to Fediverse", have all quietly come back and started to post on X again.

I joined Mastodon myself in the past year, and at least so far I have not been able to see much engagement there, though I am still trying to figure it out. The biggest issue I see is lack of cross-server search, aside from by tag, but also just not enough post activity. Maybe I am still not following the right people.

Zoom has had various touch up features for a long time (skin smoothing in particular).

Not a new article, but it seems that the rollout has been going on in stages. The switch was forced onto our org a week or so ago, and people are really struggling. Screen real estate is worse, navigation is worse, notification callouts are not clear... generally, several steps back in usability.

It's even more ridiculous than that. I don't need my OS enhancing my own gesticulations. I am enough all by myself (thank you very much).

For me, apparently I like to stick out my thumb and then index finger, when I say "first (second, third)", which triggers thumbs up first and baloons second. Really? Totally unhelpful and awkward.

What I've seen as effective as a new manager is to think about scoping whole and independent areas of responsibility for each individual on my team. This requires upfront thinking and planning, but pays in the long term as individuals come into these areas as end-to-end owners. In many cases, overtime, they became deeper experts than me in their respective areas, such that 1 + 1 > 2 in the long run.

As far as specific task delegation, if you are looking at your task list, I'd recommend a modification of "Eisenhower Matrix", that looks like this:

- Important / One-time Tasks - Keep for yourself.

- Important / Frequently Repeating Tasks - Delegate with high priority.

- Not important / Frequent tasks - Delegate with low priority.

- Not important / One time tasks - Delete.

A couple of years ago, I started having headaches and debilitating tinnitus, and it went on for months. I was tested by audiologist - my hearing was normal. It turned out that (unbeknown to me) I became extremely anemic due to another health issue. Once I got diagnosed, and anemia was addressed, headaches and tinnitus went away.

Amen, especially for reference documentation, such as API or functions / SQL commands (e.g. for a database product). Those docs should be checked in with the code, and PR should not get merged if the docs are not ready.

They forgot the most common model - "add missing docs after the fact" :)

AI-assisted sales is an interesting topic. I personally get a bit uncomfortable when I get on a call with a vendor and they have this recording software there. Previously it was just for note-taking, but now it can be much more powerful (e.g. sentiment analysis based on facial expressions and voice). I personally think this type of technology will have applications outside of sales.

Coincidentally, as someone who worked on this blog, I was surprised (and pleased!) to see that we are not the only ones who felt the need to build a Zookeeper alternative.

Looks like folks at StreamNative did as well, with their Oxia project: https://github.com/streamnative/oxia. They were just talking about this yesterday at Confluent Current ("Introducing Oxia: A Scalable Zookeeper Alternative" was the title of their talk). https://streamnative.io/blog/introducing-oxia-scalable-metad...

Seems to be a trend :)

Disclaimer: I worked on this blog with the team at ClickHouse.

I like your suggestions!

Some of the benefits we summarized in this summary page https://clickhouse.com/clickhouse/keeper include ease of setup and operation, no overflow issues, better compression, faster recovery, (dramatically) less memory used, etc..

There was actually a reason why C++ was important for us at ClickHouse, and it's because C++ is our main code base and managing a Java project as part of it was not natural, but you right - for standalone use of this alternative, that doesn't matter.

The best way I found to arrive at an MVP it is to set a very aggressive time-based deadline, start executing, and aim to release something early to only a few users. Then rinse and repeat with another intermediate milestone. It then becomes necessary to throw out stuff that does not fit, and also even a very early-stage intermediate milestone is the best way to get feedback (esp on onboarding, UX and pricing). Trying to come up with MVP on paper - I've found that whatever you define still comes out bloated.

I don't understand why "big tech" is forcing tech workers to go back. Have they specified the reasons? Perhaps at bigger companies, it is easier to "hide" and not be productive, while working from home? Or is it because, if for some team members remote work is not productive (e.g. due to the surrounding environment), then it is better for the whole team to be co-located? Honest question, because I really can't say. For the past 8+ years, I've now worked at two fully-remote companies and found it to be a perfect fit for me personally.

I bet Mastodon will see a huge inflow of users once this goes into effect. Could be a boon for them as well.

From the article: The provision would direct the Commerce Department to write rules forcing cloud companies like Microsoft, Google, and Amazon to disclose when a customer purchases computing resources beyond a certain threshold. The order hasn’t been finalized and specifics of it could still change.

The big addition in this edition is a rebuilt search facility. That matters, because it's something users expect to just work. Mastodon 4.2 has therefore made it possible to search for users using by words in their bio and their names. "The most exciting news is that for the first time, you can now search for posts," Rochko wrote.

This is helpful, but not enough. As a Mastodon user, I'd love full text search across post contents to discover something of interest. Right now, it is impossible and really hampers my ability to discover useful content, unless it is appropriately tagged.

In earlier versions, you could search for hashtags, usernames, and display names but could not search the contents of profiles or posts (except by pasting in the URL, which we’ll get to). Now, it works more like the search box on basically every other website.

This is great, lack of global FT search is the biggest obstacle for me discovering relevant content on Mastodon. Hope Hackyderm server is upgraded soon!

As someone responsible for pricing decisions at my company, it has been very interesting reading through this thread and processing the reactions.

First, I will say that - I feel for users that are reading this blog and trying to process the changes. I completely understand the reaction of those to whom at a glance it may appear that compute bills would go up by 2x as a result of this pricing change. I am sure that is shocking, especially to companies on short budgets. The change is effective on July 5 2023, which is really not that far away. If you have a mature data warehousing deployment, it could take may way longer than 3 months to plan and execute a migration.

However, speaking from experience, it is challenging to come up up with a perfect usage-based pricing model for infrastructure heavy product like BigQuery that suits everyone’s needs, and even harder to communicate it well.

Some of the reasons are…

* There is tension between keeping the pricing simple and yet have it be fair to every workload. If you really want a fair pricing model, you will need to add more and more pricing dimensions to make users pay more granularly for each resource that makes up the infrastructure cost. But a pricing model with 10-20 dimensions is really hard to understand, even if the final price tag would have been lower for each workload, than a more “simple looking” model.

* Deciding on pricing changes of an existing offering is also not easy. I am going to guess the reason for this pricing change wasn’t because Google decided to “nickle and dime” their users (could be of course, but I would hope that is not how most companies think). I would guess it was the autoscaling feature that was the catalyst for the change. Why did the compute price point have to change? Probably because current users are over-provisioning by quite a bit and the current price point is lower to reflect that, so the end price tag is still within reasonable budget and market expectations. If the autoscaling feature is going to work as perfectly as their charts in the blog show, the price point must come up to make up for the margin loss. As some point out, in that scenario, bursty compute workloads that are reserving too much capacity may actually come out ahead.

* Once you do one pricing change, you may as well bundle in other important pricing changes that have “piled up” on the product team backlog. I am guessing that is how the storage change came up - users were confused about paying for uncompressed storage, while some alternatives charged on compressed, and so they made the change. Changing pricing from one dimension to another is tricky. As some pointed out, storage price point went up, but the dimension moved from uncompressed to compressed storage, so the effective price change for most users may actually be lowered, if the compression rate make up for the price point increase.

Anyway, very interesting, I am really curious to see how this change plays out in the market in the coming months.