HermitX

Wow, there seems to be a lot of impressive new features!

Great blog, thanks for sharing!

Excellent content! It looks like the Iceberg + StarRocks open lakehouse Architecture is highly effective.

Great article, thank you for sharing! I have a question I’d like to discuss with the author. Spark SQL is a great product and works perfectly for batch processing tasks. However, for handling ad hoc query tasks or more interactive data analysis tasks, Spark SQL might have some performance issues. If you have such workloads, I suggest trying data lake query engines like Trino or StarRocks, which offer faster speeds and a better query experience.

I believe Snowflake not only needs an open-source table format but also an open-source data lake query engine. Currently, Snowflake has too little presence in this area, and its stock price is reaching new lows.

I suggest you try using StarRocks to query the data lake directly. I know many large companies, including Tencent and Pinterest, are doing this. StarRocks has a truly mature vectorized execution engine and a robust distributed architecture. It can provide you with impressive query performance.

“Specifically, online analytics workloads would be migrated to Druid/StarRocks”, I'm very interested in this part. Look forward to knowing more about it.

As long as the United States is the world’s leading superpower, there will always be a second place trying to challenge its position. Previously it was Japan, now it is China, and there will be other countries in the future.

When the times leave you behind, they won't even say goodbye.

Is ClickHouse a suitable engine for analyzing events? Absolutely, as long as you're analyzing a large table, its speed is definitely fast enough. However, you might want to consider the cost of maintaining an OSS ClickHouse cluster, especially when you need to scale up, as the operational costs can be quite high.

If your analysis in Postgres was based on multiple tables and required a lot of JOIN operations, I don't think ClickHouse is a good choice. In such cases, you often need to denormalize multiple data tables into one large table in advance, which means complex ETL and maintenance costs.

For these more common scenarios, I think StarRocks (www.StarRocks.io) is a better choice. It's a Linux Foundation open-source project, with single-table query speeds comparable to ClickHouse (you can check Clickbench), and unmatched multi-table join query speeds, plus it can directly query open data lakes.

As long as there is greed in human nature, people like SBF will continue to emerge.

To the best of my knowledge, Meta has significantly reduced its investment in the Velox project. Apart from Meta, I'm not aware of any other major company that really uses Velox in a production environment. Frankly speaking, Velox may have already missed the window of opportunity for rapid development. If you're looking for a vectorized execution engine, you could consider ClickHouse (www.clickhouse.com) or StarRocks (www.starrocks.io). If your data analysis scenarios require more multi-table join operations, StarRocks is clearly a better choice.

I This is an intriguing article, clearly showing the team's fondness for Postgres. However, Postgres is an OLTP product. I'm curious about what system Figma's data team uses for their data analysis tasks.

Oh my, the Dragon Ball series of comics are wonderful memories from my childhood! I even want my kids to read these comics now! I hope Akira can rest in peace.

Oh my, the Dragon Ball series of comics are wonderful memories from my childhood! I even want my kids to read these comics now! I hope Akira can rest in peace.

Yes, I understand

If this video call gives the interviewee a final opportunity to express their thoughts and has the potential to change the outcome, then I think it still has positive value. However, as mentioned in the article, most people feel very awkward and have nothing to say during this meeting, which is simply torturous. In that case, I think it would be better to replace the video conference with an email, after all, if you don't allow the possibility of changing the outcome, why not make it more comfortable for both parties?

In the movie The Matrix, should those people trapped in the cultivation tanks be considered as having seven days of rest a week?

I've read his book 'Amp it Up,' and I think it's pretty good. However, I would still recommend that all Snowflake users pay attention to your costs. Here is a plan that could save you 80% on costs, worth checking out. https://medium.com/p/deb87aa69bc5

Every time I see a beautiful tattoo, I always feel an itch, eager to try it. But whenever I think about how much more troublesome it would be to remove these tattoos if I changed my mind, I back down. So, I haven't even started to worry about whether these inks could potentially harm my organs :-p

What do you want? You can connect with the StarRocks eng team on the slack channel. https://join.slack.com/t/starrocks/shared_invite/zt-z5zxqr0k...

Here is a fantastic video made by Soumil Shah, using MinIO+Hudi+StarRocks+Superset. It is amazing to have an interactive query experience on a data lake directly! https://www.youtube.com/watch?v=JkKBzrQTKx0

I've already thought of the follow-up to this article, 'Querying a billion rows on a $XX budget.' Let me give you my answer directly: switch from Snowflake to StarRocks. It's an open-source project under the Linux Foundation, with speed that's more than adequate, especially for queries involving multiple tables. If you're interested, you might want to check it out, https://medium.com/starrocks-engineering/how-to-reduce-snowf....

My view on such matters is that it's best to find a solution that has a fallback option, which usually is an open-source software. That is to say, if you choose a cloud service, it's preferable that it is built on some open-source software. This way, if the costs become uncontrollable in the end, you can still fall back to an open-source software. For instance, CelerData has built its cloud service on StarRocks, and it's said that many users have used it to replace Snowflake and Big Query. Of course, you could also opt for Elasticsearch's cloud service, and if problems arise, you can replace it with OpenSearch.

The subtle influence of TikTok on people is quite astonishing. I believe that there should also be antitrust laws in the field of communication. Otherwise, platforms like TikTok can gradually change people's thoughts, making their ideas more uniform. The government should not control people's thoughts, nor should any giant company control people's thoughts.

I'm curious about what everyone considers to be a high frequency of meetings. Two 1-hour team meetings per week, along with a daily 15-minute stand-up meeting. Do you think this frequency is high?

AI will eventually be capable of performing most of the tasks humans can do. My neighbor's child is only 6 years old now. What advice do you think I should give to his parents to develop their child in a way that avoids him growing up to find that AI can do everything better than he can?

I would also like to add one more thing: hesitation. From my own experience, in my programming career, the most time my team wasted was on redoing something from scratch. Why redo it? Because the previous design was not comprehensive enough to meet subsequent requirements. If everyone could spend more time researching and discussing the plan before starting, and get things right in one go, in my view, this is much more effective than rushing into development work.

Typing can only get faster, but handwriting can get better. I myself also like to think while writing, and it indeed helps me. Of course, after writing, I will take a photo and save it to iCloud, or simply retype it into Google Docs.

In early 2017, due to some fortuitous circumstances, I got involved in investing in a cryptocurrency mining pool. Thanks to the crazy surge in cryptocurrency values that year, I made some money and still hold onto some BTC.

Because of this, in 2018, I once considered dedicating all my energy to the blockchain industry. However, after nearly half a year of in-depth research, I felt that it was really hard for the technology to generate practical value. By the way, I am a computer engineer with over ten years of work experience, having worked at some of the world's largest internet companies. So, blockchain technology was not magical to me.

I mean, the designs of BTC and ETH have their clever and interesting aspects, but at that time, I really couldn't see their practical application value. Therefore, I later diverted my time and career elsewhere. But I still worry that I made the wrong decision, so I kept a significant amount of BTC.