rxin

This is coming to Serverless SQL warehouses soon!

(OP / Databricks cofounder here)

I'm not sure what happened but I just sent an email out internally to ask people not to do this. The team might have gotten overly excited by this because they were all part of the creation of the dataset and the model.

(Databricks cofounder here)

All datasets are biased, including this specific one. However, we believe it's still very valuable to open source, for a few reasons:

- This dataset is primarily used to train instruction reasoning, not for knowledge. (Keep in mind Dolly and any of the well known models have not been specifically trained for knowledge. They are all just demonstrating instruction reasoning.) The lack of a true open source (available for both research and commercial use) instruction dataset is the primary blocker for making these LLMs available for commercial use.

- We hope this will lead to not just open source innovations in models, but also future training datasets.

- Given the international population of our employee based, it's likely more diverse than datasets created by a small number of human labelers. And it is easier to identify, discuss, and debate dataset bias in the open.

Databricks founder here. Doesn’t seem like DM work on HN anymore. Do you mind shooting me an email? Would love to follow up and understand the issues.

rxin@databricks.com

You should try Databricks, especially the new Photon engine powering Spark. In general more performant than Snowflake in SQL and a lot more flexible. (There are some cases in which Databricks would be slower but the perf is improving rapidly.)

Isn’t that what the official TPC does?

Totally. Simplicity is critical. That’s why we built Databricks SQL not based on Spark.

As a matter of fact, we took the extreme approach of not allowing customers (or ourselves) to set any of the known knobs. We want to force ourselves to build the best the system to run well out of the box and yet still beats data warehouses in price perf. The official result involved no tuning. It was partitioned by date, loaded data in, provisioned a Databricks SQL endpoint and that’s it. No additional knobs or settings. (As a matter of fact, Snowflakes own sample TPC-DS dataset has more tuning than the ones we did. They clustered by multiple columns specifically to optimize for the exact set of queries.)

Geometric mean is commonly used in benchmarks when the workloads consists of queries that have large (often orders of magnitude) differences in runtime.

Consider 4 queries. Two run for 1sec, and the other two 1000sec. If we look at arithmetic mean, then we are really only taking into account the large queries. But improving geometric mean would require improving all queries.

Note that I'm on the opposite side (Databricks cofounder here), so when I say that Snowflake didn't make a mistake here, you should trust me :)

Ah ok. Wasn't clear. I think some repro scripts will be available soon.

Check my reply, Leo.

Exactly. Not sure about Netflix special, but there are experts that have dedicated their professional careers to creating fair benchmarks. Snowflake should just participate in the official TPC benchmark.

Disclaimer: Databricks cofounder who authored the original blog post.

There's an official TPC process to audit and review the benchmark process. This debate can be easiest settled by everybody participating in the official benchmark, like we (Databricks) did.

The official review process is significantly more complicated than just offering a static dataset that's been highly optimized for answering the exact set of queries. It includes data loading, data maintenance (insert and delete data), sequential query test, and concurrent query test.

You can see the description of the official process in this 141 page document: http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3....

Consider the following analogy: Professional athletes compete in the Olympics, and there are official judges and a lot of stringent rules and checks to ensure fairness. That's the real arena. That's what we (Databricks) have done with the official TPC-DS world record. For example, in data warehouse systems, data loading, ordering and updates can affect performance substantially, so it’s most useful to compare both systems on the official benchmark.

But what’s really interesting to me is that even the Snowflake self-reported numbers ($267) are still more expensive than the Databricks’ numbers ($143 on spot, and $242 on demand). This is despite Databricks cost being calculated on our enterprise tier, while Snowflake used their cheapest tier without any enterprise features (e.g. disaster recovery).

Edit: added link to audit process doc

Co-author of the paper here.

I don't think your argument holds here at all. It's a common misconception to think high performance would require tight coupling of storage and query processing.

"Think columnar storage, high compression, vectorized query, materialized views, etc." All of those are possible in Lakehouse, and all but one (materialized views) are fully implemented on Databricks. And the remaining one isn't far away either (materialized views is really just incremental query processing + view selection, and neither problem has much to do with storage).

We have taken a very different approach with the Unity Catalog. It is designed opposite to the "cluster-centric" access control model, and will be user and role centric.

Disclosure: I'm a Databricks cofounder and have contributed to the unity catalog.

You should take a look at Databricks, in all three dimensions.

Disclaimer: I designed the system that won the 2014 GraySort based on Apache Spark, and the same system was extended by a different team to win the 2016 record in cloud sort.

"Merge sort is the only algorithm that can effectively be parallelized" is not a good explanation for the lack of quick sort usage in the sort benchmark. In a parallel sorting system, you can decouple the merging step from the sorting of partial runs. That is, you'd still want the fastest algorithm for sorting partial runs, and then implement a fast merge operation.

So what's the reason? I can only offer speculations based on my own experience.

Contrary to what the name suggests, the sort benchmark is actually not designed primarily to test the in-memory sorting part of the system. It is designed to stress the entire system. When you are required to sort 100TB, or even PBs+ of data, your software system's ability to optimize for I/O (including reading from/writing to disks, sending data across the network, pipelining) is far more important than the actual sorting itself. Of course, you shouldn't have a very slow sort algorithm either.

The reason I used TimSort was because it was relatively fast, already implemented in Apache Spark, I was really familiar with it, so I could tweak it to do what I needed. The vast majority of the time I spent in the sorting part was to tune memory layout (to make sure there's 0 JVM garbage collection, and good cache locality), and to remove any virtual function calls. Once tuned, the sorting time was a very small fraction of the overall time. Using TimSort (which can merge partially sorted runs very quickly) also allowed me to implement just one highly optimized sorting algorithm, rather than having to implement one algorithm for sorting the partial runs, and another algorithm for merging them.

The sort benchmark also defines the key space (10 byte key) as fixed length. My guess is that the fastest in-memory sorting algorithm for this case would be a highly optimized radix sort, which the 2016 Gray sort record used.

What's the reason you need to abandon scikit-learn? You can run scikit-learn on Databricks, and many of our customers do.

Disclaimer: Databricks cofounder.

We posted more details just now. Will share more next week too.

More coming next week. We believe we got to the bottom of this last night.

This is definitely not the case for China.

Apache Spark doesn't use Calcite.

This (天网, or "sky net") most likely comes from proverb "天网恢恢疏而不漏" (tian wang hui hui shu er bu lou), meaning "Justice has long arms."

This saying was from Lao Zi, going back thousands of years.

What's bloated about it?

From the UX point of view, it's pretty lightweight for most common operations. Goes into the app, finds the contact, sends message, done. There is virtually no difference between WeChat, WhatsApp, and Apple's own Messages. For many other operations that are less common, WeChat is easier to use and manage (e.g. sending emojis, voice memo, photos).

From an app size point of view, I just checked on my iPhone. WeChat takes up 157.3MB, whereas WhatsApp takes up 105.9MB. The difference seems minor (just a few photos).

I live in the Bay Area and I use both WhatsApp and WeChat and I would say WeChat is a significantly better product, with more features and better video/audio quality. When I visit China, WeChat is another order of magnitude more indispensable due to its great integration with all the other services (eg, mobile payment penetration through WeChat effectively 100%).

WeChat is pretty big in Hong Kong now. I was on Hong Kong MTR last week and half of the people standing around me used WeChat within 10 mins.

Spark actually has probably the best standard SQL support among all open source big data frameworks. It can run all of TPC-DS queries without modifications.

What you said about Spark was not true. One can totally interactively query a Kafka stream using Spark, in SQL.

DJI AFAIK moved out of Hong Kong due to the lack of funding. Hong Kong never developed any serious engineering culture & talent base and it is very difficult to start a high tech venture in Hong Kong, from a funding point of view. From a talent point of view, Shenzhen's engineering easily outnumbers and out-qualitys Hong Kong by an order of magnitude.

Take a look at the customers featured at reinvent. Computing as "utility" is the future, and the future is arriving fast.

The one-off cost is very high, and there is higher ongoing maintenance cost as well. Most organizations are moving to the cloud, because it in general makes more economic sense.