andygrove

I've just started exploring adding OpenTelemetry support to the Comet subproject of DataFusion. I'm excited to see the integration with Apache Arrow (Rust) and potentially DataFusion in the future.

Congrats on the launch!

I contributed to the NVIDIA Spark RAPIDS project for ~4 years and for the past year have been contributing to DataFusion Comet, so I have some experience in Spark acceleration and I have some questions!

1. Given the momentum behind the existing OSS Spark accelerators (Spark RAPIDS, Gluten + Velox, DataFusion Comet), have you considered collaborating with and/or extending these projects? All of them are multi-year efforts with dedicated teams. Both Spark RAPIDS and Gluten + Velox are leveraging GPUs already.

2. You mentioned that "We're fully compatible with Spark SQL (and Spark)." and that is very impressive if true. None of the existing accelerators claim this. Spark compatibility is notoriously difficult with Spark accelerators built with non-JVM languages and alternate hardware architectures. You have to deal with different floating-point implementations and regex engines, for example.

Also, Spark has some pretty quirky behavior. Do you match Spark when casting the string "T2" to a timestamp, for example? Spark compatibility has been pretty much the bulk of the work in my experience so far.

Providing acceleration at the same time as guaranteeing the same behavior as Spark is difficult and the existing accelerators provide many configuration options to allow users to choose between performance and compatibility. I'm curious to hear your take on this topic and where your focus is on performance vs compatibility.

No.

I wish I'd known about the search issue.

I do see a new opportunity for Ballista. By leveraging all of the Spark-compatible operators and expressions being built in Comet, it would be able to support a wider range of queries much more quickly.

Ballista already uses protobuf for sending plans to executors and Comet accepts protobuf plans (in a similar, but different format).

Yes, Ballista failed to gain traction. I think that one of the challenges was that it only supported a small subset of Spark, and there was too much work involved to try and get to parity with Spark.

The Comet approach is much more pragmatic because we just add support for more operators and expressions over time and fall back to Spark for anything that is not supported yet.

Original author of DataFusion/Ballista here. Having alamb and others from InfluxData involved has been a huge help in driving the project forward and helping build an active community behind the project. It is genuinely hard to keep up with the momentum these days!

I will update these old pages on my blog and redirect them!

I think it is worth pointing out that this tool does support querying Delta Lake (the author of ROAPI is also a major contributor the native Rust implementation of Delta Lake). Delta Lake certainly supports transactions, so ROAPI can query transactional data, although the writes would not go through ROAPI.

FWIW, my dealer is not adding any markup. Here is the email I received from them explaining the process:

"We finally received an update from Ford regarding the Lightning! This will be a completely unique process from Ford and they will send a detailed guideline in January, however I wanted to let you know in advance that things are in motion.

This process is going to be based on invitation to convert as demand has outweighed production capability by far. Due to demand, not all reservation holders will recieve an invitation to place an order for a 2022 model year. Ford will begin inviting reservation holders to place orders in waves starting in January. Subsequent waves will receive an invitation in two-week intervals until 2022 model year capacity is reached. Invitations are based on reservation timing, and our estimated allocation. You will be directed to an online configurator (Build and Price tool) where you will then be able to spec out your order and then submit your finalized order. At that time, we will provide you with complete pricing including MSRP, taxes, and dealer handling fee. We are not charging any dealer markup above MSRP. Once the 22MY production capacity is met, all remaining reservation holders will be notified that their next ordering opportunity will be for the next model year."

The Apache Spark project is many many years ahead of DataFusion & Ballista with more than a decade of work from more than 1,700 contributors and is going strong.

I don't see DataFusion as a competitor to Spark since it is specifically designed as an embedded library and is optimized for in-memory processing with low overhead.

Ballista is highly influenced by Spark and is capable of running some of the same queries that Spark can support. There is enough functionality to be able to run a subset of the TPC-H benchmarks for example, with reasonable performance at scale. So for users wanting to run those kind of SQL queries, maybe Ballista isn't so far off, but Spark has much more functionality than this and it could potentially take years of effort from a community to try and catch up with Spark. It will be interesting to see what happens for sure.

There is no support for Plasma. There is a TableProvider API for custom file formats and there is built-in support for CSV, Parquet, and JSON.

Ballista started out as a separate project and was donated in April 2021. They currently share a release schedule (but have different versioning) and this was the first release of DataFusion to include the Ballista crate.

My hope is that Ballista and DataFusion become more integrated over time but remain separate, with DataFusion being an embedded / single-process query engine and Ballista providing distributed execution.

The best "Getting Started" documentation right now is that on docs.rs - https://docs.rs/ballista/0.5.0/ballista/

This demonstrates using the Rust client (BallistaContext + DataFrame). There are already Python bindings for DataFrame but not BallistaContext yet.

Documentation for Ballista is severely lacking right now and this will be an area of focus for the next release.

Yes. The Ballista crate (part of the arrow-datafusion repo) provides distributed query execution and the scheduler has a gRPC service. Flight is used internally as well but not directly exposed to users. There is also work in progress to add Python bindings for Ballista (they already exist for DataFusion).

There is experimental support for distributed query execution with spill-to-disk between stages to support larger than memory datasets. This is implemented in the Ballista crate, which extends DataFusion.

https://github.com/apache/arrow-datafusion/tree/master/balli...

That doesn't surprise me at all. I have politely requested that the project considers renaming to avoid this.

https://github.com/datafuselabs/datafuse/issues/654

I thought this was about me until I got to the word Rich in the title.

Ah, yes, that makes sense. I can see how this could have been misread.

There is also a more recent blog post which perhaps led to the project being posted here (I am guessing).

https://andygrove.io/2021/01/ballista-2021/

The project uses stable Rust. Which version are you trying to compile with?

Apache Arrow is a specification for in-memory columnar data, IPC format + Flight protocol, with implementations in a number of languages. Some of the implementations contain code to perform computations on the in-memory data. Some of the implementations contain some form of query engine. All of these are single process / libraries, rather than distributed systems.

Apache Spark is a distributed compute platform, which does have some support for Arrow for interop purposes.

One of the things that led me to get involved in Arrow originally was to explore the idea of building something like Apache Spark based on Arrow (and Rust) and my latest prototype of that concept is in the Ballista project [1].

[1] https://github.com/ballista-compute/ballista

There are contributors and committers in the Arrow project working to resolve this. We recently removed specialization from the core Arrow crate and we plan on doing the same for the Parquet crate.

Arrow is a memory format and optimized for efficient vectorized processing in memory. Although it is possible to persist the Arrow format to disk, it is more common to use Parquet.

There is also https://github.com/ballista-compute/ballista (shameless plug for my own project)

Thrift is a serialization format. Arrow is a memory format.

You should be able to find this info in the specification: https://arrow.apache.org/docs/format/Columnar.html

The Arrow project contains implementations in multiple languages. Some of these languages contain code that can evaluate expressions against Arrow data, or even execute full queries. The C++ and Rust implementations contain query capabilities, and the Java implementation contains the Gandiva library that can delegate to C++ via JNI to evalulate expressions, for example.

I think it makes sense for software that is intended to run in Docker and frameworks like Kubernetes that use Docker.

Thanks! That really does seem to be the issue and I wouldn't have known about this, had I not asked. I will try this out and will update the blog post in ~8 hours time.

I get that a lot!