HN user
This is very disingenuous, I've been deploying local models to enterprise across a variety of use cases and the optimisation overhead and prompt engineering required to get good performance is huge. Let alone comparative perf to frontier models.
This is most misrepresented article on two fronts
1. tested column pruning and the dataset you access would have been 2 columns + metadata for the parquet files so probably fit in memory even without streaming.
2. Most of the processing time would be IO bound on S3 and the access patterns/simultaneous connection limits etc. would have more of an impact than any processing code.
Love that you went through the pain of trying the different systems but I'd like to see an actual larger than memory query.