aduffy

That is fair.

FWIW I think if you are just doing pure analytics and nothing else, Parquet will probably continue to do the job for you just fine, and you don't need to touch your workloads at all.

These new formats I think will find a niche where people aren't just running Spark jobs, but doing lots of systems building over large tables. If you're building a PB-scale data warehouse, you care a lot about the file format b/c it is a big factor in your performance curve, and you're willing to ship new experimental codecs in response to new datatypes you want to support that the system wasn't originally designed for, or you want to use a newly invented compressor.

Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up.

This is really more of an expectation that has been put on file formats by the query engines. Spark/Datafusion/DuckDB wouldn't really know what to do with a multi-table file.

Parquet is unfortunately very good just by virtue of being first, and so widely supported

IMO that is not how technology works. It is great that Parquet is so good at a lot of things, but that does not mean just because it came first that it deserves to be the only analytic file format forever.

Its main result seems to be improved random access which, although certainly welcome, is not the point of columnar storage, as columnar storage was invented to exchange random access for something else: fast analytics

Fast analytics, as well as newer ML-shaped workloads, are inherently mix of batch scans and random access.

Some of the authors of F3 previously authored another paper that goes into the details of the shortcomings of Parquet

https://www.vldb.org/pvldb/vol17/p148-zeng.pdf

All of the newer formats that popped up recently (Vortex, Lance, F3 now) have been working on solving the problems outlined in that paper.

Lance has some interesting ideas, Vortex focuses on extensibility and performance by replacing all of Parquet's black-box encoders with fully transparent encodings. This solves the tradeoff between bulk and element decoding, allowing you to have efficient full scans and really fast random access.

E.g. Langchain recently rebuilt a system that used to be all Parquet files to use Vortex and saw a massive speedup, which they talk about more here: https://www.langchain.com/blog/introducing-smithdb

Disclaimer: I work on Vortex, so a lot of these questions about "what is the point of building a new format" are things that I have grappled with myself.

“I never got my driver's license, and I rely on Waymo to commute to an office every day," said Sarah Paige Roland, a Waymo rider in Phoenix. "I get privacy, time back, a safe ride, and I'm not obligated to talk to someone that I don't want to talk to."

I recognize that this is a luxury product but I kind of laughed out loud at this testimonial. The amount of privilege you need to have to grow up and live in *Arizona* without ever learning how to drive is insane.

Move fast and meltdown

I think I'm missing the excitement. This is an artist's rendering of a supposed massive orb in the sky? I am more impressed by the actual UAV footage that has been released previously.

On top of this whole thing just being ridiculous, $50mm is also just not a very impressive amount of money to build out an AI data center.

In the case of the "Gulf of America" thing there was a clear and open statement by the executive that they wanted the maps changed (note that even in the US the name is still legally "Gulf of Mexico"). Apple and Google both decided to acquiesce to curry favor.

TMK there is no current government order to eliminate large swaths of Lebanon from maps. So the fact that Apple is doing this (seemingly on its own, despite all other mapping services reflecting the original place names) is the thing I'm explicitly calling out as being weird.

Sure, but I'm in the US, which is not a party to the conflict between Israel and Lebanon. To my knowledge the US continues to recognize Lebanon as a sovereign country.

OSM is a foundational data layer for GIS. If you're building a mapping service, you're almost certainly using OSM augmented by satellite imagery and other sources to find population zones that OSM has not found yet.

If you look at the Apple Maps satellite layer, you see thousands of structures spread across the area.

It is a reasonable assumption that these population centers were labeled and Apple (or one of its data partners) has withdrawn the labels.

It is straightforward to visit any other online mapping service and see many villages labeled there.

Bing: https://www.bing.com/maps?cp=33.185932%7E35.321974&lvl=11.9&...

Google: https://www.google.com/maps/@33.1649913,35.2506666,11.55z

OSM: https://www.openstreetbrowser.org/#map=11/33.1554/35.2890

I don't understand what using Scala has to do with anything here.

This reminds me a bit of the Gulf of America fiasco from last year where if you changed your location to outside the US it would go back to showing Gulf of Mexico.

I'm not sure why they would do this for US users unless the US government requested it.

I'm not sure how much of that converts to revenue. If it's free plan users, that's just cost. You can say what you want about "creating a training data moat" but that doesn't seem like it's prevented the other labs from putting out excellent models.

There's something a little off about the projection logic when you drop into the Leaflet view, you'll notice that when you pan around after zooming the planes shift their location.

Very cool demo though!

It's even funnier actually. Yes, Palantir does have free Zyn vending machines in every office, but the Zyn is only for visiting customers. Employees are explicitly prohibited from using the machines.

Just vice signaling all the way down.

They need to refund it *with interest*, according to filings cited in the article.

Yea, the group commit is the real insight here.

I read this blog post and to help wrap my head around it I put together a simple TCP-based KV store with group commit, helped make it click for me.

https://github.com/a10y/group-commit/

- They assume their hard-to-program but faster architecture will get figured out by devs. It won't.

Or it will get figured out in the niche fields where people are willing to figure out really hard stuff to squeeze out max performance (PE, hedge funds, intelligence)

Either way agree, it's hard to get mass adoption without the software ecosystem feeding back in

They’ve left the consumer market and only do defense now

…you’re free to use other editors? People like Zed. They like IntelliJ. They like VSCode. If you have an aesthetic preference against all professionally maintained IDEs, I think you’re in the minority.

I've had numerous encounters where doctors (and dentists) attempt to charge me for services they've already been reimbursed for from the insurance company.

It's only after hours of scouring my EOBs and being on the phone with my insurance that I then come back to the practice's office with evidence in hand, and they dismiss the charges.

I'm pretty sure this is just a racket because they expect most people not to put up a fight and just pay, or get sent to collections hell.

The amount of work you need to do as a patient in our health system is so dumb.

QGIS is the shit. I absolutely love it, great for visualizing GeoJSON, GeoTIFF files, open data feeds, etc. My one gripe is that their macOS installers have been out of date for ages now, the best way I've found is to actually install from Conda Forge directly:

brew install micromamba

mamba install qgis

It's really crazy the number of open geospatial data feeds that exist out there from NASA, NOAA, and ESA. If you're interested in checking any of this stuff out, I highly encourage following Mark Litwinchik's blog, this guy is a legend and he does most of his work with open tools like QGIS and DuckDB

https://tech.marksblogg.com/

We don't get a tax write-off.

The motivation is to move the IP and trademark into a separate organization so it's no longer owned by Spiral. This means we can't re-license it later, we'd have to fork it, because the Vortex trademark and all that is controlled by LF.

Believe it or not, this is how the Linux Foundation organizes itself. It's more legwork than something simpler like Apache Foundation.

Basically in the US you need a legally recognized entity to hold intellectual property. "Donating" the project involves setting up a "Series LLC" that is nested underneath the top-level Linux Foundation corporation, and donating the IP into it.

Checkout https://docs.linuxfoundation.org/lfx/project-control-center/... and ctrl-f "LF Projects, LLC"

The whole "donated by spiral" on the vortex.dev website also gives big tax write-off vibes.

Donated is the Linux Foundation terminology.

Sadly the last time I filed a tax return there was no way to itemize a Github repo. Alas.

The AnyBlox paper has some very cool ideas, and the authors are friends.

In the paper you'll notice a large portion of it analyzes Vortex, both standalone and embedded. Definitely worth a read.

You’re welcome to read the code yourself once you check it out, it’s not very big. Supply chain attacks are a thing but I don’t think this is one.

There is nothing wrong with running a receive-only hotspot. Not sure what you’re implying here.

Just wanted to advertise that the EFF recently released an open source tool for detecting cell-site simulators. The hardware is like $20 and it's pretty easy to setup yourself. Worth having around to stay aware of what's out there, especially if you live in one of the places recently targeted by the administration.

https://github.com/EFForg/rayhunter/

Sure, not directly. But most of that is imaging and communications payloads, and the vast majority there is being purchased by militaries and intelligence agencies.

Government still props up this whole market.