JDF.jl – Julia DataFrames serialization format

by xgdgsc • 3 years ago

55 60 3 years ago

moomin• 3 years ago

Isn’t the Python data frame format becoming an established format? I mean, there’s already commercial products that talk it natively.

jzwinck• 3 years ago

There is no practical Pandas DataFrame binary format on disk. They have "tofile" which is not portable across machines. They have "to_hdf" which produces files no other library understands in an intuitive way. They mostly recommend "to_csv" with all the usual problems of CSV.

NumPy does have an official documented format which is straightforward to read and write from other languages if you write some code yourself, but still doesn't have very wide support outside of Python.

dijksterhuis• 3 years ago

Parquet is a binary file format and is very widely supported.

I think there’s even an Excel plug-in for parquet these days.

benglish11• 3 years ago

Arrow is the more universal data format. It is already used to serialize non-python data structures in libraries like spark , snowflake-snowpark and polars. Pandas can read/write in arrow format as well. In pandas 2.0 the numpy backend can even be replaced with a pyarrow backend. I think this directly contradicts your statement about “no practical pandas binary format exists”. That may have been true a few years ago but certainly is not an issue any more.

constantcrying2• 3 years ago

Python is definitely more popular and its formats are naturally more widely spread.

Python is also a very bad language for many of the things it is used for (anything to do with numerics, e.g. machine learning) and having a competing format from a language which is, at least in that regard, far superior seems like a good thing.

jstx12• 3 years ago

It's not competing if it's limited to Julia. And saving a dataframe isn't on anyone's list of technical concerns anyway.

It's a big stretch to say that a tiny Julia library which does a thing we can already do well in Julia and Python and just about any other language has any impact on the language ecosystems and how they compare to each other.

keithalewis4• 3 years ago

It is impossible to have an intelligent discussion with someone who only knows Python, FSVO know. Dunning-Kruger in action.

juujian• 3 years ago

Apache Arrow sits in a similar niche, but it has support for multiple languages.

crabbone2• 3 years ago

Python data frame format

That is not a thing. The closest would be the NumPy binary format, which is "nothing to write home about". It's not good in any particular way. Some sort of a stop-gap solution for when you really want to save the contents of a NumPy array very much, and you don't want to deal with writing some commonly supported format that's (hopefully) optimized for some particular task.

amj7e2• 3 years ago

This is a pretty cool package and very useful. Is the project stale? What are the plans going forward?

markkitti2• 3 years ago

There was a release 25 days ago. The HN submitter might even be the author.

amj7e2• 3 years ago

Yeah, but it was just a project.toml update. There haven't been serious updates to the src for a long time that's why Im asking. Hopefully, it resumes.

zarkenfrood• 3 years ago

Looks interesting and the syntax appears to be very similar to R data.table. It will be interesting to keep an eye on once it gets to v1.0.

jstx12• 3 years ago

I don't see the value of a language-specific format when we have things like parquet.

thumbuddy4• 3 years ago

I think Julia is one of those researcher languages like Haskell so it's probably just someone having fun, writing a paper, or whatever.

Buttons840• 3 years ago

I don't agree with this. Julia does not have new ideas the same way Haskell does, and Julia has done a lot of work to be practical, like having a good way of calling Python and R code from early on in its life. If Julia has done anything new, it's probably related to being fast, but that's not exactly a novel idea.

VHRanger2• 3 years ago

Julia is still niche, but it's definitely used in production?

The NY Fed have their main model of the economy in Julia (ported from Matlab)

ChrisRackauckas3• 3 years ago

The repo is pretty clear on its intentions to not be a language-specific format:

Development Plans
I fully intend to develop JDF.jl into a language neutral format by version v0.4. However, I have other OSS commitments including R's {disk.frame} and hence new features might be slow to come onboard. But I am fully committed to making JDF files created using JDF.jl v0.2 or higher loadable in all future JDF.jl versions.

It's beyond v0.4 and at v0.5.1 right now so it seems delayed, but the repo describes intentions to be a language-neutral format after the dust settles. I think we're just taking a snapshot too early in the timeframe to really know what this project is truly all about. And given that the author is an OSS dev for multiple languages, this intention makes a lot of sense.

tjrgergw3• 3 years ago

What exactly is the advantage of this vs Arrow.jl?

ChrisRackauckas3• 3 years ago

If I'm not mistaken, Arrow is an in-memory format and not necessarily a storage format. Though it is common to use Arrow with the Parquet serialization format, and the benchmarks here show a much faster read performance than Parquet while achieving around the same file size. From my read, that seems to be the impetus for the project (with comparisons made on Linkedin by the author as well) and so in theory Arrow and this could be complementary like Arrow and Parquet if someone setup the links.

zokier• 3 years ago

the benchmarks here show a much faster read performance than Parquet

That felt bit surprising.. is parquet slow to read in general or is the julia implementation just slow?

tjrgergw3• 3 years ago

The benchmark comparison is vs Feather.jl.

braza• 3 years ago

It’s sad that Julia lost the so much preference in the Data Ecosystem. I think the syntax it’s clean, it’s fast, a lovely community and a clear vision on what needs to be done.

goerz• 3 years ago

Has it “lost” preference? I was more under the impression that it just hasn’t yet gained as much popularity as it deserves based on its technical merits, due to being quite a young language and the entrenchment of a large existing Python ecosystem.

bobbylarrybobby2• 3 years ago

I think the initialism JDF may already be taken

https://en.m.wikipedia.org/wiki/Jewish_Defense_League

eindiran• 3 years ago

That's "JDL", which isn't the same 3 letters.

bobbylarrybobby2• 3 years ago

Whoops, I meant https://www.jewishdefensefund.org/

derstander• 3 years ago

I think the initialism JDF may already be taken

Acronyms are lossy compression. I’d bet there’s a JDF that even predates the one you reference in one of your descendant comments.

I’d be interested in knowing (though admittedly not enough to look) if there’s even a three letter acronym using the English alphabet that’s not already in use.

akdor1154• 3 years ago

If the parquet julia lib is slow, just go and fix that? Sorry I don't mean to discourage people trying cool new things, but seriously parquet is great, and I'd much rather see a diversity of great parquet implementations than a diversity of language specific parquet competitors.

markkitti2• 3 years ago

The comparison in the README is to the parquet package for R.

zX41ZdbW• 3 years ago

Interesting to see a comparison with ClickHouse Native format.

SilverBirch• 3 years ago

The difficulty with serialization formats is that you're serializing for a reason. What if I want to read these dataframes into C++ or Python or some other language? It's pretty rare for pure performance to be the only consideration in choosing a serialization format - and unfortunately all the other reasons favour well-supported, long-established projects that can afford to hand out libraries in every language you could want.

ElectronBadger• 3 years ago

Thanks, added to the Big Book of Julia (https://adamwysokinski.codeberg.page/bbj).

JDF.jl – Julia DataFrames serialization format

Related Stories