akhong

I think Stuart Jay Raj has the best content on tones and making yourself understood in these tonal languages. Check out his Youtube channel: https://youtube.com/@StuartJayRaj

Yes, I agree with you in Pandas' case. However, for other libraries, a good chunk of the run time comes from reading the parquet files and concatenating the partial datasets. Pandas and Spark are particularly really good with reading a directory of 12 Parquet files with no noticeable performance penalty.

For this particular data and on my machine, that was certainly the case! I've been shown other benchmark results (such as this one: https://h2oai.github.io/db-benchmark/) that demonstrate otherwise. I'm not really sure what to make of it - maybe try more cases?

One possible explanation I could think of is that Pandas support for Parquet is pretty good compared to data.table and Julia. I've been asked to split the read/write part and the groupby-agg part for a more complete picture. I'll be sure to work on that in the coming weeks.

Another hypothesis by u/joinr about why Pandas performs better in the smaller dataset:

"I wonder if there's some default column size allocation that happens up front for the 2^6 case that helps prevent growth in pandas, and maybe the hueristic falls down a little as the dataset gets larger leading to more resizing."

I'd just like to say that I made this comparison, and I hope it is a fair one. Any feedback on how to improve the current versions or suggestions on other libraries/approaches to include would be greatly appreciated!

Got a "System error occurred. If the error occurs repeatedly, please contact the Help Desk."

Probably taken down?

Wrapping Spark in Clojure:

https://github.com/zero-one-group/geni

I used to work at a tech giant, where the data team relies a lot on native Spark in Scala. I've always found the combination quite pleasant to work with. However, I did miss Python's faster startup time, dynamism and REPL, especially when doing data cleaning and exploration with no intention of putting it in production.

Now that I'm doing my own thing at a much smaller scale, I naturally gravitated towards Python's data stack, namely NumPy, Pandas, Sklearn and Dask. However, I found myself missing Spark's consistent SQL API and performance!

So yea, I've been wanting to use more Clojure for work and set up a Clojure shop. During the quarantine, I find myself having more time to do focused work. I thought this would be a good opportunity to convert some of the data wrangling stuff to Clojure!

DNS error for the docs http://docs.hylang.org/ :/

I recently gave a talk on Coconut in PyCon Bangkok: https://youtu.be/24DWw6Ozkvo

Not working in Thailand too