HN user
Blas and faer are used only for small corners of the API (linalg and fft) which is exactly what numpy does. I encourage you to follow your own advice and look more closely at the interaction of ufuncs, strides, and dtypes.
Yup.
One can have fun with all manner of things. Take wood-working for example. One can have fun with a handsaw. One can also have fun with a table saw. They're both fun, just different kinds
If you read on in the post you might be interested in the section titled
Drop Python: Use Rust and Typescript
https://matthewrocklin.com/ai-zealotry/#big-idea-drop-python...
Original author of the post here. Just to give credit where it's due, that was a quote from this other excellent article written by someone else:
https://www.stochasticlifestyle.com/a-guide-to-gen-ai-llm-vi...
I mean, Dask doesn't have money. We're definitely not in a place to pay them a bounty. I imagine this is just marketing on their part, or driving up some metric to show customers.
"Our powerful AI has identified vulnerabilities in 836 projects, many of which you depend on. How can you, enterprise customer, afford not to pay us money?"
I'm not convinced that human judgement was ever applied during this situation.
Hey folks, original author here. It seems like people here are really connecting with the specifics of the code review example. The main point of the article is really "what we learn in reviewing code in community open source might not transfer well to providing feedback to human behaviors in work environments".
Please feel free to keep engaging on the code review bits (a timeless topic among programmers for sure) but I'd also encourage people to expand discussion out to how we manage humans and give them feedback in a way that both helps them grow and makes them feel supported at the same time.
Cheers, -matt
Parallel for loops are the new black I guess
$25 to process 250TiB seems cheaper than what I would expect.
I've had (very good) beers that cost that much :)
Thanks for the comments. I wrote most of that document so I'll try to explain my reasoning in line.
First and foremost, it would make more sense to compare against the DataFrame API of Spark, which is very Pandas like.
It would make more sense to me to compare dask.dataframe to spark's dataframe. This document is comparing dask to spark. dask.dataframe is a relatively small part of dask.
> "Dask gives up high-level understanding to allow users to express more complex parallel algorithms."
I don't think this is true. Their example of complex algorithms (SVD) is not that complicated, and there are even implementations of that in Spark's MLlib directly. Spark's DAG/RDD API is essentially the low level user-facing task API.
The point here is that it's quite natural for dask users to create custom graphs (here is another example matthewrocklin.com/blog/work/2015/07/23/Imperative/). Doing this in Spark requires digging more deeply into its guts. This sort of work is not idiomatic or much intended in Spark.
Loading "terabyte" of JSON into Postgres seems pretty painful
I've found that loading a terabyte of CSV into Postgres or a terabyte of JSON into Mongo to be quite pleasant actually. I'd be curious to know what problems you ran into.
Flat CSV or JSON files are hard to parse. Fast CSV parsers and gzip decompression both run at around 100MB/s. If you want to get faster than this you'll need to use better (ideally binary) formats.
This notebook might interest you: http://nbviewer.ipython.org/gist/mrocklin/c16c5c483b2b9859de... , particularly the sections starting at "Eleven minutes is a long time." It compares CSV costs (minutes) to custom binary storage formats (seconds) on a 20 GB dataset.
Interesting. What are your thoughts on the costs of dependencies? What about toolz makes you hesitate to depend on it?
You can manage this by composing merge and merge_with
In [1]: data = [{'a': {'b': 1}}, {'a': {'c': 2}}]
In [2]: from toolz import merge, merge_with
In [3]: merge_with(merge, data) Out[3]: {'a': {'b': 1, 'c': 2}}
You'd have to be a bit clever to go fully recursive