shoyer

The short answer is that tracing is way, way easier to implement in a predictable and reliably performant way. This especially matters for distributed computation and automatic differentiation, two areas where JAX shines.

AST parsing via reflection means your ML compiler needs to re-implement all of Python, which is not a small language. This is a lot of work and hard to do well with abstractions that are not designed for those use-cases. (I believe Julia's whole language auto-diff systems struggle for essential the same reason.)

Glad to see that you can make ensemble forecasts of tropical cyclones! This absolutely essential for useful weather forecasts of uncertain events, and I am a little dissapointed by the frequent comparisons (not just you) of ML models to ECMWF's deterministic HRES model. HRES is more of a single realization of plausible weather, rather than an best estimate of "average" weather, so this is a bit of apples vs oranges.

One nit on your framing: NeuralGCM (https://www.nature.com/articles/s41586-024-07744-y), built by my team at Google, is currently at the top of the WeatherBench leaderboard and actually builds in lots of physics :).

We would love to metrics from your model in WeatherBench for comparison. When/if you have that, please do reach out.

ERA5 covers 1940 to present. That's well before the satellite era (and the earlier data absolutely has more quality issues) but there's nothing from 170 years ago.

Not sure if $199 is reasonably priced in your opinion, but the Nest Wifi Pro supports 6E.

+1 one of most common mistakes for PhD students is picking a project based on research interests rather than the adviser. I believe it is relatively uncommon to speak with former students of an adviser, but this is something that everyone entering a PhD should do.

Finding a good mentor -- someone whose values you agree with and who sets you up for career success -- is far more important than working on any particular topic of interest. The world is full of interesting research topics, and very few PhDs work in the precise area of their PhD research for their entire career.

df.pipe(f, ...) is just syntactic sugar for f(df, ...). Nothing slow about it.

I work on weather prediction, both with traditional simulation methods and machine learning. I have not seen any examples of cellular automata used for useful predictions in this space.

I guess "quantum physicist" can mean either an academic credential or a job title. In my case, I have the former but no longer the later. I finished my PhD nine years ago and no longer work in the field.

As former quantum physicist, I find it little troubling to read "quantum theory has reached a dead end" in specific reference to the interpretation of quantum mechanics. Most quantum physicists could not care less about how quantum mechanics is interpreted when it makes highly accurate quantitative predictions, and there are still plenty of interesting open problems for quantum theory (e.g., related to the practical design of algorithms and hardware for quantum computers).

This article also misses what is likely the leading interpretation of quantum mechanics by actual quantum physicists, namely that the measurment problem is solved by decoherence (the quantitative theory of how classical states emerge from quantum states):

https://en.wikipedia.org/wiki/Measurement_problem#The_role_o...

https://royalsocietypublishing.org/doi/10.1098/rsta.2011.049...

My experience was that robo-investors are great until you need something special. Then they can become rather painful.

Exmaple: I got divorced last year. Betterment took weeks of time and many phones calls until they were able to figure out a way to divide our assets evenly, without a large difference in cost basis. Their automatic algorithm for dividing accounts just didn't know how to handle it.

If UBS figures out how to offer a higher level of service on top of robo-advising, that could be a real win.

Foxconn once said it would spend $10B building a factory in Wisconsin. Now the claim is $672M -- over next six years: https://en.wikipedia.org/wiki/Foxconn_in_Wisconsin

We'll believe it when it happens.

Most of the "bugs" caught here (including in TensorFlow and in my own project, Xarray) seems to actually be typos in the test suite. This is certainly a good catch (and yes, linters should check for this!), but seems a little oversold to me.

NERSC is part of DOE’s Office of Science. Nuclear weapon development is done by DOE’s National Nuclear Security Administration (NNSA), which sponsors labs like Lawrence Livermore and Los Alamos. It definitely isn’t NERSC, NNSA has its own dedicated supercomputers for weapons work.

Neural networks need completely different optimisation methods, and there is no practically useful application of any of the Newton or Quasi-Newton methods for their optimisation.

I don't think this is quite fair. There are several variations of 2nd order methods, notably KFAC and Shampoo, that seem to quite effective for large-scale neural network training, e.g., see the intro of this paper for an overview: https://openreview.net/forum?id=-t9LPHRYKmi

A PhD is certainly not for everyone (or even most), but it's rather unfair to say that it does not teach any employable skills. Even PhDs who go on to work in completely unrelated fields learn how to make progress on poorly defined problems and to advance the frontier of human knowledge. And a PhD is also of course a hard prerequisite for career in academia or research.

As for that survey of PhD students, I suspect you would get similarly dismal reviews of parenthood from parents of 0-5 year old children -- but of course that doesn't mean that nobody should have children! A better survey would ask PhD students how they feel about the experience well after they are done with it.

ECMWF makes probabilistic forecasts, in the form of an ensemble of 50 IID examples. So this is mostly matter of Windy figuring out how to put that information into their UI.

The general case of implicit differentiation, i.e., for functions y(x) defined by the constraints F(x, y(x)) = 0, where x and y are vectors, is solved by "implicit function theorem": https://en.wikipedia.org/wiki/Implicit_function_theorem

∂ y(x) = -(∂_1 F(x, y(x)))^{-1} (∂_0 F(x, y(x)))

where ∂ denotes partial differentiation.

This turns out to be an incredibly useful identity for calculating derivatives. No matter how you calculated a solution to the equation, computing derivatives is "just" a matter of performing a linear solve.

If the calculation you performed is a solution to solving an equation, implicit differentiation is typically much faster, less memory intensive and more accurate than calculating derivatives by differentiating through your solver. For examples, you might check-out a recent paper I co-authored with colleagues at Google: https://arxiv.org/abs/2105.15183

This looks really interesting.

Does anyone know where the money is coming from? I'm guessing foundations like Schmidt Futures? It seems like financing is the crucial challenge for getting anything like this off the ground.

This post is yet another example of why you should never use APIs for random number generation that rely upon and mutate hidden global state, like the functions in numpy.random. Instead, use APIs that explicitly deal with RNG state, e.g., by calling methods on an explicitly created numpy.random.Generator object. JAX takes this one step further: there are no mutable RNG objects at all, and the users has to explicitly manipulate RNG state with pure functions.

It’s a little annoying to have to set and pass RNG state explicitly, but on the plus side you never hit these sorts of issues. Your code will also be completely reproducible, without any chance of spooky “action at a distance.” Once you’ve been burned by this a few times, you’ll never go back.

You might think that explicitly seeding the global RNG would solve reproducibility issues, but it really doesn’t. If you call into any code you didn’t write, it might also be using the same global RNG.

By the way, is there another reference with a full description of how the benchmarks are setup? I'd be curious how Xarray (end user API) + Dask (distributed compute) + Zarr (distributed storage) compares.

Xarray definitely takes a different philosophical approach based on its roots in the Python data science ecosystem, compared to "all in one" solutions like a full array databases.

Hi Peter, good to run into you again :)

I agree, formulating open benchmarks is great work, and there's nothing suspicious about performing well on benchmarks you write. It's just worth keeping in mind: https://matthewrocklin.com/blog/work/2017/03/09/biased-bench...

My initial remarks were a little careless and overly provocative! I don't doubt that there are cases where a true "array database" provides value. I do think the use-cases are less clear for arrays than they are for tabular data, because the users of arrays tend to be more sophisticated.

The short answer is that I wrote Xarray before I showed up at Google :)

It’s worth noting that the author of this report is Peter Baumann, the author of Rasdaman. So it shoukd be no surprise that Rasdaman comes out on top in the various benchmarks and is presented as the leading “array database.”

My two cents (as the author of Xarray, one of the Python libraries mentioned in this report) is that it’s questionable whether we need “array databases” at all. Certainly we need to be able to store arrays and compute with them, but do we need an integrated solution that does both at the same time with a query language that looks like SQL? Maybe not, in an era of cloud computing, prolific open source software and when everyone who works with big array datasets already knows Python.

Here’s the paper: https://pdfs.semanticscholar.org/53e2/70a3f29cf556849362130e...

Unfortunately on closer examination the results look pretty meaningless to me. The study involves rather small numbers (e.g., 4/33 vs 10/44 divorces), but an even bigger issue is that the “control” arm in this study wasn’t selected randomly, but rather consisted of couples “declining active treatment” or who “could not be scheduled for active treatment.”

So a better summary might be “Divorce rate cut in half for coupling willing and able to take part in a relationship building program, of any sort.”

If these psychologists had only been a bit less greedy about wanting to compare three different treatment arms rather than a real control arm, they could have set up a study that could actually have taught us something...

I'm a big believer in auto-diff, but I'm skeptical that any autodiff tool would differentiate a 100k line simulation code correctly and efficiently without manual intervention. I'd certainly love to be proven wrong, though and absolutely AD can be a big time saver :)

To be more precise, in backwards mode auto-diff, inputs only need to be saved if they are used in a non-linear way.

All good points! "Non-deterministic" behavior within the same program/process is still a bridge I would not want to cross. This could result in subtle glitches, e.g., when a user hits "refresh" with the same inputs, and could make reproducing bugs impossible.

I am a strong believer in always using a seed for random number generation for exactly these sorts of reasons. (Side note: deterministic RNGs is one of my favorite features about JAX.)

As someone who builds neural networks routinely, this sort of non-reproducibility sounds troubling to me. We expect small differences for floating point arithmetic between platforms, but integer math is typically exact.

This is all the more concerning for 8-bit quantized arithmetic, where off-by-one means a relative error of about half a percent. If a individual layers in a quantized neural net have off-by-one errors with a consistent bias, I can imagine these errors accumulating into significant losses in model quality in deep networks. There isn't a huge margin for error in quantized neural nets.

One concern about the article: it uses the word "non-deterministic" in a slightly misleading way. I assume any specific hardware is still expected to produce consistent results when run twice on the same input. So it's more non-reproducible than non-deterministic. Compensating for inconsistent arithmetic on different devices sounds much more feasible than compensating for stochastic arithmetic.

Not being a “conscious racist” doesn’t make you any less racist. Racism isn’t about intentions, it’s about consequences. It’s a convenient slight of to pretend otherwise.

This is not a great idea, unless you’re sure the specific expertise you are building will get you a job.

In tech, we typically hire PhDs and postdocs in at the same level. Sure, you’ll probably advance a little faster with experience from a postdoc, but not faster than if you started straight out of your PhD.

Postdocs are preparation for academic research in a specific field. Most industry jobs don’t work like that, and most PhDs don’t end up working in the subfield of their PhD, either.

Don’t mislead yourself into thinking that a postdoc is a good default option for a PhD graduate. Way too many people do postdocs, and end up looking a little foolish when they look for industry jobs later.