ajtulloch

- https://www.cs.utexas.edu/users/flame/laff/pfhp/index.html (e.g. here https://www.cs.utexas.edu/users/flame/laff/pfhp/week2-blocki...)

- https://gist.github.com/nadavrot/5b35d44e8ba3dd718e595e40184...

might be of interest

It's quite unfavorable on modern hardware. A Sapphire Rapids core can do 2 separate 32 half-precision FMAs (vfmadd132ph, [1]) per clock, which is 128 FLOPs/cycle. It is not possible to achieve that kind of throughput with an 8-bit LUT and accumulation, even just a shuffle with vpshufb is too slow.

[1] https://www.intel.com/content/www/us/en/docs/intrinsics-guid...

There’s a big long list of these in https://web.math.princeton.edu/generals/, not just Tao’s.

I think the mathematical concept that you are looking for is that of the dual space. Essentially if you have a vector space V, you can construct a dual space V* where the elements of the dual space are functions taking elements of V to the underlying field F, and under certain conditions these spaces are isomorphic (the same) - so there is a 1:1 correspondence between elements of the vector space and the functions in the dual space.

One way to view this formula is to use the fact that the Beta distribution is a conjugate prior for the binomial distribution.

Essentially if you have a Beta(a, b) prior then your prior mean is a/(a+b) and after observing n samples from a Bernoulli distribution that are all positive, your posterior is Beta(a+n, b) with posterior mean (a+n)/(a+n+b). So in your example you effectively have a Beta(0, x) prior and x (“suspicious”/“gullible”) is directly interpreted as the strength of your prior!

https://www.inference.org.uk/itprnn/book.pdf is a classic text on this connection.

You're probably thinking of https://en.wikipedia.org/wiki/Oliver_Cromwell and not https://en.wikipedia.org/wiki/Thomas_Cromwell.

https://github.com/facebook/folly/blob/main/folly/docs/Synch...

https://quantian.substack.com/p/from-trinity-to-liquidity is an excellent example of applying this idea to the problem of inferring the yield of a nuclear explosion and price impact of a trade in an order book.

“Fast Search in Hamming Space with Multi-Index Hashing” (https://www.cs.toronto.edu/~norouzi/research/papers/multi_in...) is a great paper. Note that you can do significantly better (in terms of query latency/throughput) with specialized implementations as in eg FAISS (https://github.com/facebookresearch/faiss/wiki/Binary-hashin...).

TF32 is not IEEE-754 float32, it is a reduced precision format designed for machine learning usecases. The correct specsheet number for FP32 (and FP64 which is the relevant precision here) throughput on H100 is more like 60TFLOP/s, so your number is off by roughly an order of magnitude.

https://news.ycombinator.com/item?id=7266618

The GotoBLAS paper (“Anatomy of High-Performance Matrix Multiplication”, https://www.cs.utexas.edu/users/flame/pubs/GotoTOMS_final.pd...) is really a masterpiece.

It’s worth internalizing almost every single detail if you’re an engineer interested in writing high performance numerical codes on modern hardware.

You might find https://en.m.wikipedia.org/wiki/Late_Victorian_Holocausts interesting!

I think the two key texts here are Graeber’s Bullshit Jobs and Mark Fisher’s Capitalist Realism, specifically his riff on “market Stalinism”:

The idealized market was supposed to deliver ‘friction free’ exchanges, in which the desires of consumers would be met directly, without the need for intervention or mediation by regulatory agencies. Yet the drive to assess the performance of workers and to measure forms of labor which, by their nature, are resistant to quantification, has inevitably required additional layers of management and bureaucracy. What we have is not a direct comparison of workers’ performance or output, but a comparison between the audited representation of that performance and output. Inevitably, a short-circuiting occurs, and work becomes geared towards the generation and massaging of representations rather than to the official goals of the work itself. Indeed, an anthropological study of local government in Britain argues that ‘More effort goes into ensuring that a local authority’s services are represented correctly than goes into actually improving those services’. This reversal of priorities is one of the hallmarks of a system which can be characterized without hyperbole as ‘market Stalinism’. What late capitalism repeats from Stalinism is just this valuing of symbols of achievement over actual achievement.

If you see his job to be at the cutting edge of Ruy/Marshall/anti-Marshall the results of games 1-6 suggest he’s done a phenomenal job.

https://rigtorp.se/notes/hashing/ and links therein (https://github.com/martinus/better-faster-stronger-mixer ) are almost certainly preferable to this mixer.

Good news! https://pytorch.org/docs/stable/notes/hip.html

and he's a genius!

For folks wondering about applications of this theorem: it is a key building block in the theory of reproducing kernel Hilbert spaces (RKHS), which in turn are the building block of kernel support vector machines (kernel SVMs), which are widely used in machine learning applications.

The "kernel trick" from kernel SVMs only works because of the existence and uniqueness result from the RRT on the underlying Hilbert space.

Underappreciated fact - the Apple SoCs since A10 (iPhone 7, etc) are also big.LITTLE designs! https://en.m.wikipedia.org/wiki/Apple_A10

A bit petty, but the first example is an unstable softmax implementation:

  defn softmax(t) do
    Nx.exp(t) / Nx.sum(Nx.exp(t))
  end

The OP article was submitted on Mon, 30 Nov 2020 23:02:47 - the arXiv identifier 2012.xyz means it was published in the 12th month of 2020, not 2012.

For everyone saying "oh wow, we can go back to SVMs now, huge speedups, etc" - that's more than a little premature. This is purely a formal equivalence, but not very useful computationally.

The math here is pretty much first-year undergraduate level calculus, and it's worth going through section 2 since it's quite clearly written (thanks Prof Domingos).

Essentially, what the author does is show that any model trained with "infinitely-small step size full-batch gradient descent" (i.e. a model following a gradient flow) can be written in a "kernelized" form

   y(x) = \sum_{(x_i, y_i) \in L} a_i K(x, x_i) + b.

However, none of those properties apply here, which is why this isn't particularly exciting computationally. The "trick" is that both a_i and K(x, x_i) involve path integrals along the gradient flow for the input x, and so doing a single inference is approximately as expensive as training the model from scratch (if I understand this correctly).

Awesome work Marat.

This sucks! Please don’t standardize this.

St. Petersburg in revolt gave us Vladimir Nabokov, Isaiah Berlin and Ayn Rand. The first was a novelist, the second a philosopher. The third was neither but thought she was both. Many other people have thought so too. In 1998 readers responding to a Modern Library poll identified Atlas Shrugged and The Fountainhead as the two greatest novels of the twentieth century—surpassing Ulysses, To the Lighthouse and Invisible Man. In 1991 a survey by the Library of Congress and the Book-of-the-Month Club found that with the exception of the Bible, no book has influenced more American readers than Atlas Shrugged.

– Corey Robin in "The Reactionary Mind"

The entire Landscape series from Macfarlane:

- Mountains of the Mind (a history and first-person account of mountain climbing)

- The Wild Places (a history and exploration of the 'wild' landscapes of the British Isles)

- The Old Ways (a history and exploration of the ancient paths of the world)

are all really excellent (and can be read in any order). He's a fabulous writer, kind of like a Kapuściński for the natural world.

Lots of companies use BT in a datacenter environment to efficiently distribute binaries to a large number of hosts.

I watched the documentary last night and it was fantastic. Wonderful work and thank you.