necroforest

yeah, and as far as I'm aware there isn't even a definition of "civilization" or "intelligence" that doesn't boil down to "sufficiently like me".

I don't entirely disagree with you, but "what products do people want" is overly conservative. Pre-ChatGPT, very few people wanted a (more or less) general purpose chatbot.

It's branding (see: TensorFlow); also, pretty much anything (linear) you would do with an arbitrarily ranked tensor can be expressed in terms of vector ops and matmuls

cosine similarity is (isomorphic to) "distances to nearby objects". and not all embeddings are word embeddings.

If you need to relearn your iPhone/Mac interfaces every year, you might want to get screened for Alzheimers.

Who doesn’t think China has a ten year AI algorithm to takeover Taiwan?

anybody who works in either AI or natsec

We arent going to see more progress until we have a way to generalize the compute graph as a learnable parameter

That's a bold statement since a ton of progress has been made without learning the compute graph.

which laws are broken exactly? it's not remotely settled law that "training an NN = copyright infringement"

What's an order of magnitude between friends?

Maybe "best word processor for mac" is a better search for you?

LLMs are trained in parallel. The model weights and optimizer state are split over a number (possibly thousands) of accelerators.

The main bottleneck to doing distributed training like this is the communication between nodes.

which jargon here is "just for show"?

it's actually impossible to have a non-Hermitian matrix C such that x†Cx > 0

Nice catch, it's been a few years since I had to think about these details.

no, at least not for anything non-contrived.

I used Bayesian analysis for an industry problem where we had a lot of data (~1M samples), but the relationship between the observed data and the latent variables of interest was complicated and non-identifiable. In the non-identifiable setting, point estimates don't converge in the large data limit and quantifying uncetainty becomes critical.

JS divergence isn't just the average. it's 0.5(KL(P||M) + KL(Q||M)) where M is the mixture of P and Q: M = 0.5(P + Q)

kind of. you can decompose an arbitrary matrix into symmetric and antisymmetric components: R = S + A. Since A = -A^H (anti-symmetric), for any vector x, <x, Ax> = -<x, Ax> => <x, Ax> = 0. So for any matrix where <x, Rx> > 0, you can add an arbitrary anti-symmetric matrix and keep the same induced quadratic form. So people typically enforce symmetry in their definitions because it is the only part that contributes to the quadratic form and is "nicer" to work with (always diagonalizable, positive eigenvalues, etc.)

This should generalize easily to the complex/Hermitian case.

Does anyone use Ada in 2023? Back in college (~2008), I had a prof who was obsessed with it but AFAIK almost nobody used it even back them.

Nobody (to good approximation) plays with weights during the peer review process.

Having a project page with extra results is very common in computer vision, even in academia.

To answer (2): You are token i. In order to see how much of a token j's value v_j you update yourself with, you compare your query q_i with token j's key k_j. This gives you the asymmetry between queries and keys.

This is even more apparent in a cross-attention setting, where one stream of tokens will have only queries associated with it and the other will have only keys/values.

you can't store a function as a tensor. the tensors are the inputs/outputs that flow along the edges of the graph. TF stores things directly as a graph: https://github.com/tensorflow/tensorflow/blob/master/tensorf... while Jax represents computations in an intermediate representation: https://jax.readthedocs.io/en/latest/jaxpr.html

AI research has put hardly any effort into building goal-directed agents

The entire (enormous) field of reinforcement learning begs to differ.

no

not in collaboration with openai, one of the authors joined openai before the paper was written and arxiv'd

the genius of bfloat16 is that conversion is just truncation / zero padding

lol, because whining about your ex is a "basic fact"

yes, they are called Program Managers

i don't think the NSA has hitmen.

...yes. that's exactly what they do.