markisus

The post makes it seem like ONNX is CPU only. I've used ONNX runtime to run models on Nvidia GPUs. The runtime can even dispatch to TensorRT. I'm not sure what the performance is on Apple hardware so maybe that was the motivation for moving away from ONNX.

This is every cool research.

Do you have any idea why the authors chose Z80 as the program language? I have seen other studies in the same spirit that use simpler toy languages like Brainfuck (https://arxiv.org/abs/2406.19108) and I wonder if you could get higher execution speed if you didn't have to execute so much emulator code.

The programs/genomes are extremely tiny. I would be very interested to see what kind of hardware is needed to scale this approach up. How long until we can feed in giant corpuses of text and evolve these little organisms to predict the next letter?

It's funny that the machine intelligences that arose out of the bitter lesson cracked this benchmark by meticulously modeling each individual case with rule based approaches.

When humans tried making AI through rule based approaches, maybe we only failed because we couldn't type out the rules fast enough. Or we got burned out after writing the thousandth heuristic to fix the never ending edge cases. And we also got side tracked by concerns like maintainability, modularity, and code re-use. But I wonder if we take modern or near-future LLMs who never tire and write code faster than any human, could we make a frontier level GOFAI agent?

Claude code has some basic security features like asking for user confirmation for bash commands, or restricting commands to the current directory. If these features are not being code reviewed, what assurances do we have that they actually work?

The article mentions Triton for this purpose. I don’t think you will get maxed out performance on the hardware though because abstraction layers won’t let you access the fastest possible path.

Don’t get hung up on the Claude part. We already know an algorithm that produces AGI: evolution. We don’t have the compute to run this algorithm because it requires simulating the whole Earth. But with enough compute, it becomes possible.

But this ruling will surely set precedent for other cases where AI is used. It may cover the case of AI summaries as well.

If you sell and then rebuy isn’t that considered a wash trade and therefore exempt from taxes?

QQQ is problematic because it’s influenced by strange back room dealings with Space X, if the article is to be believed.

VTI is different. It literally tracks all public stocks, weighted by market cap so no such manipulation is possible.

If a bunch of people will be forced to buy Space X (QQQ holders), active investors will short the stock in anticipation of market correction and money will flow from those who were forced to buy. I’m sure there are other ways to take advantage of a forced buyer situation.

Total market will be unaffected, assuming efficient market hypothesis / no arbitrage.

I’m trying to understand the mechanics here. I get that SpaceX and Nasdaq are in cahoots to get SpaceX bundled with a bunch of other stocks (and that bundle is called QQQ?)

But why must retail investors hold this bundle? If I’m holding now, I can sell it and buy a different bundle right? And if I’m not holding it now, I can just continue not to buy it after SpaceX gets included.

Like what you mean when you say “frames per second” in a video game. The image that is finally presented to the user, and then quickly thrown away and rebuilt, again and again to give the illusion of motion.

I also seem to be developing an immune response to several slopisms. But the actual content is useful for outlining tradeoffs if you’re needing to make your Python code go faster.

I wish there were more details on this part.

Missing @cython.cdivision(True) inserts a zero-division check before every floating-point divide in the inner loop. Millions of branches that are never taken.

I thought never taken branches were essentially free. Does this mean something in the loop is messing with the branch predictor?

Yeah if your program has a natural notion of a "frame" (eg most video games), you can do memory management by simply incrementing an integer (bump allocation). At the end of your frame, you reset the integer to zero. You can't really get any faster than that.

An additional benefit of this style of allocation over malloc/free is that you can get a lot of the same type of objects contiguous in memory so that iteration over them is a lot faster because there are fewer cache misses.

I'm not sure everyone knew that gaps reflected incorrect reasoning. It would have been natural to assume that all infinite sets were qualitatively the same size, since uncountable infinity was not an idea that had been discovered yet. Zeno's own resolution wasn't that his reasoning wrong, but that our perception of the world itself is wrong and the world is static and unchanging.

As for the importance of visualization (of the reals), I don't think you can cleanly separate it from formalism (as constructed in set theory).

I think we all have built in pre-mathematical notions of concepts like number, point, and line. For some, the purpose of mathematics is to reify these pre-mathematical ideas into concrete formalism. These formalisms clarify our mental pictures, so that we can make deeper investigations without being led astray by confused intuitions. Zeno could not take his analysis further, because his mental imagery was not detailed enough.

From clarity we gain the ability to formalize even more of our pre-mathematical notions like infinitesimal, connectedness, and even computation. And so we have a feedback loop of visualization, formalism, visualization.

I think the article was saying that Dedekind and Cantor clarified what we should mean when we talk about the number line, and dispelled confusions that existed before then.

Before their papers, mathematicians had assumed that even though the number line might look like a continuous object, if you zoomed in far enough, you’d eventually find gaps.

I'll try to interpret this sentence.

We all have some mental imagery that comes to mind when we think about the number line. Before Cantor and Dedekind, this image was usually a series of infinitely many dots, arranged along a horizontal line. Each dot corresponds to some quantity like sqrt(2), pi, that arises from mathematical manipulation of equations or geometric figures. If we ever find a gap between two dots, we can think of a new dot to place between them (an easy way is to take their average). However, we will also be adding two new gaps. So this mental image also has infinitely many gaps.

Dedekind and Cantor figured out a way to fill all the gaps simultaneously instead of dot by dot. This method created a new sort of infinity that mathematicians were unfamiliar with, and it was vastly larger than the gappy sort of infinity they were used to picturing.

Of course a contractor could not decide to unilaterally shut off their missile system, because that would be a contract violation.

A contractor may try to negotiate that unilateral shut off ability with the government, and the government should refuse those terms based on democratic principles, as Luckey said.

But suppose the contractor doesn’t want to give up that power. Is it okay for the government to not only reject the contract, but go a step further and label the contractor as a “supply chain risk?” It’s not clear that this part is still about upholding democratic principles. The term “supply chain risk” seems to have a very specific legal meaning. The government may not have the legal authority to make a supply chain risk designation in this case.

I felt the same.

This isn’t a sign of incompetence. It’s a sign that you’re actually doing real work.

Your value as an engineer isn’t in executing the same playbook forever. It’s in expanding the domain of what’s possible.

That experience doesn’t produce arrogance. It produces humility.

They’re not confident that they already have all the answers. They’re confident that they’ll find them.

This really reminded me of the first part Flowers for Algernon. The main character undergoes a treatment which improves is intelligence and the story is narrated via a series of diary entries which become successively more fluent and sophisticated.

I’ve confirmed this on my iphone as well.

Using swipe, no space bar after kill: Kill maps Jill myself Jill myself

Using swipe, manually pressing space bar after kill: Kill mussels Kill mussels Kill mussels

On the contrary, I think it demonstrates an inherent limit to the kind of tasks / datasets that human beings care about.

It's known that large neural networks can even memorize random data. The number of random datasets is unfathomably large, and the weight space of neural networks trained on random data would probably not live in a low dimensional subspace.

It's only the interesting-to-human datasets, as far as I know, that drive the neural network weights to a low dimensional subspace.

Each fine tune drags the model weights away from the base model in a certain direction.

Given 500 fine tune datasets, we could expect the 500 drag directions to span a 500 dimensional space. After all, 500 random vectors in a high dimensional space are likely to be mutually orthogonal.

The paper shows, however, that the 500 drag directions live in a ~40 dimensional subspace.

Another way to say it is that you can compress fine tune weights into a vector of 40 floats.

Imagine if, one day, fine tunes on huggingface were not measured in gigabytes, megabytes, or even kilobytes. Suppose you started to see listings like 160 bytes. Would that be surprising?

I’m leaving out the detail that the basis direction vectors themselves would have to be on your machine and each basis direction is as big as the model itself. And I’m also taking for granted that the subspace dimension will not increase as the number of fine tune datasets increases.

I agree that the authors decision to use random models on hugging face is unfortunate. I’m hopeful that this paper will inspire follow up works that train large models from scratch.

Where do lock free algorithms fall in this analysis?

Their VGGT, Dinov3, and segment anything models are pretty impressive.

The problem becomes complicated once the large discrete objects are not actuated. Even worse if the large discrete objects are not consistently observable because of occlusions or other sensor limitations. And almost impossible if the large discrete objects are actuated by other agents with potentially adversarial goals.

Self driving cars, an application in which physics is simple and arguably two dimensional, have taken more than a decade to get to a deployable solution.

The authors somewhat address your questions in the accompanying paper https://arxiv.org/abs/2410.24206

We emphasize that the central flow is a theoretical tool for understanding optimizer behavior, not a practical optimization method. In practice, maintaining an exponential moving average of the iterates (e.g., Morales-Brotons et al., 2024) is likely a computational feasible way to estimate the optimizer’s time-averaged trajectory.

They analyze the behavior of RMSProp (Adam without momentum) using their framework to come up with simplified mathematical models that are able to predict actual training behavior in experiments. It looks like their mathematical models explain why RMSProp works, in a way that is more satisfying than the usual hand waving explanations.

First I thought this would be just another gradient descent tutorial for beginners. But the article goes quite deep into gradient descent dynamics, looking into third order approximations of the loss function and eventually motivating a concept called "central flows." Their central flow model was able to predict loss graphs for various training runs across different neural network architectures.

Can someone explain the bit counting argument in the reinforcement learning part?

I don’t get why a trajectory would provide only one bit of information.

Each step of the trajectory is at least giving information about what state transitions are possible.

An infinitely long trajectory can explore the whole state space if there are no absorbing states. Such a trajectory would provide a massive amount of information about the system, even if we ignored the final reward.

These problems seem to have the flavor of interview questions I heard for quant positions.

Interesting article. It’s actually very strange that the dataset needs to be “big” for the O(n log n) algorithm to beat the O(n). Usually you’d expect the big O analysis to be “wrong” for small datasets.

I expect that in this case, like in all cases, as the datasets become gallactically large, the O(n) algorithm will start winning again.