HN user
https://chriscummins.cc
I'm not a big fan of the encroachment of AI into Adobe's apps, so I'm using AI to build a replacement for those apps (a small web-based photo organizer and editor, just the tiny subset of the Lightroom features I need for my workflow)
Hey! The idea isn't to replace the compiler with an LLM, the tech is not there yet. Where we see value is in using these models to guide an existing compiler. E.g. orchestrating optimization passes. That way the LLM won't break your code, nor will the compiler (to the extent that your compiler is free from bugs, which can tricky to detect - cf Sec 3.1 of our paper).
Is a MacOS build of YNAB 4 still available?
We use the same architecture as other LLMs, but we include no natural language in our pretraining. We figured a single-domain training corpus would make evaluation easier. We’ll be looking at layering this on top of something like Code Llama next
Hey, we’re targeting code size in this work, not runtime performance. You would use an option like -O3 to optimize for runtime and -Oz to optimize for code size. The pass pipelines are different for both
Program synthesis is part of the loss function, which is what makes it a auxiliary learning task.
We haven’t experimented with model size yet, we just used the same configuration as the smallest Code Llama. We did play with dataset size and found thah performance tracks the usual scaling laws. Details in the paper
There’s a researcher at Edinburgh who is looking into this. You may want to reach out. Check the related work section in this paper
Hey Jon, we don't change semantics. We just choose optimization passes to run to match a particular input code. Agreed that code size wins could regress performance. We don't measure that yet, but will be looking into it next. There are some applications where codesize above-all-else is what to optimize for.
Hey, yes that's right, and good callout on GEP instructions. We admit that instruction count is a bit handwavy, but we use it as a starting point as that's what the prior works we compare against optimize for. We'll be looking at true binary size next, and code runtime after.
Hey, author here. We use machine learning to generate a list of _optimization passes_ for the compiler to run. These optimization passes give us a 3.0% improvement over the default (-Oz), and are what generates correct code. We don't do anything to the code that breaks semantics (assuming no bugs in the compiler passes; some nuance is needed here ;) ).
We also train the model to generate what it thinks the optimized code will look like. We find that this helps the model choose better pass lists, but obviously the code cannot be trusted and semantics are not guaranteed. It only compiles in 91% of cases. "Perfectly emulating the output of the compiler" means the model spat out code that is character-for-character identical to what the compiler generates with the given pass list (even choosing the same variable names etc). IMO this is no mean feat, but is still a long way to go before using LLMs for codegen. We provide a bunch of examples in the paper of things that LLMs can and cannot do.
This. For now we rely on differential testing against a gold-standard implementation (e.g. unoptimized). For the action space we expose, any semantics-breaking change induced by our tool is a compiler bug.
Author here. In the sense that a trained neural net produces the same output from the same input, this is deterministic. But I don’t think that’s what you’re getting at. Where it gets interesting is if we inserted a feedback loop (such as FDO) so that the compiler could fine tune itself on past decisions. In that case, the build would still be deterministic, but the compiler would keep changing, invalidating the build cache.
I'd be careful about asking your co-workers for "dick pics".
No, HI is Brady Haran and CGP Grey, though I suspect Brady may have collaborated with Tom.
The paper says "we started a high-priority project to quickly produce a custom ASIC for inference (and bought off-the-shelf GPUs for training)."
Give texstudio a try. I find it offers exactly the kind of IDE-style features that I was missing from Emacs and other editors.
For offline writing (e.g. emacs/vim), I've found textlint[1] to work really nicely, and fits well into a LaTeX+Makefile workflow.
No. && would cause the next command to begin once the previous had successfully completed. A successful install shoudn't start the next install attempt, hence ||.
On the other hand, OP and this comment miss the beauty of Randall's implementation, which is each operation is launched asynchronously ('&'), so it isn't the case that you try a few approaches one at a time, i's that you try all approaches at the same time.
I'm not sure if I can agree with that reasoning. Sure, no-one is impressed with an Apple II now, but that doesn't degrade how significant it was then. By that measure, did the Wright bothers really need to put in all of that time since today we have 747s?
I love how in all of their "meet the fleet" headshots, the birds are totally out of focus, only the grass in the background is clear
Great post, definitely giving this a try.
Off topic, but does anyone know how the author generated this web version of the paper? It looks like as though they've automatically generated it from the LaTeX source of the PDF. It looks brilliant, and much easier to read in the browser.
I'm no biologist, but I don't think our brains are going around trying to do a grid search for the best hyperparameters.
Is that not essentially the process of evolution through natural selection? One giant optimisation problem?
I think the author does a great job at venting frustration at the state of autotuning systems research, though I would disagree that the research interest has dried up. On the contrary, autotuning research is alive and kicking, the problem is that there has been few attempts to unify all the competing systems that exist (with some exceptions [1]). As such, the state of autotuning is fragmented, with no one approach able to achieve the critical mass needed to hit the mainstream.
Disclaimer: I'm doing a PhD in autotuning ;-)
Great stuff. I personally find it useful to have an additional "Regression" label to tag bug reports that have cropped up as a result of new work (this happens surprisingly often).
Since this is for git repositories you'd probably want to sub that `find -type f' for `git ls-files'.
The fitness value represents the difference between the pixels of the "goal" image and the pixels of the current individual [1]. It will not necessarily reflect the perceived similarity between two images, which will change from person to person.
https://github.com/ChrisCummins/chriscummins.github.io/blob/...
You are right. The "DNA" label is inaccurate, but jal278 explained the reasoning perfectly. I'd be happy to hear suggestions for a better term though! ;)