Finding Miscompiles for Fun, Not Profit

https://newsletter.semianalysis.com/p/finding-miscompiles-for-fun-not-profit

by tmoertel • 2 months ago

41 9 2 months ago

newsletter.semianalysis.com

mNovak• 2 months ago

Given the $10k price tag for tokens and high rate of bugs (several per minute) they mention, it'd be very interesting to see this experiment run with cheaper models too.

I wonder if we get to a world where a full repo sweep like this is a default Github action after commit.

1over137• 2 months ago

Most C/C++ projects I know don't even run tests with ASan/TSan/UBSan before each commit/merge.

Quarrel• 2 months ago

and in the meantime, just a sweep of the committed code (or the to-be-committed code for lots of us) and the code it interacts with, is increasingly catching lots of problems.

jlebar• 2 months ago

Author here; I'm happy to answer questions, take criticism, etc etc.

eqvinox• 2 months ago

Codex and I collaboratively wrote a fuzzer.

Why are you using phrasing that equates AI and humans? You used Codex to write a fuzzer. It didn't decide to join you.

derdi• 2 months ago

Why are you using phrasing that equates AI and humans? Codex isn't in a position to decide whether to do work.

ebiederm• 2 months ago

Thank you for posting this.

I had heard LLMs were finding a lot of bugs very quickly and now I can see what that looks like from a user perspective.

rurban• 2 months ago

Boy, I told DeepSeek V4 Flash Free to find compiler bugs, esp. with the gcc torture test suite, and it did find plenty. For free. To fix them in my compiler it did cost about $40 or so. Cooperate guys just have too much budget in their hands to throw at the rich.

Traubenfuchs• 2 months ago

I wonder how much damage all those countless bugs caused in real life.

Does MRI low level code produce wrong images? Do some kind of unexpected http connection quirks happen? Does (LL)M inference produce randomly wrong and non reproduceable output? Graphical artifacts in video games? Application crashes that happen once every billionth request? Security vulnerabilities? Race conditions?

Finding Miscompiles for Fun, Not Profit

Related Stories