HN user
Building something cool. Email me at: will.wilson@antithesis.com
I’ve been running one for a while and love it. Have also built a Nix -> OpenWRT config language transpiler so that I can keep my router state in Nix files and have nice deterministic rollbacks etc. It’s been great!
Author here. That’s right — the challenge was representing a tree in an existing SQL database that we’d chosen for its pricing model. I think encoding a B-tree in SQL would be a lot less fun even than what we did.
Thanks,our company is in the DC area so I just reached out with an offer to chat. Wesnoth is an incredible project, I can't believe he doesn't have a programming job.
[Disclaimer: Started Antithesis]
If you run your Hegel tests in Antithesis, you get this for free (along with various sorts of “non-local” assertions, perfect reproducibility even for concurrent or distributed code, etc.).
But yeah, not hard to hack together basic coverage guidance outside Antithesis. That works well for large classes of programs, just not a majority of them.
If I had been wearing my fiendish CEO hat at the time, I might have even said something like: "somebody pointing this out will be a great way to jumpstart discussion in the comments."
One of the evilest tricks in marketing to developers is to ensure your post contains one small inaccuracy so somebody gets nerdsniped... not that I have ever done that.
Glad you enjoyed the talk! Making Bombadil able to take advantage of the intelligence in the Antithesis platform is definitely a goal, but we wanted to get a great open source tool into peoples’ hands ASAP first.
Not our first customer, but one we certainly appreciate!
Antithesis | Distributed Systems Breaker Extraordinaire | Full-time | SF or London or DC
---
Have you always wanted to channel your inner Kyle Kingsbury? We are looking to hire somebody with a deep understanding of distributed systems who can bring open source or customer software under test in our platform and find fun issues in it.
Preferred locations: SF, London, DC. Remote is a possibility for this role if you're really good (though be aware that we're a very in-person company culturally speaking, so it might be annoying for you).
To apply: email 'will' at our company domain with a description of something interesting that you've worked on.
I am one of the people who was on the panel. As always, it's a very lossy process when a journalist is summarizing another journalist summarizing a 90 minute discussion. Happy to expand in the comments here on any of the issues that got brought up.
It’s not quite a fair comparison, since an RL agent is trying to learn a policy that wins fair and square, while a fuzzer is able to take back moves. But if you’re working in a domain (like anything that can be simulated) where “time travel” is possible, you’d have to be crazy not to use it!
The longer you run it, the cleaner the run gets. But Metroid is a very compute-intensive game to fuzz, and we were already nearing the limits of what BigQuery could do for us with that run.
Yes, this is a really fun idea and something that we want to do. Though these days we’re setting our sights higher than Nintendo…
A funny story though: a regular conference gimmick we have is “Man vs. Machine” where we have attendees race our fuzzer to the end of Mario level 1-1. We did this at the final year of Strange Loop, and the fuzzer was winning handily until not one, not two, but three different professional speedrunners walked by and destroyed us.
You were successfully trolled. :-)
Haven’t tried Castlevania II, but here’s the first one: https://antithesis.com/blog/castlevania/
Corensic was impressive tech. I actually debriefed with one of their founders years ago. IIRC, their product was focused on finding single-process concurrency bugs.
Deterministic hypervisors are by no means new. Somebody once told me that VMWare used to support a deterministic emulation mode (mostly used for internal debugging). Apparently they lost the capability some time ago.
Hi Marc, thank you for the correction! We started doing it around 2010, and were not aware of any prior art. But I am not surprised to hear that others had the idea before us. I will give Al credit in the future.
There have historically been two giant adoption challenges for DST.
(1) Previously, you had to build your entire system around one of the simulation frameworks (and then not take any dependencies).
(2) It’s way too easy to fool yourself with weak search/input generation, which makes all your tests look green when actually you aren’t testing anything nontrivial.
As you say, Antithesis is trying to solve both of these problems, but they are very challenging.
I don’t know of anybody else who has a reliable way of retrofitting determinism onto arbitrary software. Facebook’s Hermit project tried to do this with a deterministic Linux userspace, but is abandoned. (We actually tried the same thing before we wrote our hypervisor, but found it didn’t work well).
A deterministic computer is a generically useful technology primitive beyond just testing. I’m sure somebody else will create one someday, or we will open-source ours.
DST was i̶n̶v̶e̶n̶t̶e̶d̶ popularized at FoundationDB a little over a decade ago, and has been quietly gathering steam ever since. If you’re interested in the technique, the FDB paper has some good info in section 4 and section 6.2: https://www.foundationdb.org/files/fdb-paper.pdf
(Disclosure: I am an author.)
I also gave a talk overview of it at Strange Loop back in 2015, but don’t have the youtube link handy.
If you’re interested in trying DST, we have a new company that aims to make it a much easier lift, especially for existing projects that can’t be retrofitted onto one of the many simulation frameworks: https://antithesis.com
Happy to answer any questions about the approach, here or over email.
My guess is rather that he's conflating the US with US + Western Europe.
No, we don’t require any paravirtualization at all, and nothing needs to be manually rewritten. I’m not sure where you got that impression.
It also is not in any sense a replay engine. We don’t need to record anything except the inputs!
This makes the mapping "injective": https://antithesis.com/blog/deterministic_hypervisor/
The "onto" direction doesn't really matter.
Yes! See Alex's talk here: https://www.youtube.com/watch?v=0E6GBg13P60
In fact, we just made a radical upgrade to this functionality. Expect a blog post about that soon.
Any third party service does need to be mocked or stubbed out. We have a partnership with Localstack that lets us provide very polished AWS mocks that require zero configuration on your part (https://antithesis.com/docs/using_antithesis/environment.htm...).
If you need something else, reach out and ask us about it, because we have a few of them in the pipeline.
You're right that if you tried to do something like this using record/replay, you would pay an enormous cost. Antithesis does not use record/replay, but rather a deterministic hypervisor (https://antithesis.com/blog/deterministic_hypervisor/). So all we have to remember is the set of inputs/changes to entropy that got us somewhere, not the result of every system operation.
Yes, unfortunately we have not figured out how to rewind time in the real world yet. When we do, there are a lot of choices I'm going to revisit...
The simulation is a completely generic Linux system, so we can run anything (including NodeJS). If your build tool can produce Docker containers, then it will work with us.
We don't run this on your production server, but in the same simulation that we use to find your bugs. See also: https://antithesis.com/product/how_does_antithesis_work/
Good suggestion on leaving LD_PRELOAD unset if it was previously unset. We will fix that.
I’m torn on whether MD_PRELOAD or LD_PRELOAF is more obnoxious to other programs.
Fun fact: A previous version of this program used an even more inscrutable `env[0][0]+=1`, which is great as a sort of multilingual C/English pun, but terrible in the way that all “clever” code is terrible.
Great point! I'll update the post to mention that.
Biggest advantages I know of for dynamic linking:
* You can use the LD_PRELOAD trick to override behavior at runtime.
* You can run with entirely different implementations of the dynamically linked library in different places.
* Software can pick up interface-compatible upgrades to its dependencies without being re-compiled and distributed again.
We use all three of these tricks in our SDKs, FWIW. But it is still a giant pain in the ass.
Our artist is on vacation, and some fool gave the CEO access to Midjourney.