HN user
It depends how much the code uses manually spawned goroutines, and how complex the lifecycle of these goroutines is… in big codebases such as kubernetes, docker, etc, it has been a problem. There have been research papers and blogs about this, but most Go developers are not aware of this issue it seems.
Interesting to see an ebpf approach to this idea, some time ago I did the same with Dtrace: https://gaultier.github.io/blog/detecting_goroutine_leaks_wi...
However I did not think of observing the ‘castgstatus’ runtime function.
Always nice to be able to compare the DTrace and Ebpf approaches, they both have different strong points.
The name clashes with a similar project that has existed for decades: https://zeroc.com/ice
I am not an expert in incident reaction, but I thought the safe way was to image the affected machine, turn it off, take a clean machine, boot a clean OS image with the affected image mounted read only in a VM, and do the investigation like that ?
Assume that the malware has replaced system commands, possibly used a kernel vulnerability to lie to you to hide its presence, so do not do anything in the infected system directly ?
Nice article, thank you. Did you also consider using bpftrace while debugging?
I do not have much experience with it, but I think you can see the kernel call stack with it and I know you can also see the return value (in eax). That would be less effort than qemu + gdb + disabling kernel aslr, etc.
Author here, thanks for the feedback on legibility, I have now just learned about the CSS `tab-size` property to control how much space tabs get rendered with. I have reduced it, should be better now.
I love a good SIMD article!
Just one point about testable assembly: I like the Golang guidelines which require to write a Go version before implementing the same in assembly. That way there is a baseline to compare with in terms of correctness and performance. Also the Go project has recently introduced mutation testing and test coverage on assembly code which is very interesting.
Finally about detecting at runtime the presence of simd: I am confused why a third-party dependency or even the std is needed. At least on x64, it’s just one instruction (“cpuid”) and then checking if a bit is set, right?
Your are right about shadowing: I just checked and the go specification calls it “redeclaration” and it works as you described:
Redeclaration does not introduce a new variable; it just assigns a new value to the original
I added a mention about that in the article. Thank you.
I agree with the premise, just use a specific version of your dependencies, that’s generally fine.
However: You absolutely do need a lock file to store a cryptographic hash of each dependency to ensure that what is fetched has not been tampered with. And users are definitely not typing a hash when adding a new dependency to package.json or Cargo.toml.
I wrote a toy Kotlin compiler, for fun. Then one day a Jetbrains employee opens an issue which only says: “Why? Just why?”. Maybe it’s the language barrier… but I did not find that particularly polite.
On the other hand I open sourced my blog and received lots of small contributions to fix typos or such which were nice.
Miri and Valgrind will usually catch this kind of issue. I did lots of work mixing C and Rust and that tripped me as well at the beginning. I wrote about it if someone is interested: https://gaultier.github.io/blog/perhaps_rust_needs_defer.htm...
I wrote a blog article about exactly this, using a C++ class from C++, C and Rust: https://gaultier.github.io/blog/rust_c++_interop_trick.html
I have implemented polling against a cluster of mixed mariadb/mysql databases which do not offer listen/notify. It was a pain in the neck to get right.
- The batch size needs to be adaptative for performance, latency, and recovering smoothly after downtime.
- The polling timeouts, frequency etc the same.
- You need to avoid hysteresis.
- You want to be super careful about not disturbing the main application by placing heavy load on the database or accidentally locking tables/rows
- You likely want multiple distributed workers in case of a network partition to keep handling events
It’s hard to get right especially when the databases at the time did not support SKIP LOCKED.
In retrospect I wish I had listened to the WAL. Much easier.
The x64 assembly would probably work natively on the Mac, no need for docker, provided the 2 syscall numbers (write and exit) are adjusted. Which llms can likely do.
If it’s an ARM Mac, under Rosetta. Otherwise directly.
I low key want to buy t-shirts of these now.
About not having perf on macOS: you can get quite far with dtrace for profiling. That’s what the original flame graph script in Perl mentions using and what the flame graph Rust reimplementation also uses. It does not have some metrics like cache misses or micro instructions retired but still it can be very useful.
Reading the article I thought exactly the same! I‘d be curious to know how much time the same would take with zfs.
I have worked for financial services companies that used FreeBSD both in EC2 and on the metal in data centers (self managed). The two features we used all the time were zfs and jails. Each service ran in its own jail for isolation. One (not even beefy) server could run all the services which was insanely cost efficient. A cloud migration was undertaken at some point to have a hybrid setup, using a mix of Linux (k8s) and FreeBSD, and costs skyrocketed. It’s a trade off because in the data center we had to buy and replace our own disks, react to fires taking place, being only in one country etc. AWS gives you multi region, and tons of good stuff, and that has a price.
ZFS was not leveraged that much but it saved our beacon once when a table in the production database was accidentally dropped and we could instantly rollback to the previous zfs snapshot (there was a tiny bit of data loss as a result but this did not matter too much for this application - uptime was more important). ZFS was also used for backups I believe.
A few times I used dtrace in production to troubleshoot.
When we introduced Linux to our fleet of FreeBSD servers, every team picked a different distro organically so it was a bit of a zoo. With FreeBSD on the server you only have the one variant.
I still use and like both, but I must say I really like that FreeBSD is a kernel+OS integrated together.
I can tell you some financial services I have worked for do use FreeBSD on EC2 as well as on the metal in data centers to do millions of transactions a month. I like the OS, thanks for your work.
Good article, I also have a C program (a compiler) that I would like to compile to webassembly to offer a playground web page, so that is good information. Thank you!
About the file system stuff: modern browsers ship with SQLite which is available to JavaScript (is it available to webassembly? No idea) so I would probably use that instead. Ideally you could use the sqlite API directly in C and emscripten would bridge the calls to the browser SQLite db. Something to investigate.
That reminds me of a blog article I wrote some time ago, where “timeout” gets mentioned: https://gaultier.github.io/blog/way_too_many_ways_to_wait_fo...
It’s more useful if you are implementing this in a general programming language, not in the shell, or if you want to know how it works under the hood.
Dtrace can do the same and much more I believe with destructive actions, and it is supported on Windows.
I initially did not see the use case for sending hundreds of MiB to the browser, but near the end, data visualization is mentioned which is fair.
I could also see video games and observability platforms needing to do this, for example Datadog, which must send lots of stack traces with long function names along with numerical data.
Glibc’s one does and it caused a security vulnerability: https://www.qualys.com/2024/01/30/qsort.txt
TBH it was also news to me, I discovered it randomly while browsing vulnerabilities… Printf also allocates, and a ton of other stdlib functions as well.
I have done something similar for Linux under 2 KiB in assembly some time ago: https://gaultier.github.io/blog/x11_x64.html
As others have said, doing so in pure C and linking dynamically, you can easily remain under 20 KiB, at least on Linux, but Windows should be even simpler since it ships with much more out of the box as part of the OS.
In any event, I salute the effort! You can try the linking options I mentioned at the end of my article, it should help getting the size down.
Fantastic article! But when I read “inspired by the Gameboy form factor”, I thought the cartridge would insert much deeper, like a Gameboy (maybe not all the way so that the picture on the cartridge is still visible) . That way it cannot be accidentally yanked out. It’s probably a dust trap but it’s more robust and compact.
I love reading optimization stories so thank you!
So essentially the code got faster by switching to smarter regexp engine that can transform (hopefully) the regexp into a FSM. Cool, but what about replacing the regexp with straightforward parsing code written manually? That way it is statically known that this function is fast all of the time.
At least that’s what I would do in a natively compiled language with access to simd, not sure if ruby has access to it and can detect the best variant at runtime.
And I’d argue plain code is simpler to understand, debug, troubleshoot than a complex regexp.
The regexp engine part reminds me of writing scalar code, relying on the compiler doing autovectorization to make it fast. And what very often happens is that the new compiler version changed their heuristics and now there is no autovectorization that kicks in and the performance falls off a cliff . It’s a risky bet.
Thank you, I will definitely have a look, and update the article if there’s any interesting finding
Yes, that's the obvious (and boring!) answer, that I mention in the introduction and that's in a way the implicit conclusion. But that does not teach us SIMD then :)
At work the Go backend was also using gob encoding in one case to store some values in the database. After some profiling, I noticed it was the top allocation part of the whole app. It was also showing on the CPU performance profiles. Finally some very light “fuzzing” (just a case were the wrong value was written to the database, and reading it back crashed the app) made me decide to rip it out and use JSON instead. Instantly, this section of the code disappeared from all profiles. I honestly do not understand why this format exists in the Go standard library and I would never use it. There are plenty other formats available…and a decoder should never crash in the face of invalid data!