imtomt

For sure. My laptop has an arm64 chip, which is why this is written in arm64. If I had an intel chip, it would be written in x86_64. RISC-V is very interesting, though, and I'd love to learn more at some point.

Good luck with your RISC-V asm stuff! Hit me up if you publish any of it :)

Honestly it's easier than you'd think! All the syscall numbers are in /usr/include/asm-generic/unistd.h (on linux), and you can read the man page for any of them.

Well, if security is a major concern, definitely don't use ymawky in production! That said, I did try my best to harden it. I've fuzzed the parser extensively with afl-fuzz, and got several hours without a single hang or crash. There's no major vulns I'm aware of, but in a ~4500 SLOC assembly project, there's probably gonna be some vulnerabilities that are hiding.

You could do that, but it's not really necessary, and adds extra overhead and complexity. And as the other commenter pointed out, it wouldn't work for text file types without magic numbers. I considered reading the magic number at first, but after doing more research, I've found most web servers (nginx and apache, anyway) just match based on file extension. I figure if it's good enough for them, it's good enough for ymawky.

The language matters much less than the architecture of the web server. A server written in assembly using a fork-on-request model like ymawky is going to be much slower than a server written in C using an async event loop like nginx, because forking is very inefficient at scale. Plus a big bottleneck is the networking syscalls, rather than the code itself, so two servers with the same model written in Assembly and C would likely be roughly equal.

Thank you! Definitely, some macros would probably be super helpful. Register renaming, too, I'm sure. When I started this project I didn't even know about register renaming lol, and at this point it's so big I'd have to dig pretty deep to find 'em all. Definitely worth doing, though, I'm sure.

slowloris mititgations actually weren't too bad, just a couple syscalls to setitimer(), sigaction(), and setsockopt(). Range header parsing was awful, so was content length. I'd say all in all, probably Range headers. Just string parsing in general is pretty awful in assembly.

Thank you. My favorite reasons for projects are "just because I wanted to" and "because someone told me I shouldn't", and this one ticks both boxes, haha.

Oh wow! Your project was actually really inspiring to me, thank you for making this. I was really impressed reading through it a while back. Is it alright if I add a link to your repo in my README?

Aw, that’s too bad. Sounds like it was a really fun project.

Thanks for the tips on CGI! Definitely going to look into it more. The server-side execution of CGI scripts definitely interests me more than the CGI scripts themselves, so I’ll probably just look for some existing (simple) CGI scripts and work on building the env vars and executing them.

It would be possible, but impractical for where this project is now. In retrospect, I could have written it with more portability from the start, but since it was just a personal learning project I didn’t think too hard about it. Aside from syscall numbers all being different (which would just require changing the consts in defs.S), syscall numbers are stored in the x16 register on MacOS vs x8 on Linux, and the kernel is called with svc #0x80 on MacOS vs svc #0 on Linux. The bigger issue at this point is MacOS-specific syscalls. proc_info() doesn’t exist on Linux, so that would need to be rewritten. The sigaction struct also has different members (Linux doesn’t have sa_tramp), as well as every other struct having different offsets and sizes. getdirentries64() also doesn’t exist on Linux, it has getdents64() which is called differently and has different output as well I believe, so directory.S would need to be reworked too.

Woah! I honestly feel more intimidated writing a CGI script in assembly than I was writing a server, lol. CGI support has been on my mind for a couple weeks, but I haven't really dug into it yet. I'd love to see yours if it's hosted anywhere! Could be a great reference when I do.

Yeah, that's what I realized during this, too. You need to be much more explicit, but the way any given function works isn't fundamentally different. "strlen" will always iterate through a string searching for a NULL byte whether it's in C, Rust, Assembly, or whatever other language. I think it can feel almost more straightforward than other languages, since you're laying out exactly what the CPU needs to do, in what exact order.

That's so cool! I would love to see it if you're sharing it anywhere.

Yeah, I know MacOS syscalls aren't stable. Interesting point about Go, I hadn't heard about that. Unfortunately I'm a masochist though, and want to avoid libSystem.dylib as much as possible. The only reason I link against it at all is because MacOS requires it for executables to run, I never actually call into it. Figured I'd just update the syscall numbers if/when they change.

Honestly haven't benchmarked it, but I would imagine ymawky would be considerably slower than most fully-featured web servers. ymawky uses fork-per-connection, which is fundamentally slower than what production servers like nginx or Apache use. nginx uses event-driven IO (kqueue/epoll), which can handle thousands of concurrent connections without the overhead of forking the process on each request. Apache uses pools of threads which handle multiple connections without needing to be spawned per-request. A head-to-head against any other web server would mostly measure "fork-per-connection vs event loop/thread pools", which assembly has nothing to do with.

In a comparison between a similar fork-per-connection server written in C and this, I would imagine the throughput would be about the same, because the bottleneck in this model is fork() itself rather than the actual code. It probably matters more for binary size and startup time than requests/sec. Would be fun to actually benchmark, though.

Ahh you caught me. I just kept telling ChatGPT dot com "no, make it less efficient" and copied whatever output it gave me. jk, thank you!

Woah, that's really cool! I'm glad you did that even if you didn't need to. I honestly think everyone needs to write more assembly, because it's so much cooler.

Honestly, just reading existing assembly to get a feel for how it works, and then violently googling everything that goes wrong. The ARM Architecture Reference Manual (aka "The ARM ARM") ended up being really helpful for looking up what specific instructions do and how they're called. Another really helpful tool is writing something in C/C++, and compiling with "gcc -O1 -S file.c" to see the assembly gcc generated. It helps to mess around a lot with smaller programs in gdb or lldb.

Thank you! This is one of the nicest things I've heard in a while.

Why stop there? Next, I'm prying open a CPU and poking the transistors with a 9V battery and paperclips to make it execute what I want. Slower, but you get so much control.

Thanks for the link, bookmarking. I should note ymawky's main portability issues are unfortunately at the syscall layer rather than the asm layer. proc_info() and getdirentries64() are pretty Darwin-specific, so making it portable would require reworking that whole area rather than adjusting register/calling conventions.

I wrote it for MacOS because I don't have a Linux machine right now :( Once I get one up and running again, I'll probably work on porting this.

As for why it wouldn't run on Linux, there are some pretty big differences in the actual assembly. One pretty superficial difference is calling conventions -- MacOS uses the x16 register for syscall numbers, Linux uses x8. Calling the kernel in Mac uses "svc #0x80", in Linux it's "svc #0". That's ~120 lines that need to be replaced, but easy enough to just use sed. Syscall numbers are all different, as are the struct layouts for sigaction(), MacOS has an "sa_tramp" field that Linux doesn't have. Enforcing max processes is done here using the MacOS-specific proc_info() syscall, which can be used to get the number of children any given process has. Linux doesn't have an equivalent, so process tracking would need to be done differently. Finally, Linux has the getdents64() syscall, rather than getdirentries64(), which uses a different struct and is called differently.

I'm sure an LLM could make all those changes, but it's a pretty large codebase, so it would probably make some mistakes or miss things.

Thank you! The thing about eventually building your own meta language ends up happening all the time with bigger assembly projects. I do have a fair few quality-of-life macros too, but probably fewer than I should. I did end up needing to implement by hand what would be standard functions, things like atoi, itoa, strlen, memcpy, streqn.

Higher level languages are more convenient for 99% of things, but the directness of Assembly gives me a rush unlike any other. I didn't live through the C64/Amiga, but I was obsessed with old C64/ZX emulators growing up.

Hey, thank you! Means a lot. It's an odd sort of meditation, but is surprisingly the most almost-therapeutic project I've worked on. Something about the constraints of Assembly that really pull you into the minutiae and clears your head, maybe.

Honestly, read the main file, ymawky.S first. Then I'd read through get.S maybe, checking parse.S on an as-needed basis for parsing-related functions. delete.S or options.S are pretty short, too, so give those a read too.

Modularizing it into multiple files was easier than I expected it to be, you basically have other functions/labels in other files, and mark them as .global at the top. The Makefile compiles each file into their own .o, which you then link all together. You can "b" or "bl" to any label from any other file, as long as it's global and linked together. Same with data in .bss or .data, mark them as .global and they can be accessed from elsewhere.

Thank you! I've been obsessed with this idea for a while, finally decided to start on it, then obsessed over it for a couple weeks. I'd love to see some of your projects if you have anything similar, I'm glad I'm not the only one too! I think most programmers would benefit a lot from taking a few weeks or months to try and learn some assembly, and demystify how CPUs and compiled languages work.

That book is exactly what inspired me to make this in the first place, haha. The subtitle of the book gave me the acronym I named it.

Thanks, I appreciate it a lot! I tried to comment my code pretty heavily (~3000 lines of code, ~1000 lines of comments all together), since this was a learning project for myself in the first place. Hopefully those will be of some use. But separate in-depth documentation is definitely a good idea, I'll work on adding that. In the meantime I'm always down to answer any questions about it!

This post seems to now link to the writeup rather than the repository, sorry! The repo can be found at the top of that page, or directly here: https://github.com/imtomt/ymawky