x86 assembly doesn’t have to be scary (2018)

https://blog.benjojo.co.uk/post/interactive-x86-bootloader-tutorial

by aphrax • 5 years ago

133 109 5 years ago

blog.benjojo.co.uk

This webpage hijacks the keyboard in a very nasty way, disabling not only scrolling, but also a bunch of other functions.

heinrich5991• 5 years ago

Up/Down/PgUp/PgDown/CTRL-F all work for me. What's disabled for you?

andrewshadura2• 5 years ago

All of those. Also F12 doesn't work, so I have to use the mouse to inspect the internals.

benjojo12• 5 years ago

Really? (Author here)

What browser are you using?

erik-smit• 5 years ago

I have similar issue with Chrome 94 on Windows.

It's fine when I initially open the page, but once I click 'run' I can no longer use PgUp, PgDn, F12, etc.

I'm guessing the emulator captures some input. But even when clicking on other parts of the page, it doesn't 'uncapture'.

FWIW, it doesn't really bother me.

p_l• 5 years ago

Happens to me as well (current chrome w/ vimium on mac), but only after running the example in the emulator.

Before hitting the emulator (and/or editor, not sure) all keys work fine. After, well, only scroll on the mouse worked for moving through the page.

badrabbit6• 5 years ago

I feel like X64 doesn't get enough love these days. I think new tutorials should be X64 first and then after you get a hang of it talk about X32. It's harder the other way around, especially when it comes to calling conventions.

coopierez2• 5 years ago

On the contrary, I felt that understanding where x86 had come from helped me understand the reason why the registers were named what they were, why some instructions are longer than others, etc.

It's confusing as a beginner to encounter the "di" and "dx" registers, and to understand why the 8-bit version of "r13" is "r13b" but the 8-bit version of "rdx" is either "dh" or "dl".

sabas1232• 5 years ago

I think if you start from a point where you need to know all the registers beforehand you already lost.

I like x64 a lot more because IMO it allows you a few important memory related ones in x86 making life a lot simpler.

badrabbit6• 5 years ago

Wouldn't it be better if you knew about all the current registers first then learn about older types? I learned like most people X32 first and I feel like I still read X64 relative to X32.

mettamage• 5 years ago

I started with x86, after that x64 felt simply like a different flavor. There was more Googling to do, but I did that a ton already for x86, so I was used to the type of game I was playing.

mike256• 5 years ago

I think it's much easier to begin with x86. Especially when the assembled code is being run the way it is on this website. x64 is a little harder because just the switch to x64 mode alone almost fills the boot sector.

Joker_vD6• 5 years ago

And then you lose the ability to temporarily switch to the real mode to call BIOS functions so you better load the rest of your program before switching to x64.

Ygg22• 5 years ago

Umm, did anyone read the reference manual for x86_64? Intel's big enough to kill a goat (~2000). Add AMD on that and you can kill a small tribe of goats.

sabas1232• 5 years ago

Depending on what you're after there is a lot more than 2000 pages of documentation :)

That said, the C standard is also an impenetrable mess, yet people can start using it without resorting to it.

badrabbit6• 5 years ago

There are not that many compiler generated instructions both for X32 or X64 if you look at it by percentage of code. You have to look up the manual for rare instructions no matter what, you hardly need to read maybe a dozen pages to know enough to understand the common operations and control flow. My argument if you start with the current arch, that will be your frame of reference instead of X32 (why not X16 if earlier is better?)

HelloNurse• 5 years ago

New tutorials? Don't they all date from when 32 bit registers were new, 16 bit registers mainstream, and MMX and x87 the new complicated instruction set additions?

It would be certainly a smaller tutorial, but historical perspective might be valuable for understanding cruft layers of design decisions that made sense decades ago and addressed problems that have ceased to exist.

raxxorrax• 5 years ago

You also need to define the system when talking about assembly, I think this is more relevant than the difference between architecture variants. I would recommend Linux because available documentation. Without doing any system specific syscalls and learning about calling conventions you won't get very far.

https://github.com/torvalds/linux/blob/master/arch/x86/entry... (subject to change, see your specific kernel source)

I think more fun is doing assembly on a µC without OS with some interesting peripherals and many MCUs have an instruction set inspired by x86.

badrabbit6• 5 years ago

I think it depends on your goals. I am with you for writing assembly. For reading compiler generated coder however, I think windows is ideal as a starting point because you don't often get spoiled by the availability if source code and debugging symbols.

pengaru2• 5 years ago

x86's purely stack-oriented C ABI calling convention makes things far less tedious. In x86_64 sure it executes faster with the register passing, but what a miserable thing to read/write as a human.

Maybe this is just a case of having learned and written 32-bit x86 assembler for years in my youth, but I strongly prefer it to x86_64.

badrabbit6• 5 years ago

But would you still prefer X32 if you started with X64? My argument is that some would not find it tediuous if that was the first thing they learned. Similar to how a python programmer would share your sentiment about C because he learned python first. I learned C first so while I accept that it is more verbose and more work in general, I enjoy writing C more.

inkyoto3• 5 years ago

C stack-oriented ABI calling conventions also started with PDP-11, for which the first portable C compiler was written, and the convention then tagged along to nearly every other architecture as the pcc was progressively ported to other platforms.

If a function return value could fit into %r0 on the PDP-11 (%ax/%eax/%rax on x86), it would be returned in there. %r1-%r4 would be used to pass function parameters in, if they could fit, and/or spill over into the stack.

Heck, even UNIX system call conventions on x86 can be traced back to PDP-11, i.e. the syscall number is passed in %r0 (%eax) followed by a TRAP (INT on x86) instruction (can't remember which TRAP number, though).

begueradj• 5 years ago

I learned x86 at university. I miss it.

Koshkin4• 5 years ago

You don't have to: Windows programming using NASM is easy:)

  ; To build:
  ;   nasm -fobj hello.asm
  ;   alink -oPE -subsys con hello -entry main

  bits 32

  import imp_CreateFileW      kernel32.dll    CreateFileW
  import imp_WriteConsoleW    kernel32.dll    WriteConsoleW
  import imp_CloseHandle      kernel32.dll    CloseHandle
  import imp_ExitProcess      kernel32.dll    ExitProcess
  import imp_GetLastError     kernel32.dll    GetLastError

  extern imp_CreateFileW
  extern imp_WriteConsoleW
  extern imp_CloseHandle
  extern imp_ExitProcess
  extern imp_GetLastError

  %define CreateFile      [imp_CreateFileW]
  %define WriteConsole    [imp_WriteConsoleW]
  %define CloseHandle     [imp_CloseHandle]
  %define ExitProcess     [imp_ExitProcess]
  %define GetLastError    [imp_GetLastError]

  global main

  section .text use32
    
  main:

    push dword 0 ; hTemplateFile = 0
    push dword 0 ; dwFlagsAndAttributes = 0
    push dword 3 ; dwCreationDisposition = OPEN_EXISTING
    push dword 0 ; lpSecurityAttributes = NULL
    push dword 2 ; dwShareMode = FILE_SHARE_WRITE
    push dword 0x40000000 ; dwDesiredAccess = GENERIC_WRITE
    push dword filename 
    call CreateFile
    mov [handle], eax
    
    push dword 0 ; lpReserved = NULL
    push dword nChars ; lpNumberOfCharsWritten
    push dword 16; nNumberOfCharsToWrite
    push dword hello ; lpBuffer
    push eax ; hFile
    call WriteConsole

    mov eax, [handle]
    push eax
    call CloseHandle
    
    push dword 0
    call ExitProcess
    
  err:

    call GetLastError
    push eax
    call ExitProcess

  section .data use32

  filename:   db      __utf16__('CONOUT$'), 0, 0
  hello:      db      __utf16__('Hello Console!'), 13, 0, 10, 0, 0, 0
  handle:     resd    1
  nChars:     resd    1

wiz21c2• 5 years ago

Long time ago I coded much of my professional stuff in assembly, I was hired specifically to optimize stuff. But it was 20 years ago, compilers were not very smart.

How smart are compilers these days ? Say, to optimize small function, for example computing a scalar product or applying a 3D matrix transformation to a set of points.

thegeomaster• 5 years ago

Pretty smart. The examples you gave are math-heavy, so to get the best performance you need to do use some kind of SIMD instructions. For these you need to drop down a level, although not really assembly - there are compiler intrinsics that you can use. And for simple functions, compilers are getting fairly good at autovectorization, meaning to introduce SIMD instructions automatically. But it's not something you can rely on.

Generally, they do lots of inlining, and then once you inline you can get some more optimizations in, rinse and repeat. Ends up pretty optimal. (This is C++, can't speak for other languages.)

We work on very perf-sensitive code and we never drop down to assembly. For hot loops, we usually inspect the generated assembly and if it's not great, it's fairly easy to "nudge" the compiler towards the better-performing solutions by tweaking the source code. Also some manual unrolling might be needed to better saturate the vector processing cores of modern CPUs.

And when you're working with signed integers, you still have to do stuff like a >> 1 instead of a / 2 :)

wiz21c2• 5 years ago

If it was on StackOverflow, I would give it a "correct answer" flag :-)

But if it's now up to some nudging, it's vastly better to me.

I provided math stuff and vectorisable stuff on purpose :-) Happy to see that vectorisation is somewhat automatic. I remember the MMX days and there were not that funny :-)

flohofwoe4• 5 years ago

IME: "it depends". You'll have to check the generated assembly code and then tweak your "high-level" C code to appease the compiler to generate code that's acceptable. Sometimes a small change in the high-level code is enough to break the "pattern matching" in optimizer passes.

Here's an example that looks like magic at first glance where the compiler converts manual bit twiddling code to a popcnt instruction (with the right compiler setting), but do the bit counting any other way, and the whole thing falls apart:

https://www.godbolt.org/z/KaM6jWjdx

orbifold• 5 years ago

GCC and ICC typically do a reasonable job, with the right flags: https://godbolt.org/z/6a8no4n5a.

BeeOnRope2• 5 years ago

Someone well-versed in assembly can still do much better than the compiler in many cases over reasonably small function. You could of course do better for large functions too, but at some point the cost becomes prohibitive: if you're going to do this, stick to a few hotspots which are as compact as possible.

flohofwoe4• 5 years ago

I sometimes wonder whether assembly code would even be considered "scary" today if IBM had picked the Motorola 68000 instead of the Intel 8088 for their PC.

The x86 instruction set was a cobbled-together mess from day one, while 68k assembly coding was pure joy because of its elegant and consistent instruction set.

mjbrusso• 5 years ago

I agree. 68k was one of the most orthogonal architectures ever.

inkyoto3• 5 years ago

After the PDP-11 architecture, yes.

Taniwha• 5 years ago

well except for the two sorts of registers

123pie123• 5 years ago

totally agree. I learnt a bit of 6502/6809 in my early teens and in to college/ uni. I couldn't not be bothered with the x86 - a complete pita, although to be fair that may have been the not so great manuals I had at the time.

but I started in my first job dissassembling 68000, very easy to work with

gardaani• 5 years ago

ARM assembly code looks pretty nice. Mobile phones and Macs are using ARM, so there's lots of devices using it. Only Windows is using x86/x64.

ChuckNorris892• 5 years ago

Only Windows is using x86/x64

Yeah, X64 is only used by windows, that's like ~80% of the desktop computing market, plus nearly every server and cloud instance out there, so not much at all. /s

Why do some users assume that the whole world revolves around Apple's iOS/M1 Mac ecosystem as if it exists in a vacuum?

cbm-vic-20• 5 years ago

RISCV also has a nice assembly language.

bodhiandpysics1• 5 years ago

ARM assembly is easy to learn, because of its relatively small and orthogonal instruction set. However, load store architectures are annoying to program in, since you're constantly having to juggle memory. The lack of a convenient way to spill registers is also really really annoying! (you can only spill and load registers two at a time)

michaelcampbell2• 5 years ago

As an owner of the original 1984 Mac, I bought a copy of the classic Lance Levanthal 68000 assembly book. I never got much into actually coding in it, but I remember how "this makes sense, mostly" the book felt.

Then I read some of the Intel stuff. OMG.

pjc502• 5 years ago

It's "scary" on almost every architecture because there's no fault tolerance, recovery, exception handling, or reporting. If you make a mistake you get to reassemble the smashed plate of your memory image in order to work out what happened - assuming you can get at it at all. Some platforms will just reboot on you.

flohofwoe4• 5 years ago

Hmm, I don't agree, stepping through assembly code in most (C/C++) debuggers works just as well as with high level languages, and process isolation in operating systems also works for assembly coding, so it's just as unlikely to crash the whole computer with assembly code as with high level languages (otherwise we'd have a massive problem if operating system security would just depend on banning low level programming).

Bare metal embedded coding is a different topic of course, but regular application development with assembly works just fine.

ajross3• 5 years ago

68k assembly coding was pure joy because of its elegant and consistent instruction set

That's maybe a little spun. The separated address/data registers (there are two kinds of registers, and memory operations need to use one from each to combine to a final address) played hell with optimizer strategies for years.

In fact 68k compiler output was significantly sub-par pretty much throughout its lifetime. The "cobbled-together mess" had (post-386 anyway) a significantly more orthogonal instruction set and was just plain easier to optimize, even for humans.

notriskfree• 5 years ago

The 68k was certainly great fun after 8bit systems, and the x86 was almost just more of the same. ARM32 is also nice, apart from some corner cases. I think everything has corner cases, apart from x86 where the whole thing is an ugly mess.

Banjo9112• 5 years ago

No, on the 68000 you can do memory addressing using none, one address register, two address registers, or one of each.

On the 68020 you get even more flexibility.

Koshkin4• 5 years ago

Ideally - ideally IBM would have created and used their own microprocessor based on the System 370 architecture. But no - for them the PC was a glorified typewriter: even PS/2 (based on 80286) was mostly positioned for use merely as a "smart" terminal for mainframes. So, today the entire world is basically run on faster "typewriters" (sometimes enhanced to include "windows" - even on the server side).

agumonkey• 5 years ago

Somehow seeing prolog, APL, sml code made me think in terms of small combinators that felt a lot like assembly.

DonHopkins• 5 years ago

x86 assembly doesn’t have to be scary

...just ugly.

Joker_vD6• 5 years ago

It doesn't matter that much, really. What truly is ugly is interfacing with the rest of the computer, ugh. You can't use the VGA BIOS from x64, so go-o-o-od luck doing it via PCI. And properly setting up IOAPIC?

eatonphil• 5 years ago

I know this article is about bootloaders but more generally there are two ways I've tackled x86 as a total noob: compile a simple language to like ~5 instructions [0, 1] and, separately, write an emulator for ~5 instructions that can run basic C programs [2, 3].

I think it's fairly common to recommend beginners to write a compiler but I think it's less common to actually recommend trying to emulate parts of x86. I think it's a particularly easy way to get started just because it's the architecture you already know or because its the architecture that all compiler tutorials use (if they don't use LLVM). And if you are a programmer you probably incidentally have gcc, gdb, and objdump on your system ready to help you out.

Doing both compilers _and_ emulators really helped my understanding of x86 and C (even if I wasn't writing C).

My background is in web development and my reason for doing these projects/writing is purely educational.

[0] https://notes.eatonphil.com/compiler-basics-lisp-to-assembly...

[1] https://notes.eatonphil.com/compiler-basics-an-x86-upgrade.h...

[2] https://notes.eatonphil.com/emulating-amd64-starting-with-el...

[3] https://notes.eatonphil.com/emulator-basics-a-stack-and-regi...

secondcoming2• 5 years ago

Compiler Explorer [0] is a godsend for seeing what the compiler is generating.

For example, the codegen for some C++ that was discussed a few days ago [1]

[0] https://godbolt.org/

[1] https://godbolt.org/z/cfYKqajMd

BeeOnRope2• 5 years ago

Looks interesting, can you share what post it was discussed in?

nvarsj• 5 years ago

Doing CTFs is also a way to get intimately familiar with x86 assembly, and they are pretty fun too.

iota8• 5 years ago

Here is an article that explains how C code is translated into assembly: https://www.avabodh.com/cin/cin.html

This website also has another article which explains how a C++ code can be understood in terms of C code. https://www.avabodh.com/cxxin/cxx.html

rajeevk• 5 years ago

Here is my article that explains how C code can be understood in terms of assembly: https://www.avabodh.com/cin/cin.html

I also wrote another article which explain how C++ code can be understood in terms of C code: https://www.avabodh.com/cxxin/cxx.html

Decabytes• 5 years ago

The problem I’ve always had with learning assembly (arm in my case) was that I never really had a project to work on that would use any of that knowledge

ThomasBHickey• 5 years ago

WASM is a nice 'assembly' language to play with.

not-elite• 5 years ago

Same here, mostly doing math stuff on x86.

There was a recent thread [1] optimizing some very primitive trig functions.

I recently watched a djb interview where he talked about the importance of fully utilizing available hardware [2 at 5:15]. That can be a good starting point, although at work its usually easier to just consume more resources than to use what you've got more efficiently.

[1] https://news.ycombinator.com/item?id=28209097

[2] https://www.youtube.com/watch?v=1svxNxG6hHc

JoeAltmaier2• 5 years ago

You resort to assembler for critical operations or performance. Having a nice, clean orthogonal instruction set helps none of that.

So while 68000 ASM may look pretty and be easy to remember, its obvious why x86 won - you could get the critical jobs done.

tenebrisalietum• 5 years ago

I do know x86 for example has things like the STOS/STOSB/STOSW/STOSD which specifies a lot of behavior for a single instruction, and combined with the REP prefixes is a pretty elegant way to do memory block operations. I don't think 68000 had anything like that.

JoeAltmaier2• 5 years ago

To be fair, Intel nerfed those instructions for a long time. See, DOS was using REPNZ STOSW for timing things. So as the processors got faster, those instructions didn't, so as to make DOS BIOS continue to operate correctly. It was a terrible time for x86.

At the same time, I used a clever 12-byte sequence of those opcodes to swap software interrupt vectors on process switch. So each process could have its own floating-point exception handlers etc. We called it the 'soft vector chain' and it was among the least known parts of our kernel.

A colleague (John McGinty) suggested in our old age we could consult by scratching our beards and saying "Ah! It must be the soft vector chain!" for every problem.

inkyoto3• 5 years ago

its obvious why x86 won - you could get the critical jobs done.

It is a bit less obvious, in fact.

Motorola never looked at 68k CPU's as a serious business, more like toying around with the CPU's all the time or looking at them as a less important spin-off of the main business. Their main sources of income were defence contracts (e.g. specialised or hardened microchips), microcontrollers, DSP's (which were pretty cool, by the way, – all implementing the Harvard architecture), memory chips (I think), radios and the field radio equipment and later mobiles.

They carried largely the same attitude to 88k RISC and PowerPC CPU lines (albeit trying to compete more seriously for a while), but ultimately failing to catch up and leading to the eventual PowerPC demise. After that failure, they spun off anything CPU, DSP and microcontroller related into Freescale, and the rest is now history.

HappyMans2• 5 years ago

I am fascinated with bootloaders and kernel writing. I am not very good at it, but I am fascinated by it, and every so often I try to learn some more. It feels a bit like a useless skill to learn (legacy BIOS bootloaders, that is) given UEFI dominance. But it connects me with my childhood playing with 286es and wondering how to program them.

I love articles like this that break it down. The biggest challenges so far have been getting the assembler to output the correct format (ie, 16 bit real mode) and learning inline assembler in C (and getting GCC to output the correct format).

kingcharles3• 5 years ago

This is the book I learned from in the 90s: https://www.amazon.com/Developing-32-Bit-Operating-System-Cd...

It's basically "Writing an Operating System from Scratch for Dummies".

I actually wrote my own graphical x86 OS starting from the code in there.

HappyMans2• 5 years ago

Thank you so much for the recommendation. Been looking for something like this for a while.

Do you think it's a waste of my time to be learning this in 2021?

7thaccount4• 5 years ago

Super dumb question (haven't taken a microprocessors class in over a decade):

How hard would it really be to custom build a chip that was really simple, but modern. I'm thinking like the 6502 in the Commodore, but much faster. Or is the complexity in x86 an inherent property of modern performance? I guess what I'm getting at, is could you build something that is actually pretty darn fast if you don't need to run a modern OS like windows or Linux on it, but keep it drastically simple?

I've been thinking it might be neat to have a blazing fast NUC sized computer that just boots into some barebones forth sitting on top of a few assembly words. Maybe with just enough peripherals to do some actual work (like load from SD card).

ampdepolymerase• 5 years ago

It's called a FPGA; a software defined microprocessor (most higher tier processors can be reprogram to a certain extent through microcode, but a FPGA is a specialised chip designed specifically for this). You can get ridiculously high performance for your specific application, but it will never excel at general purpose computing. Good enough for Forth, great for ML, DSP, and number crunching, bad for running random apps from GitHub.

7thaccount4• 5 years ago

I've used FPGAs before (also ages ago). I guess I could create a simple chip on an FPGA and run a Forth on that, but I was thinking something more physical.

Koshkin4• 5 years ago

I find it kind of funny that majority of uses of FPGA is indeed to implement a simple CPU (basically a programmable state machine), and the problem at hand is then solved by writing a program for it.

sgtnoodle2• 5 years ago

There's inherent complexity in the x86 instruction decoder, but after decoding, modern x86 CPUs don't particularly resemble legacy x86 CPUs internally. The native machine code is all proprietary "micro ops".

A lot of the performance gains over the last couple decades haven't come from machine code changes, but rather from various forms of pipelining and superscaling. Rather than run one instruction at a time, CPUs run hundreds of instructions at a time. The complexity of doing that is that many instructions often depend on the result of instructions immediately before them, and so the CPU needs a lot of shortcut paths internally to keep from stalling out.

You could get to Ghz speeds with a custom ASIC, and you could use base RISC-V as a modern, not crufty ISA. It still won't be anywhere near as fast as a high end x86 or ARM CPU, though, unless it's similarly pipelined and superscaled.

7thaccount4• 5 years ago

So, it sounds like there likely is unavoidable complexity if you want the screaming performance we're used to.

ip26• 5 years ago

The convergence of RISC & CISC has shown that both things are true. A really simple ISA with a modern design can be pretty darn fast. At the same time, additional ISA complexity helps you wring the utmost out of your hardware, and IMO also helps in the bazaar environment - macro ops let a menagerie of hardware do work in the way most efficient for itself without hardware-specific binaries.

Underneath the covers, however, the underlying hardware complexity is an inherent property of modern performance. For example, branch prediction.

pajko• 5 years ago

Now let's talk about SIMD...

Koshkin4• 5 years ago

As my professor told once, "The only difference between programming in assembler and a high-level language is that you have to type more." I can confirm - early in my career I was able to sustainably produce more than a thousand lines of working x86 assembly code a day.

x86 assembly doesn’t have to be scary (2018)

Related Stories