Intel X86 Encoder Decoder

Probably not (but who knows, you could probably use this as part of a fuzzer). Instruction emulation is a superset of instruction decoding. You need to decode and then emulate the behavior.

Aside: Instruction emulation is pretty finicky and bug prone. I'm not too familiar with Xen, but KVM has had at least 10 instruction emulation CVEs. There were talks at both KVM Forum and Xen Summit last summer mentioning the sketchiness of instruction emulation.

majke• 10 years ago

This is awesome. I thought it's closed source. I wonder how close it is to the hardware.

Can I assume that whatever is parsed by XED is going to be parsed in the same way by real CPU's?

mjcharney3• 10 years ago

XED author here...

That's one goal certainly. Not claiming it is perfect. I have been working on it a long time and many cycles have been run on it between Pin, Intel SDE and some internal simulators that boot OSes, etc.

dreamcompiler2• 10 years ago

"As new instructions are made public"? WTF does that mean? We're not even allowed to know the full instruction set of an X86 CPU?

mmastrac• 10 years ago

Read this as "as upcoming instructions in new chips are documented"

wyldfire• 10 years ago

Yes, that's likely what the author meant. But, also, the author might have access to development specs or early samples not publicly announced yet.

user5994461• 10 years ago

There is a many thousands pages instructions manual for that, that is freely downloadable from Intel. It's re-posted on HN from time to time.

lobster_johnson2• 10 years ago

Speaking of which... I'd really love to have a reference manual on one of the early x86 processors. Maybe the 386, since it is one of the iconic ones (or the one I have the most nostalgia for) that also started to get interesting features like protected mode, but still wasn't so complicated as to be a multi-volume, 5,000-page series (maybe?).

Back in the 1990s, when I was doing assembly language on the Amiga, Motorola sent me the official 68000 manual for free when I called and asked them. It was a really cool book to have on the shelf and occasionally lead through.

I looked at abebooks.com, and there are Intel books, but I wouldn't know exactly which one would be the reference manual. Anyone got an ISBN?

SolarNet• 10 years ago

This is (partially) meant to be used for chips that don't exist. As future instructions are made public software vendors (like compiler vendors) can use this to be ready when the chips come out day 1.

Sephr3• 10 years ago

Don't expect any management engine/system management mode instructions any time soon/ever.

derefr3• 10 years ago

No reason to hide them; presumably, as with firmware updates, you have to load an Intel signing key into a few registers along with your IME/SMM opcode to get it to execute.

amluto2• 10 years ago

Huh? SMM is fully documented at the CPU level. QEMU can even emulate it. Not sure what you mean by "management mode".

dbcurtis2• 10 years ago

Not allowed yet.

Ex-Intel here. It takes years to gestate new instructions. First specs are in controlled documents available to Intel employees only. Eventually, when things are nailed down, preliminary specs are available under an NDA where a VP approves/signs the Intel side of the NDA. Tool vendors will get those specs. Finally, when the chip is announced the previously NDA documents become public. I was a CPU designer at multiple companies, and everyone follows a similar process.

derefr3• 10 years ago

You're just talking about the design phase of a new ISA extension. I think the parent was under the impression that there were instructions in released processors that were undocumented.

CalChris• 10 years ago

previously NDA documents become public

I'd say some are eventually made public. I went to the IDF in San Francisco expecting this sort of information on Skylake. What a waste of a day.

Not that I'm complaining. Intel's information is excellent. It just arrives when it arrives and it is what it is at that point in time. There are many ways involving effort which glean more information: Agner Fog, articles, patents, .... BTW, Intel folks are helpful on their dev board.

yuhong• 10 years ago

The story of FXSAVE/SSE (I wonder if you were there for it):

http://web.archive.org/web/20000817193452/http://www.tbcnet....

http://web.archive.org/web/20000817084210/http://www.tbcnet....

Obviously, XSAVE/AVX was handled better.

celdon25• 10 years ago

What about the infamous i386 LOADALL instruction? Certainly there is at least one that is kept secret.

http://www.rcollins.org/articles/loadall/tspec_a3_doc.html

mjcharney3• 10 years ago

XED author here...

@dbcurtis: Yes, that's what I meant. Thanks for clarifying.

thecompilr• 10 years ago

About time they open sourced it. Funny how it is Python according to github :)

mjcharney3• 10 years ago

Python is used to generate C code from tables. Also use python to build/package/test.

chubot• 10 years ago

Why is that funny? Python seems like the perfect language for this kind of thing. You can see the logic with minimal noise.

unwind• 10 years ago

It's funny since it's not very accurate: the Python is used to generate C code from tables describing instruction encodings (that's my understanding, haven't cloned this and built it). But there is plenty of actual C code in there too, and the resulting "thing" is a C library.

Python would not be as suitable as the full implementation language, since C is much more easily accessed from other languages, which is a good property for a library like this.

derefr3• 10 years ago

Seems like this would make one able to write the object-code codegen phase of a compiler at a rather higher level (at least if you're only targeting x86, or are willing to write similar libraries for your other arch targets.)

Or, to put it another way: looks like a good library for adding "just a bit of JIT" to an interpreter, without going full LLVM.

dsfyu404ed• 10 years ago

Something that is universal is not remarkable. People will not make as much use of an unremarkable opportunity as they will of a remarkable one.

struct• 10 years ago

How does this compare to ARM's VIXL?

[1] https://github.com/armvixl/vixl

Intel X86 Encoder Decoder

Related Stories