Ask HN: How are you using LLMs for traversing decompiler output?

https://news.ycombinator.com/item?id=42573232

by mjbale116 • 2 years ago

105 45 2 years ago

news.ycombinator.com

I need to reverse a binary made years ago, and I have zero experience with cpp, so I think it would be a good experiment to get an LLM to help me in any way

carom2• 2 years ago

Binary Ninja has an AI integration called side kick, it has a free trial but I'm not sure it can be used in the free web version. [1]

In my experience, the off the shelf LLMs (e.g. ChatGPT) do a pretty poor job with assembly, they can not reason about the stack or stack frames well.

I think your job will be the same with or without AI. Figuring out the data structures and data types a function is operating on and naming variables.

What are you reverse engineering for? For example, getting a full compilable decompilation has different goals than finding vulnerabilities or patching a bug.

1. https://sidekick.binary.ninja/

th0ma5• 2 years ago

This is what I gather from reverse engineering material I've read and groups I've been around. Hidden state, hidden data structures, hidden automations all abound, and there simply isn't enough detail in the assembler itself to bridge the hardware's internal conceptualization and processes.

aidanhs• 2 years ago

Out of curiosity, what would you say the current state of the art is for full compilable decompilation? This is something I have a vague interest in but I'm not involved enough in the space to be on top of the latest and greatest tooling.

Retr0id• 2 years ago

Looking at an individual function, IDA hex-rays output is often recompilable as-is (or with minor modifications), but it won't necessarily be idiomatic, especially if you don't have symbol information.

feznyng3• 2 years ago

Echoing IDA but its pricing is a huge PITA if you’re using it in a hobbyist capacity i.e. you don’t have an employer willing to pay for it. Could opt for the home version but that’s a yearly cost and you have to use their cloud decompiler. Ghidra’s your best bet if you want something FOSS and community-driven although not as great at decompilation.

carom2• 2 years ago

Most decompilers do not strive for recompilability. [1] I believe there are (or were) some academic projects that aimed for recompilation as a core feature, but it is a hard problem.

On the commercial side, IDA / HexRays [2] is very strong for C-like decompilation. If you're looking at Go, Rust, or even C++ it is going to be a little bit more messy. As other commenters have said, you'll work function-by-function and it is expensive, though the free version does have decompilation (F5) for x86 and x64 (IIRC).

Binary Ninja [3] (no affiliation) is the coolest IMO, they have multiple intermediate representations they lift the assembly through. So you get like assembly -> low level IL -> medium level IL -> high level IL. There are also SSA forms (static single assignment) that can aid in programmatic analyses. The high level IL is very readable but makes no effort to be compilable as a programming language. That being said, Binary Ninja has implemented different "views" on the HLIL so you can show it as pseudo-C, Rust, etc. There is a free online version and the commercial version is cheaper than IDA but still expensive. Good Python API, good UI.

Ghidra [4] is the RE framework released by NSA. It is free and open source. It supports a ton of niche architectures. This is what most people use. I think the UI is awful, personally. It has a decompiler, the results are OK. They have an intermediate representation (P-Code) and plugins are in Java (since it is written in Java). I haven't worked much with it.

Most online decompilations you see for old games are likely using Ghidra, some might be using IDA. This is largely a manual process of doing a function at a time and building up the mental map of the program and how things interact.

Also worth mentioning are lifters. There were a few projects that aimed to lift assembly to LLVM IR (compiler framework's intermediate representation), with the idea being that then all your analyses could be written over LLVM IR as a lingua franca. Since it is in LLVM IR, it would be also recompilable and retargetable. [5][6]

1. https://reverseengineering.stackexchange.com/questions/2603/...

2. https://hex-rays.com/ida-free

3. https://binary.ninja/free/

4. https://ghidra-sre.org/

5. https://github.com/avast/retdec

6. https://github.com/lifting-bits/mcsema

netsec_burn2• 2 years ago

I made a site to use LLMs to help me with reverse engineering. The output is surprisingly readable, even with C++ classes. Let me know any feedback you might have: https://decompiler.zeroday.engineering/

btown• 2 years ago

What kind of file should be uploaded?

netsec_burn2• 2 years ago

The allowed types are a bit misleading. Any binary is accepted, any architecture. You can upload shared objects, ELF executables, PE binaries, etc.

I like to give it bomb executables (reverse engineering challenges) to test it.

readyplayernull• 2 years ago

This is great! With Ghidra I had to look for the corresponding libs of a very specific RiscV vendor, your SRE did it by itself. You should have your own HN thread in front page!

ianhawes• 2 years ago

Highly recommend it. I reversed an app with o1 Pro Mode and the analysis of the obfuscated C# code matched up accurately with what I eventually discovered by manually reversing.

chc4• 2 years ago

Reverse engineering C# is extremely different from C++ binaries.

flashgordon• 2 years ago

Interesting. Wouldn't this actually be a deterministic problem based on graph analysis. Id have thought LLMs would have been more effective taking the out out some graph recognizer and then identifying what those higher level constructs map to?

warkdarrior• 2 years ago

Deterministic maybe, but surely undecidable in the general case since you need whole program analysis to understand, for example, the purpose of a memory location. ML may help approximate this undecidable problem.

lumb63• 2 years ago

It has nothing to do with LLMs, but Ghidra is a wonderful tool.

jkstill• 2 years ago

I've only played a with this, but it was impressive.

https://ghidra-sre.org/

feznyng3• 2 years ago

You could use the LLM to help you write utility scripts for whatever disassembler you’re using e.g. python for IDA. That might work better than feeding it raw assembly.

Game RE communities also have all sorts of neat utilities for decompiling large cpp binaries. Skyrim’s community is pretty active with ghidra/ida.

Guessing you’re not lucky enough to have a PDB?

tonetegeatinst• 2 years ago

PDB?

feznyng3• 2 years ago

Program database file - only relevant if the binary is Windows. But that makes decomp an order of magnitude easier. I’d be surprised if OP had one though.

klmitchell2• 2 years ago

https://github.com/radareorg/r2ai

Dwedit• 2 years ago

Have you tried Ghidra yet? If you still have your debug symbols, then it can do a really good job.

JosephRedfern• 2 years ago

These guys are building foundational models for this purpose: https://reveng.ai/. The results are quite compelling, and they have plugins for your favourite reverse engineering tools.

latexr• 2 years ago

The domain makes it look like “Revenge AI”. Terrible name. Not as risqué as some others¹ but not as fun or memorable either.

¹ https://www.snopes.com/fact-check/domain-thing/

avg_dev• 2 years ago

i don't think so (that it is a terrible name); it's a pretty common term https://www.urbandictionary.com/define.php?term=reveng

olalonde• 2 years ago

I was about to comment that the domain was quite clever...

__alexander• 2 years ago

Do you have experience reverse engineering? If not, LLMs are not going to help much. LLMs are useful for aiding the analysis but they don’t do the analysis.

uncomplexity_• 2 years ago

Yea this one. If you have solid fundamentals these LLMs are really handy in assisting and never leading.

For example I have a minified javascript file, way obfuscated. I can paste the code and make it break down the initial structure. And then I tell it which parts to focus on and which parts to dig in deeper.

seba_dos1• 2 years ago

Good luck. If that's how you're approaching it, you're going to need it.

2-3-7-43-1807• 2 years ago

op apparently never even heard about reddit

sitkack• 2 years ago

Do you know the compiler and what the source possibly looks like? I found LLMs are pretty good at recovering code from binaries, they need help though.

If you are able to run the program and collect traces, that will help a ton.

u53rn4m3• 2 years ago

RevEng.AI have their own foundational AI models for decompilation with English language summaries.

stackghost• 2 years ago

The Advent of Cyber side quest this year needed some Ghidra and I found Pickman's Model was pretty good at helping me craft a heap exploit from a decompilation.

apatheticonion• 2 years ago

Inspired by the work out there that reverse engineers game engines, I've always wanted to try my hand at reverse engineering to contribute to the world of game preservation.

Is it actually legal to decompile a game engine from executables/dll files, write new sources by making sense of the output and rewriting it such that it can be compiled targeting modern APIs?

I feel like that must be illegal

userbinator• 2 years ago

Unfortunately LLMs are not good at precision and details, which is exactly what you need for the sort of analysis you're trying to do.

menaerus• 2 years ago

Right. Have a look at the paper above from Meta on how they fine-tuned the Code Llama with LLVM IR to beat the compiler in producing size-optimized binaries.

rgovostes• 2 years ago

The LLM4Decompile project (https://github.com/albertan017/LLM4Decompile) provides some open models for binary to C decompilation and Ghidra pseudocode refinement, along with some training sets.

RevEng.ai, linked a few times already, discusses their approach here: https://blog.reveng.ai/training-an-llm-to-decompile-assembly...

svilen_dobrev• 2 years ago

cpp? that's a preprocessor. u mean c++?

LLM won't help you much if u can't understand what it's talking about.

Manual way is, given ELF (linux executable format) somexe,

$ strings somexe

$ objdump -d somexe

$ objdump -s -j .ro data somexe

then look+ponder over the results.

and/or running ghidra (as mouse'd UI) over it.. which may help somewhat but not 100%

Have in mind, that objdump and ghidra have opposite ways of showing assembly transfer/multi-operand instructions - one has mov dest,target , other has mov target,dest - for same code.

no idea on (recent) windoze front. IDA ?

mahaloz• 2 years ago

I like using it for library function comments, variable name recovery, and sometimes types. The comments are usually hit or miss, but I find the variable names to be a bit better than auto-generated ones. I implement most of this in my decompiler plugin: https://github.com/mahaloz/DAILA; check it out if you are interested :).

Ask HN: How are you using LLMs for traversing decompiler output?

Related Stories