Show HN: Mitmproxy2swagger – Automagically reverse-engineer REST APIs

https://github.com/alufers/mitmproxy2swagger

by alufers • 4 years ago

691 85 4 years ago

alufers12• 4 years ago

Wanted to show off my little project which helps whith reverse engneering APIs used by various apps. It takes HTTP traffic capturewd by mitmproxy and generates an OpenAPI specification for a given REST API.

I have used it already on two apps and the results are good enough to write an alternative client or quickly automate some stuff.

upupandup2• 4 years ago

does it capture route/server rendered pages too?

alufers12• 4 years ago

It does, but it will only generate schema descriptions for JSON endpoints. Whis means that the URL and method will appear in the spec, but not the response/request schema.

mhils• 4 years ago

mitmproxy dev here, very awesome! :) This seems to be particularly useful to quickly generate clients for reverse-engineered APIs.

mohsen1• 4 years ago

Swagger Editor dev which now works at Airbnb here. This is hilarious!

lancebeet• 4 years ago

This is a really clever project. It seems like an obvious idea once you've seen it, but it clearly isn't. Thank you for sharing it.

anitil• 4 years ago

What a fantastic idea! I have so many half baked things that some idiot (me) built without documenting the underlying API. This will make life so much easier

ludovicianul2• 4 years ago

This is great :) You can then fuzz your APIs for issues using https://github.com/Endava/cats.

oneweekwonder• 4 years ago

little bit off-topic, but do anybody know of something similar for soap/wsdl? I'm aware of soapui mock service.

alufers12• 4 years ago

Doesn't wsdl just expose the schema on the server?

efitz3• 4 years ago

WSDL and OpenAPI/Swagger solve similar problems.

Roughly speaking: WSDL is to XML web services as OpenAPI is to REST

They both model the API and message structure of an API. AFAICT WSDL goes a little farther in that you can declare message sequences (I might be giving short shrift to OpenAPI here).

upupandup2• 4 years ago

this is absolutely insane!!! I understand capturing the REST api network part, is it then examining the request body, headers being sent back and forth to figure out the API?

alufers12• 4 years ago

Yes, this is basically what this program does.

ninkendo• 4 years ago

From what I understand it’s also somewhat how JIT works in various JavaScript engines: observe the sorts of objects (which naively have the performance characteristics of hash tables) you see, and start defining static offsets for fields you observed. The JIT’d (fast) objects may morph over time as new fields are observed, but I’d imagine it’s a similar idea to creating documentation… “this object tends to have these fields, so just pretend those are the only fields it can have, until another request proves otherwise”, with similar guess/checking for their types/etc.

useful• 4 years ago

bravo, I've wanted something like this

captn3m0• 4 years ago

Almost exactly a fit against my idea[1] to generate OpenAPI from HAR files. Going to read through to see if I can add HAR support.

[1] https://github.com/captn3m0/ideas#openapi-specification-gene...

efitz3• 4 years ago

OpenAPI is just the latest version of swagger. Should not be hard to change.

I was able to translate HAR to OpenAPI with this web site's free preview: https://www.apimatic.io/transformer/

I also see others are working on the same thing: https://github.com/dcarr178/har2openapi

kaidon• 4 years ago

Also https://github.com/anbuksv/avantation

alufers12• 4 years ago

Hey! Just writing to let you know that I've added HAR input support to mitmproxy2swagger.

jeroenhd5• 4 years ago

Very interesting! Would this also be able to determine what kind of auth (header tokens, cookies, etc) the APIs require or is that something you still need to detect manually?

alufers12• 4 years ago

At this point yes, but I am working on adding this.

SemanticStrengh3• 4 years ago

Can this be used to generate a REST documentation for your own frontend just by interacting with it? This should be augmented via a crawler, that click everyclickable element recursively.

alufers12• 4 years ago

Totally, but you would need to do some manual cleanup and naming afterwards to make it more useful than just reading the source code. You could also for example use your integration tests if you have some to capture as much routes as possible.

SemanticStrengh3• 4 years ago

of course the generated doc should be refined (e.g. filling missing types, error codes) but your lib would save us a lot of work and make the world a better place.

Divyeshkharade• 4 years ago

This looks amazing. Will it also capture data types like enumerators by someway detecting patters?

alufers12• 4 years ago

I thought about it, but it would be hard to distinguish between an enumerator and just static data. For example if you logged in with only one account it could classify the "username" field as an enumeration, because there is only one captured value.

freedomben• 4 years ago

Yeah I imagine that is nearly impossible without capturing data at scale. Awesome tool! I'm super grateful :-)

BWStearns• 4 years ago

This is fantastic!

efitz3• 4 years ago

This is awesome; I’m going to try it as soon as I get back to my desk. I’ve been working on trying to glue together tools to translate Charles proxy output to OpenAPI (swagger). I think it would be a great tool to have in a web app reverse engineering toolbox.

dudus• 4 years ago

This is a great idea. Kudos.

chrisweekly• 4 years ago

Awesome idea! Thank you for creating and sharing!

instagary• 4 years ago

How did you bypass cert pinning in the video for the Airbnb app?

alufers12• 4 years ago

I didn't, just added a self-signed cert to my keychain on macOS and launched the app as downloaded from App Store.

I guess Airbnb doesn't use cert pinning.

paxys• 4 years ago

It doesn't have anything to do with mobile. The web client uses the same APIs.

nickysielicki• 4 years ago

This is really incredible. With a rooted android phone and these tools, plus a couple others [1,2,3], you can get a skeleton to implement a backend for any app you want.

[1] https://github.com/koxudaxi/fastapi-code-generator

[2] https://github.com/ioxiocom/openapi-to-fastapi

[3] https://infosecwriteups.com/hail-frida-the-universal-ssl-pin...

andreidd• 4 years ago

That's interesting, but it won't work with native code that statically links a SSL implementation.

jeroenhd5• 4 years ago

In many applications you can bypass built-in verifications with some Frida [1] code. It requires more effort to do so, of course, as you'd need to find the OpenSSL methods (with a script like this [2] and bypass the verification in there.

If you're really intent on getting it to work, downloading the binary, patching out the verification function and putting it back is also possible if you're root.

[1] https://frida.re/docs/android/

[2] https://mobsecguys.medium.com/exploring-native-functions-wit...

ducktective2• 4 years ago

Is it possible to do this on wireshark/tcpdump pcap dumps? Like for finding out hostnames, endpoints and request packets of HTTPS requests that an android app is making?

alufers12• 4 years ago

The problem with pcap is that whe requests there would be encrypted and basically there is no way to practically decrypt them.

Mitmproxy solves that by being between the client and server and injecting it's own self-signed certificate (which you need to add to the trusted certificates on the phone, which requires root).

resoluteteeth2• 4 years ago

See SSLKEYLOGFILE

mro_name• 4 years ago

awesome take

jwong_• 4 years ago

Really neat! Gives me an idea on using something like this to generate e.g., CURL commands to mimic SSO flows.

Even just documenting an SSO flow as a diagram would be quite neat.

john-tells-all• 4 years ago

Note that for single resources, Chrome/Edge can do this now. There's a semi-hidden "copy this resource as Curl" option:

https://everything.curl.dev/usingcurl/copyas#:~:text=From%20....

When it works, it's effing magic! Spectacular for very quickly knocking out Bash scripts that test multiple APIs.

klyr• 4 years ago

Hi, I would also like to add another tool I'm contributing to at work (cisco) called APIClarity [1]. It aims at reconstructing swagger specifications of REST microservices running in K8S, but can also be run locally.

This is a challenging task and we don't support OpenAPI v3 specs yet (we are working on it).

Feel free to have a look, and get ideas from it :)

We'll also be presenting it at next Kubecon 2022.

[1] https://github.com/openclarity/apiclarity

sohaibtariq• 4 years ago

Try out https://www.apimatic.io/transformer/ for converting Swagger Specs to OpenAPI

eligro91• 4 years ago

Really amazing.

We're having hundreds of undocumented endpoints created over the years, and running this tool on our backends will create instantly good documentation

Thanks for that! Will give feedbacks if any issues

andrewstuart2• 4 years ago

I've always wanted to build something similar to this, by reading HAR files captured right out of the devtools. Have you given any thought to that as an alternative input?

alufers12• 4 years ago

Hey! Just writing to let you know that I've added HAR input support to mitmproxy2swagger.

Sytten• 4 years ago

Super nice! We might integrate something similar in Caido proxy.

a-dub• 4 years ago

lol!

step 2: features for training a language model on the request and response variables in the mitm stream and a shim for standing up a fully ml data driven zero code mock backend.

dnnssl2• 4 years ago

Starred. Does this work with non-emulated iOS or Android http calls in which you may need to disable app level security?

jeroenhd5• 4 years ago

For Android you'll probably need root access (unless the app developer has opted in to loading your user-imported certificate authorities). For iOS this should be easier.

However, many apps apply cert pinning in production builds, which will require tools like Frida to disable them, which in turn requires root access/a jailbreak to function.

Alternatively, you could pull the apps from your phone without root (at least on Android), patch the most obvious cert pinning out (usually in the network manifest file) and install the new version.

Cilvic2• 4 years ago

The question is maybe a bit off-topic a d vague. That's because I struggle to express it with the right terms:

I'm looking for a generic tool to build and then serve:

Accept Incoming request (API contract A) Send outgoing request (API contract B) potentially with parameters from the incoming request Receiving incoming response (API contract B) Do some translations/string manipulation Send outgoing response (API contract A)

jeroenhd5• 4 years ago

mitmproxy (https://mitmproxy.org/) has scripting support that will let you do most of this.

For example, you can expose mitmproxy, listen to HTTP requests for a specific host (using this API: https://docs.mitmproxy.org/stable/api/mitmproxy/http.html), intercept the request, do whatever API calls you need, and inject a response without ever forwarding the request to the original server.

Alternatively, you could modify the request and then change the request destination, like in this example here: https://docs.mitmproxy.org/stable/addons-examples/#http-redi.... Using the WSGI support, you could even use normal Python annotations to build your own API without doing too much pattern matching: https://docs.mitmproxy.org/stable/addons-examples/#wsgi-flas...

Cilvic2• 4 years ago

Ok. This sounds great for easy developing. But when I'm hosting this I'm not a mitmproxy. I want to act like a normal server/endpoint for API A.

Labo3333• 4 years ago

Very nice!

On the same note, I wrote a program to generate Python code (requests) from a HAR capture: https://github.com/louisabraham/har2requests

I think using HAR captures is simpler for the end user than spawning mitmproxy as they don't require any installation and are extracted from the network tab of the browser devtools. Is there a reason why you didn't use them?

EDIT: I realized that mitmproxy can also get traffic from other devices like phones. Very cool project, I will think about modifying mine to support mitmproxy captures!

olabyne• 4 years ago

Oh, I used a python script to generate pre-made requests from HAR recently, I'm pretty sure it was your git ! Very useful :)

Labo3333• 4 years ago

Thanks!

alufers12• 4 years ago

Hey! Just writing to let you know that I've added HAR input support to mitmproxy2swagger.

Labo3333• 4 years ago

Wow that's super cool! Thanks!

aleksiy123• 4 years ago

Really awesome, I tried my hand at writing something similar and was surprised at how well it actually ended up working.

I feel liken the next step is automatically generating load tests and/or fuzzing tests. Felt like that could be a real product.

ludovicianul2• 4 years ago

Here you go: https://github.com/Endava/cats

andrewstuart• 4 years ago

Be interesting to run a fuzzer on the API whilst doing this.

julianlam• 4 years ago

This is great work!

This would come in very handy for codebases where an OpenAPI v3 spec would be welcome, but is too onerous to create by hand. Run this for a bit, have it spit out a nearly complete spec, and tweak it a bit to output the final product.

In fact, it is precisely what we did to generate the OpenAPI docs for NodeBB [1]. We had an undocumented API that we turned into an OpenAPI v3 file.

[1] https://docs.nodebb.org/api/read

dsfiguer• 4 years ago

Oh I love this so much! This would help me with scraping certain sites.

renewiltord• 4 years ago

This is great. Good example too since Airbnb could use with some improvement to the user chrome: include cleaning fees, etc

POPOSYS• 4 years ago

Can we have this as a browser dev tool please? F12 -> Tab REST -> Create spec from API

h1fra• 4 years ago

very nice !

thefilmore• 4 years ago

This is one of the most clever projects I've seen in a while. Nice work.

mutant• 4 years ago

This is absolutely phenomenal!

difu_disciple• 4 years ago

This is fantastic. Thank you

nattaylor• 4 years ago

I gave this a try today. It was silky smooth! Is it possible to tell Swagger to omit OPTIONS methods?

evnix• 4 years ago

I did something similar a year ago at the company which I work, I basically wrote a middleware that intercepts all the requests(express JS) and writes to a OpenAPI YAML file. It diffs previous requests to see which parts of the request path could be variables. The system isn't perfect but you are 95% there which is better than having no documentation or to hand write documentation or keep that spec file updated with changes that people introduce in the code. (got promoted to tech lead after this :-) )

lsferreira42• 4 years ago

Congrats, this is really awsome and i have a use for it right now, it will be really useful for debuging old and undocumented api's

Show HN: Mitmproxy2swagger – Automagically reverse-engineer REST APIs

Related Stories