Reverse-engineering the Kayak app with mitmproxy

http://www.shubhro.com/2014/12/18/reverse-engineering-kayak-mitmproxy/

by shbhrsaha • 12 years ago

73 39 12 years ago

xrjn• 12 years ago

Great writeup, there's definitely potential to do something cool, especially if it's possible to get around any tracking and the following price manipulation. I tried installing your demo client, however on running it I got the following error:

  root@kayak:~/kayak-mobile-client# python client.py 
  Departure airport code: LBG
  Destination airport code: HAM
  Departure date (MM/DD/YY): 12/26/14
  Traceback (most recent call last):
    File "client.py", line 56, in <module>
      searchid = json.loads(r.text)["searchid"]
  KeyError: 'searchid'

shbhrsaha• 12 years ago

Appreciate your kind words.

Have you set up your UUID and HASH according to the post? You'll actually need to run the app through mitmproxy yourself to get your unique credentials. Then, just drop those into the variables at the top of the script.

I just added some error handling to the script to make this a bit more obvious!

ianlevesque• 12 years ago

A useful technique for sure. The only technique I know to slow this down is to use certificate pinning, but it's probably pointless. Some of your users are probably extremely motivated (like ours [1]) and it's obvious to them that what they are doing is unsupported.

1. http://difm.eu/dox/

nnd6• 12 years ago

Certificate pinning would help to prevent sniffing traffic with tools like mitmproxy, but once you decompile the binary, you can disable certificate validation.

I'm curious, in your case did you consider preventing using your API by third-party clients?

101914• 12 years ago

Note that mitmproxy, which requires a Python install, is not necessary to monitor what is being sent out from your computing device.

The same results can be achieved using only socat and the openssl binary.

While I understand the terminology is popular, I would not call this "reverse-engineering"; to me this is simply viewing your own traffic.

I believe users have a right to see the traffic they (or the apps they use) are sending, and for security reasons alone they should monitor what is being sent. https plus third party CA usage complicates such transparency, making proxying techniques necessary.

I wish more users would view their own traffic.

Keep up the good work.

userbinator2• 12 years ago

I've been MITM'ing my connections ever since I discovered the Proxomitron ( http://en.wikipedia.org/wiki/Proxomitron ) - it lets me customise webpages and filter out the crap before it gets to any browser, and works in all the browsers on the system; even the ones embedded in apps. It has a logging feature to show traffic too.

It would be great if a proxy could be run directly on the mobile device, as then it could be used anywhere along with the apps its monitoring.

As you said HTTPS is a bit of a pain since it's actively designed to resist such "attacks", but as long as there's the option to specify your own CAs it will work. Certificate "unpinning" is still a manual process, however...

dmix• 12 years ago

How would you use a proxy on a mobile phone? Just curious. I'm not familiar with Proxomitron, so I assume adding filters like privoxy or captures?

People have done this in ways by hacking the built-in VPN system I believe.

pmoriarty2• 12 years ago

Is Proxomitron any better than Privoxy?

uaygsfdbzf4• 12 years ago

Personally I would just use the Firefox web developer tools rather than mitmproxy, much easier.

perishabledave• 12 years ago

To log app requests? Firefox doesn't have a proxy unless I'm mistaken.

pmoriarty2• 12 years ago

Could you give a specific example of how socat can be used with the openssl binary in the way you suggest? I'd like to try it.

chrisan• 12 years ago

Another great mitm proxy is Charles Proxy http://www.charlesproxy.com/.

It has a really nice UI for looking at JSON responses such as the Kayak. Sometimes a collapsible tree is invaluable in looking through a response.

The easy filtering and formatting is primarily why I like it so much. Here is how it handles SSL for various ways http://www.charlesproxy.com/documentation/using-charles/ssl-...

Here is a screen shot of my iPhone Kayak app request for comparison http://imgur.com/gvKB6fr

mrbombastic• 12 years ago

I love Charles Proxy, I use it all the time for debugging service issues. Breakpoints and the ability to easily mock responses is really invaluable.

nnd6• 12 years ago

How is it better than mitmproxy? Also, it's not free/open-source.

brad0• 12 years ago

In this case you get what you pay for. You get filtering. Local and remote mapping. SSL decryption. Request and response detail views. Breakpoints. The list goes on. Seriously, it's the best $50 I spent as part of my developer toolkit.

vertak• 12 years ago

Does anyone know the possible legal repercussions open-sourcing a web service's API when the company doesn't explicitly grant permission? This is really neat, but could also raise the ire of a service that doesn't offer an API for a reason.

declan• 12 years ago

There are two obvious issues, as I see it. #1 is whether someone's liable for poking around inside the Kayak app in the way the author of the post did. #2 is whether it's legal to "open-source" the API by disclosing this information, once obtained.

My quick take is that #1 likely violates Kayak's terms of use for the app, its web site, or both. The TOU for the website has a host of prohibitions, including don't "bypass or circumvent other measures employed to prevent or limit access to Our Website," no deep-linking (such as a link to the API endpoint), and as a catchall also prohibits all inappropriate-as-defined-by-Kayak activity. I haven't read the app TOU, but it would be standard practice to prohibit reverse engineering of the app itself.

So we don't really need to reach point #2, which raises some 1A issues the first doesn't. (Note I'm excluding DMCA and CFAA issues because Kayak isn't our sue-happy friends at the MPAA or RIAA.)

As a practical matter, though, the author of the linked post, Shubhro Saha, appears to be an undergrad, so is probably judgment-proof and not the most likely target of litigation.

moyix2• 12 years ago

Just to add to this: the relevant case law seems to be Bowers v. Baystate Technologies [1], which held that EULA clauses prohibiting reverse engineering are enforceable. I personally think this is a terrible decision, since it also prevents a lot of good uses of reverse engineering (see some examples, like fighting censorship and diagnosing software vulnerabilities, in a recent paper of mine [2]).

[1] http://en.wikipedia.org/wiki/Bowers_v._Baystate_Technologies [2] https://mice.cs.columbia.edu/getTechreport.php?techreportID=...

nnd6• 12 years ago

I'm not a legal expert, but I bet there is a clause in TOS to prevent that. I wonder if anyone actually got sued for violating TOS this way though.

userbinator2• 12 years ago

DMCA has an exception for "interoperability" to allow reverse-engineering, which AFAIK is meant specifically for this case.

However, I wouldn't consider this any real "reverse-engineering", but just inspecting network traffic --- and it should be absolutely legal to do so when the traffic is generated from an app you legally obtained, running on a device you own.

IANAL, etc.

moyix2• 12 years ago

I'm not sure DMCA applies here, as it only bans tools that circumvent technical measures intended to enforce copyright protections. I don't think that Kayak's data is copyrightable (on the basis of Feist Publications, Inc., v. Rural Telephone Service Co., which says that mere compilations of public information with no creative aspect aren't considered copyrightable), so DMCA shouldn't apply.

But likewise, IANAL.

dthakur• 12 years ago

From that folder, get the mitmproxy-ca-cert.pem file onto your mobile device by emailing it to yourself, for example. Then follow certificate installation steps for iOS or Android.

You can just go to http://mitm.it on the device. It's a 'magic domain' for the proxied host. See http://mitm.it/doc/certinstall/webapp.html

nnd6• 12 years ago

I've tried this website on iPhone 6, and it redirects to the standard version of the website instead.

ketralnis• 12 years ago

I don't know about Kayak's economics, but at least at Hipmunk we pay our data providers per search and it's really quite expensive. If they aren't offering an API anymore, it's probably because it was too pricey to operate.

You could easily cost a travel search company thousands of dollars very very quickly using an API they don't want you to use. I don't know if it's illegal or not, but it's certainly immoral.

uaygsfdbzf4• 12 years ago

How is using the API any different to using the website or app? Sounds like you need to move your access controls one layer down, into the API. If the app does rate limiting, the API should too/instead.

Take the github approach and let open source devs write innovative apps based on your open API!

rdeboo• 12 years ago

The difference is that Kayak is not able to show you ads if you use the API, so they generate less income from API usage compared to (web)app usage.

edward2• 12 years ago

I'm happy to pay to use a flight search API. Kayak and Hipmunk could provide an API and pass on the cost to the API consumers.

nnd6• 12 years ago

Sabre has a free flight search API: https://developer.sabre.com/docs/read/REST_APIs

kremi• 12 years ago

The problem with any paid API is that the API consumer needs to make money somehow so that he can offer the price of the service.

How will the API consumer make money? I don't think Ads will generate enough revenue -- especially not on mobile.

Kayak will get some revenue from the bookings resulting from the searches, and would need to circle back some of this revenue back to the API consumer. This is technically complicated because it requires a lot of tracking infrastructure.

In the end, a service that would make sense is a lot more complicated than simply having an endpoint and saying it's X dollars per Y calls.

nnd6• 12 years ago

I wouldn't call this reverse-engineering.

How are UUID and HASH are generated? Are they unique to every installation?

javiercr• 12 years ago

Interesting! We've used a similar technique to reverse-engineering mobile apps from different banks in order to create a Ruby gem to fetch bank data (balance and transactions).

https://github.com/ismaGNU/bankScrap

Reverse-engineering the Kayak app with mitmproxy

Related Stories