iOS Apps Using Private APIs

https://sourcedna.com/blog/20151018/ios-apps-using-private-apis.html

by tptacek • 11 years ago

195 79 11 years ago

a3n2• 11 years ago

Since we also identify SDKs by their binary signatures, we noticed that these functions were all part of a common codebase, the Youmi advertising SDK from China.
We believe the developers of these apps aren’t aware of this since the SDK is delivered in binary form, obfuscated, and user info is uploaded to Youmi’s server, not the app’s.

Know your binaries?

adevine• 11 years ago

Which is impossible with any app of reasonable size that uses third party SDKs (which are often required for business reasons). Apple didn't catch it, how can we expect hundreds of small developers to catch it?

mikeash10• 11 years ago

Apple spends a few minutes reviewing any given app. The developers have much more time to examine this stuff.

NateLawson14• 11 years ago

Well, that's what we do for a living -- help thousands of small or large developers find these problems ahead of time.

It's not an easy problem and not something you can do by hand. We've indexed the code in millions of apps and then learn patterns from it. For example, you can discover a new SDK being adopted by apps in one region purely from the co-occurrence of binary signatures.

We're working hard to give developers this kind of insight, helping them detect problems in their apps before they are affected.

makecheck2• 11 years ago

I'm actually very surprised this hasn't happened years ago. The power of Objective-C's runtime has always made this pretty straightforward.

Apple can defend against unauthorized calls to even runtime-composed method names though. I can think of a few ways.

They could move as much "private" functionality as possible outside of Objective-C objects entirely, which requires that you know the C function name and makes it obvious when you've linked to it. This should probably be done for at least the really big things like obtaining a device ID or listing processes.

Even if they stick with Objective-C, they could have an obfuscation process internal to Apple that generates names for private methods. Their own developers could use something stable and sane to refer to the methods but each minor iOS update could aggressively change the names. If the methods are regularly breaking with each release and they're much harder to find in the first place, that may be a sufficient deterrent to other developers.

They could make it so that the methods are not even callable outside of certain framework binaries, or they could examine the call stack to require certain parent APIs. At least that way, if you want to call a private API, you have to somehow trick a public API into doing it for you.

And, I think Apple does say somewhere that developers shouldn't use leading underscores for their own APIs. They could hack NSSelectorFromString(), etc. to refuse to return selectors that match certain Apple-reserved patterns in all circumstances.

BooneJS• 11 years ago

What's so magical about Obj-C's runtime that this cannot be detected by Apple? If a third-party can scan for them, why can't Apple? If you're going to have a walled garden, by all means please ensure the walls can't be scaled or dug under.

matthewmacleod• 11 years ago

I'm sure they can, and will probably start doing so.

K0nserv2• 11 years ago

AFAIK the combination of NSSelectorFromString and NSClassFromString would allow you to build and make calls from obfuscated strings during runtime. Since Objective-C uses message passing this is non trivial to catch. Compared to C were you must explicitly reference symbols that are easy to automatically check for detecting the use of third party APIs is more difficult.

I am not sure if you have to explicitly link against certain frameworks/dylibs though, someone with more knowledge feel free to correct me

NateLawson14• 11 years ago

We don't know why these apps got through in particular. However, I do think we have a unique vantage point by being outside any particular app store.

We scan code on both iOS and Android, identifying libraries and patterns of behavior. OpenSSL, for example, is the same code, regardless of platform, but the versions of it may vary depending on packaging and porting. We found one guy who had compiled OpenSSL for Android and was distributing binaries from github. Not the safest way to get your software.

As a third-party, you can move quickly and correlate info across app stores. But I also think we've developed a pretty unique code analysis engine that neither Apple nor Google has. :-)

mikeash10• 11 years ago

Private API isn't a security mechanism. Apple attempts to detect it, and reject apps that use it, purely to be friendlier to their users. Private API usage makes apps more likely to break with OS updates, and Apple doesn't want apps breaking.

So while Apple is motivated to catch this stuff to an extent, they aren't that motivated, and they may well be perfectly happy with the current level of sophistication.

joosters5• 11 years ago

Does anyone know why they aren't access protected? It seems like poor security to just hope that apps don't call these functions.

hellbanner• 11 years ago

Ironically new iOS updates tend to break older apps anyways.

toucharcade.com/2015/09/23/bioshock-removal-2k-support-response/

masklinn• 11 years ago

I'm actually very surprised this hasn't happened years ago.

It has, and very early on Apple didn't check for their usage so developers started using class-dump results to look for interesting private/undocumented APIs.

Things went south when Apple started checking and rejecting applications which did that: https://www.cocoanetics.com/2009/11/forbidden-fruit-apple-ap...

The youmi stuff is creepy, but generally speaking it's more a question of there being no guarantee about these API's stability or continuity, so the application may break between even minor OS updates, which is undesirable.

jws• 11 years ago

dlopen() can be used to call a C function by a string name.

More ambitious programmers can scan executable regions of memory for the instruction stream of the target function and call the address directly without any help from symbolic names at all.

conradev2• 11 years ago

Correct. Even easier, they can store a hash of the symbol name and walk the dyld cache until they find a match. No dlsym required.

The runtime isn't as straightforward as SourceDNA makes it seem.

makecheck2• 11 years ago

I didn't say the function would be in a dynamic library, although you're right that this would work if it was.

A statically-linked function would have to be referenced directly in the executable somewhere.

lnanek2• 11 years ago

It's kind of an arms race. In the early days of iOS app development, developers would just not call the private APIs during the review period and hide them in an easter egg or enable them from the client checking with the server to see if the setting had been flipped to start using them. Ever since Apple started actually scanning the binaries developers have had to go to greater lengths to obfuscate the usage, but still give users the most powerful features.

Some of the things developed during the arms race have become best practices, though. If you are an iOS game developer and you release a game update without the ability to adjust the game substantially from the server side, you are really in trouble. It takes another long review period from Apple to change anything, so best practice is to control the functionality from the server where if it makes the game too difficult for the users or something it can be adjusted without suffering the Apple review delay.

mikestew• 11 years ago

I'm actually very surprised this hasn't happened years ago.

It has happened years ago. About a year ago I left a company that was doing this for at least the last two or three years. Their motivations were strictly a technical workaround to get functionality that Apple didn't present to a public API, not to nefariously gather user info, but the technique is the same, and not particularly difficult to figure out.

Probably a good five years ago Apple rejected an app of mine for use of a private API (this was about the time their static analysis was implemented). Another company that sold a buttload of copies of their app had to have used the same API, and their app came out a little bit before mine. Could be Apple just missed it, but I suspect the company in question was doing something similar. (They made millions, I languished in obscurity, but I'm not bitter. <g>)

And those are just the ones I directly know about, or strongly suspect. I'm sure there are/were plenty more.

Hello71• 11 years ago

They could move as much "private" functionality as possible outside of Objective-C objects entirely, which requires that you know the C function name and makes it obvious when you've linked to it.

https://stackoverflow.com/questions/6530701/is-the-function-...

United857• 11 years ago

I wonder if it's possible to for the ObjC runtime to randomly obfuscate a processes' "private" selectors on startup, and rewrite ObjC invocations within system frameworks dynamically, to make private API usage harder.

Like what ASLR does for memory addresses, but for selectors.

kinofcain• 11 years ago

Seems like a lot of the things they're putting behind private APIs should instead/also be behind a user permission. Getting the list of installed apps, device serial number, and the users email address shouldn't be protected simply with obfuscation.

musesum2• 11 years ago

Yeah, the user's email address is a surprise. It's pretty easy to bypass a simple scan with objc_msgSend.

Conversely, it would be nice if the user could grant access to email and iMessage sandboxes. This would allow us to apply machine learning to personalize services. Ironically, by allowing opt-in, Android is an easier platform for creating private services.

BinaryIdiot• 11 years ago

I'm not an iOS developer (well, not really; I don't know what I'm doing) but this seems like it would be a really easy thing for Apple to detect. Does Apple simply not care about access to these to the degree of adding in better checking or is there something fundamental about their platform that makes checking for Apple seriously difficult?

RandallBrown• 11 years ago

Objectiv-C works by sending a message to an object. If that object responds to the message, it does something. The message is essentially just a string and in fact you can use strings to build selectors.

Normally, these private API selectors would show up in a class dump and Apple will reject you app. But, if you're clever, you can hide them from a class dump. You could encrypt the strings, then decrypt them at runtime and Apple could no longer find your private API usage in a static scan.

At runtime they could detect you calling private APIs, but it would be easy enough to code it so that you don't call any private APIs for a few days after first launch or make them so they're turned on with a server side flag. That way Apple would never notice the private api usage during an App Store review.

achr2• 11 years ago

The question is why don't these APIs require authorization in the first place? Obfuscation is a terrible security practice.

saurik• 11 years ago

Your same argument would stand without modification for reading all data from the keychain, accessing files saved by other applications, using your location, or any other things that require permissions: clearly, then, it is wrong on the face of it, because Apple has simply failed to secure a wide class of things from these processes. It honestly doesn't matter whether there is an "API" for it at the C or Objective-C layers, and that is a red herring: the issue is that there is an IPC layer somewhere (whether a Mach server or a file on disk) that exposes an API by which this data can be obtained by this process.

Alphasite_• 11 years ago

I imagine its for performance reasons, every obj-c call uses it and adding extra instructions would massively hurt performance.

musesum2• 11 years ago

This is not new. Check out "Microsoft AARD code" -- an inverted example of surreptitious analytics, in 1992. TL;DR: the beta version of Windows 3.1 showed a warning if user was using DR-DOS, a competing OS. The payload was encrypted and could be triggered in the production version of Win 3.1 by changing a flag.

yuhong• 11 years ago

Yea, there was a reason why I mentioned DR-DOS when I was discussing the OS/2 2.0 fiasco.

anonymousDan• 11 years ago

Is it just me or are they totally ripping off the research done by the Iris team and making it sound like they came up with these vulnerabilities themselves? I know they give the researchers a cursory mention, but it's buried at the bottom of the article.

tptacek3• 11 years ago

No, that's not at all what they did. They discovered the issue independently and, as you can see if you read the whole article, in a manner very different from that of the other team.

(Fair warning: I am not just a disinterested observer of SourceDNA).

peterclary• 11 years ago

"The apps using Youmi’s SDK have been removed from the App Store and any new apps submitted to the App Store using this SDK will be rejected." I'll be interested to see what happens to Youmi now that they're blocked from iOS. SDK developers: Consider yourselves warned.

knd775• 11 years ago

I imagine that new versions with the private API stuff removed will be allowed. I don't believe that this is some sort of lifetime ban on Youmi.

NateLawson14• 11 years ago

Hey all, I'm the founder of SourceDNA and happy to answer any questions about how we found this or about binary code search in general.

We take a different approach to understanding code than the traditional antivirus world. Rather than try to hunt for a needle in a haystack, we've created a system for finding anomalies in code that's already published. For example, you can build a set of signatures for "bad apps" and then repeatedly search for them (AV model) or you can profile what makes an app "good" and then look for clusters of apps that deviate from it (SourceDNA).

Consider an ad SDK like Youmi here. They weren't always scraping this private data from your phone. There are some apps that have this library but that version is a typical, only sorta intrusive, ad network.

But, over time, they began adding in these private API calls and obfuscating them. This change sticks out when you track the history of this code and compare to other libraries. There was more and more usage of dlopen/dlsym with string prep functions beforehand. This is quite different from other libraries, where they stick to more common syscalls.

By looking for anomalies, we can be alerted to new trends, whatever the underlying cause. Then we dig into the code to try to figure out what it means, which is still often the hardest part. Still, being able to test our ideas against this huge collection of indexed apps makes it much easier to figure out what's really going on.

vitd• 11 years ago

One concern I have is that they'll just move this one step further down the road. For example, I believe that you can get the address of a function like dlopen() by manually loading the bundle and looking up the function name (via something like CFBundleGetFunctionPointerForName()), constructing the name "dlopen()" through some obfuscated method as they do with Objective-C symbols. Then it becomes harder to detect that they're even using dlopen(). Any plans on how to detect that? It seems like an arms race that can't easily be won.

NateLawson14• 11 years ago

You're exactly right. You can go even further in gathering environmental data from syscalls and using it to construct strings at runtime. At some point, you have to include dynamic analysis in addition to static analysis.

The iRiS paper we mentioned in the blog post describes a really great approach to doing this. They do "forced execution" using a port of Valgrind to iOS. They also do the exact right thing by resolving as many call targets as possible statically, then using dynamic analysis only for the call sites that can't be resolved. This saves on runtime and complexity, though you might notice that even this approach didn't resolve 100% of them.

Ultimately, you're dealing with a variant of the halting problem, where the app uses a specific value only on a full moon, on iOS 4.1, where the username is "sjobs". And that's why computers are still fun.

PDF: http://www.cse.buffalo.edu/~mohaisen/classes/fall2015/cse709...

anatari• 11 years ago

How did you get access to the binaries? I would think the AppStore requires a session and valid purchase in order to download. Or is this analysis based on binaries submitted by your users?

NateLawson14• 11 years ago

We both download apps and scan apps developers upload to us.

pradn• 11 years ago

How did SourceDNA have access to millions of iOS app binaries? Can anyone just download all the apps in the App Store?

asadlionpk• 11 years ago

I think you can download .ipa file using iTunes. Alternatively, using iFunbox to backup apps.

jevinskie• 11 years ago

AFAIK you still have to run those on a jailbroken device, letting the kernel decrypt the __TEXT segment for you, then dumping the decrypted binary from memory to disk. Though every binary is encrypted the same, no matter the device, so supposedly there may be some key that you can extract to enable decryption off-device.

viraptor• 11 years ago

Blog about 2 groups discovering the bad apps and reporting it to Apple, but then: "Apple has issued the following statement. “We’ve identified a group of apps that..." Stay classy Apple - great attribution.

iOS Apps Using Private APIs

Related Stories