iOS 6.0 Causes CDN Overages

http://labs.prx.org/2012/11/14/ios-6-0-devours-data-plans-causes-cdn-overages/

by woodhull • 14 years ago

157 92 14 years ago

labs.prx.org

chrisrhoden3• 14 years ago

Author of the post here, happy to answer any questions. I'm pretty confident in our findings here, and some of the behavior is just bizarre.

sigzero• 14 years ago

Does 6.0.1 fix it?

jonny_eh• 14 years ago

The article says that the problem appears to be fixed in 6.0.1, at least for wifi.

kookster3• 14 years ago

We have some ideas on how to test 3G with 6.0.1, but haven't proven it one way or another yet.

binarycrusader• 14 years ago

From the fine article: "We have been unable to reproduce the issue using iOS 5 or using iOS 6.0.1".

So the problem's already been fixed as far as they can tell.

dsl• 14 years ago

I saw similar behavior when I was working on the backend for an IOS app.

We were able to track it down to a retry mechanism for streaming content when the server was responding too slowly. Maybe you are running into something similar.

owenfi3• 14 years ago

I'm seeing this line in my log as well, this person downloaded about 750MB and they don't seem to be on iOS 6.0.

  Reqs  User Agent
  5,378 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14

Mizza• 14 years ago

Ouch, elsewhere on the internet it seems like people are actually getting hit by this. Hopefully damage won't be too bad. Good research, PRX!

kannan2• 14 years ago

Now that explains the sudden spurt in my 3G data usage in the last 3 weeks. This forced me to switch off my 3G :(. Hope Apple releases a fix soon

farski• 14 years ago

We (PRX) haven't seen any evidence to show that it is still an issue in iOS 6.0.1, so you should be all set if you've upgraded, or an upgrade should fix it. If you are upgrade already and seeing overuse, please comment.

kannan2• 14 years ago

I was (and still) using iOS 6.0.1 when I started seeing the surge in data usage

dageshi• 14 years ago

Is it just me or are apple making more dumb mistakes lately? There's this and that ajax caching issue from a month or two ago.

Perhaps I'm just having selective memory?

hesdeadjim• 14 years ago

I've had the exact same feeling. It seems like their quality control has gotten worse over the last year or so.

publicfig2• 14 years ago

I think Apple has always had the tendency to make dumb mistakes, but I will say that I think reporting on it has changed. Remember all of the jokes about "It's not a bug, it's a feature" back when Jobs was there? Apple is by no means immune from mistakes and I don't think they ever were.

nikatwork• 14 years ago

Exactly. The reality distortion field has collapsed and everyone is less willing to forgive mistakes. I mean come on, remember the MobileMe launch failure?

That said, the famous Apple attention to detail does seem to be slipping, I sincerely hope this is just a transitional period and they tighten their game back up.

danso2• 14 years ago

I dunno, how crummy the App Store has become is a perfect example of this. The App Store is one prominent way they make loads of money and build loyalty to the ecosystem, and yet it is a shitshow on IOS6...the responsiveness is as bad as I've seen on non-iOS software, but for me, the two really inexplicable failures are:

1. the use of modals to show individual apps on the iPad. So if you get to an app page and accidentally touch outside the modal, the app page disappears and you have to go back to Safari to find the link that brought you there.

2. On my brand new iPod, the Genius View has never NOt crashed. I know Genius (and most of Apple's search services) are poor to begin with, but it's a prominent button on the nav. How does it just not work on a brand new device?

3. I'm also guessing the lack of a light sensor on the new iPod was the result of a screwup in the dev cycle...I would've never guessed that its absence would be such a deficiency but when you can't see a single thing on the iPod merely because you stepped from indoors to outdoors...to the point where you have to go back inside to adjust the brightness...that's just infuriating.

I can't remember the last time I've been infuriated by an Apple device...I feel infuriated all the time with my expensive Sony camera and how its video button is right where you can accidentally bump into it when shooting a pic...which prevents you from shooting a photo. The last few Apple devices I've had (two laptops, 2 phones, 2 iPads, countless iPods) have always been satisfactory in terms of polish...not iOS6 nor the new iPod touch

blinkingled4• 14 years ago

> the use of modals to show individual apps on the iPad

This bugged me a lot after iOS6 update. Also the small things such as when there are App Updates and badge shows up on the App Store icon, clicking it takes me to Featured Apps section - not updates as one would expect. On Android for instance, the update notification takes you directly to a screen allowing you to update as expected.

MaysonL• 14 years ago

The thing that continually annoys, and occasionally infuriates me, is the f'ing no-button trackpad on the MBP: a dozen times a day my finger or thumb slips on the thing, just as I'm clicking, and it misses, or hits the wrong link.

scrumper• 14 years ago

Your Sony: is it an NEX-7? There's a firmware update just out which fixes the video button problem: http://blog.sony.com/alphafirmwareupdates

stevoski• 14 years ago

Ever since I started using Apple gear, I've found apple's software quality is way below hardware. iTunes on Windows was always a disappointment.

pretoriusB7• 14 years ago

>Is it just me or are apple making more dumb mistakes lately? There's this and that ajax caching issue from a month or two ago. Perhaps I'm just having selective memory?

No, it's not just you. It's the whole tech pop culture sector that has selective memory.

Apple did as much mistakes on the old times. It's just that it's not the underdog anymore, and all the hip kids are looking to fry Apple for those, instead of understanding that, heck, mistakes happen in every software/hardware offering.

The iPhone 4 antenna issue? Not even a blip compared to the iBook G3 logic board fiasco. But everybody forgets those things and adds it to some all-encompassing theory of decline...

Same with the notion of the "non-innovative update" for iPhone 5, etc.

The forgotten that less than a decade ago Apple did even far more minor incremental updates to the iPod line (a new click wheel, some more storage, a little thinner, now with a color screen, now it plays video, etc) and everybody was overwhelmed everytime a new device came out.

Suddenly they expect miracles out of every Keynote.

kemiller• 14 years ago

This is exactly what I saw with the Podcast app last week. Used up 2GB in about 6 hours and put me into overage. I bet the class action lawyers are already circling.

owenfi3• 14 years ago

Our very small outfit at verbmill.net is seeing similar signs. For instance, 700MB+ downloaded by a single client when an individual podcast is a maximum of ~50MB. Checking my logs now to see if I can narrow down device/version number. All of our files are hosted on S3.

owenfi3• 14 years ago

Ominous example.

  Reqs	Total	Avg	B/s	Client
  666	2G	3M	127K	mobile-198-xxx-xxx-088.mycingular.net.
  666	AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)

I hope this user is on an unlimited plan :(

fruchtose• 14 years ago

Wow, that is truly unfortunate. How long did it take for that 2GB to be chewed up?

noinput3• 14 years ago

Same here, noticed a huge spike in bandwidth for my startup's shows hosted on CloudFront. 206 responses are flooding my logs and I moved everything off of CloudFront for my users to triage the cost. This breakdown explains it all: http://cl.ly/image/2H3u3u2O333g

Talk about a pain, not to mention the 10x bill from Amazon that came as a surprise.

kookster3• 14 years ago

Exactly - we see this causing a crazy number of 206 calls.

donavanm4• 14 years ago

"I moved everything off of CloudFront for my users to triage the cost."

Can you explain more? Was there some cloudfront behavior exacerbating the issue? AFAIK this is just a client (mis)using range requests and CloudFront replying as requested.

noinput3• 14 years ago

You are correct and I should have wrote that note differently. I moved some high traffic files to some endpoints I have where bandwidth isn't a concern to change the formula a bit (primary) and the rest are back to using basic S3 for now. The bandwidth is still being consumed, however now it's easier on the wallet.

After looking more closely at the differences for pricing between S3/CloudFront I realize it's appears cheaper to be on CloudFront. I assumed wrong that CloudFront data-out was billed in addition to S3 because of the way it's notated on the usage page ("AWS Data Transfer (excluding Amazon CloudFront)"). To prove it I ran two reports on CF (left) and S3 (right): http://cl.ly/image/0q0y3u2X0g1a and you can tell where I made the change, as well data is only counted on the service in question and not on both.

Can anyone else confirm the above?

blinkingled4• 14 years ago

Wonder if the CDNs don't have any safeguards in place to avoid single client downloading same resource multiple times?

chc• 14 years ago

That doesn't seem practical. How would they know if it's the same client? Just watching for multiple requests from the same IP will result in a lot false positives (e.g. most college students will be SOL). And what would they do if they could detect it but the client didn't realize it already had the file? Just drop the request?

blinkingled4• 14 years ago

Load balancers can insert cookies to identify clients uniquely. (That solves the proxy/single-ip-multiple-clients problem - multiple clients from one proxy IP can still be uniquely tracked this way.) The newer load balancers (F5/iRules for e.g.) are also capable of doing per resource, per client throttling using the inserted cookies or session table.

meej• 14 years ago

Hmmm, I had to turn off the LTE on my iPad two days ago because I hit my limit 10 days prior to my monthly renewal. I upgraded to 6.0.1 last week, but I wonder if this bug explains how I managed to chew through my data plan faster than ever before.

riobard2• 14 years ago

Hmm, so this explained my 3G data overage last month. I have 6GB/month of data, and usually I'd be using around only 700MB. But last month my iPhone 4S (running iOS 6.0) consumed nearly 8GB in total, resulted in quite some overage fee.

Should I ask Apple for a refund for this? :|

pk2200• 14 years ago

Same problem here. My monthly data usage increased by 10x after I upgraded to iOS 6.0. It's annoying that there's (apparently?) no mention of this issue in the 6.0.1 release notes.

frogpelt• 14 years ago

Maybe Apple was trying to stimulate the economy.

mintplant• 14 years ago

Try your data carrier, rather than Apple.

gigamike• 14 years ago

I had a similar experience. I normally consume 750-800 GB a month and on my first full month of iOS 6, Verizon said I consumed 8.89 GB. Luckily i'm grandfathered into an unlimited plan.

malyk2• 14 years ago

Last month I moved and didn't have an ISP at home for the first 3 weeks or so. I set up tethering on my phone and a 5GB data plan. Yesterday I got a notice that I was over my data limit. I haven't done a whole lot of web surfing in the new house, but I have listened to a ton of podcasts using the Podcast app which I have never done before.

So if I just blew through 5 GB of data in 2ish weeks and the only thing different was using the podcasts app...

(I did watch 3 episodes of walking dead on Netflix which obviously contributed to the data use, but a 45 minute episode can't be that big)

burriko• 14 years ago

Netflix's 720p streams use 2.3GB/hour, which would certainly put you over 5GB. I'm not sure about their lower quality streams though.

riobard2• 14 years ago

Interesting. I was using the Podcast app too, which I believed was the reason for the data overage. It shouldn't be, though. A few podcasts are definitely under 1GB. This AV Foundation Framework issue seemed to be the root cause.

bquarant2• 14 years ago

NoAgendaShow.com has been discussing this since iOS 6 launched. Don't upgrade.

berberich• 14 years ago

As pointed out above, 6.0.1 fixes this, so upgrading shouldn't be an issue.

bquarant2• 14 years ago

Good point. But then you have Apple Maps.

untog2• 14 years ago

This is where Apple's closed nature is very annoying. Assuming the article is correct, it was bug introduced in 6.0, and fixed in 6.0.1. So Apple knew about it, investigated it and published a fix.

Yet they didn't tell anyone about it. Or the crippling effect this bug could have on their data allowances. Why not? Are they really that terrified of admitting to making a mistake?

rachelbythebay2• 14 years ago

This same behavior has been happening to me since before iOS 6 came out. I can show the same "206 storm" happening as far back as August of last year with iOS 4.3.5.

randall• 14 years ago

I used to work at http://castfire.com/, and we've seen that similar issue from a LOT of iOS clients. Stats become increasingly difficult, because we saw one unique request bounce with upwards of 50+ 206s for similar / overlapping media... and then it bounced through a cell provider, to a WiFi network, and back, etc.

(we identified by redirecting all requests through a complex process of assigning an ID to them)

Hoping this is fixed. Podcast metrics are tricky, and this has been exacerbating the issue.

chrisrhoden3• 14 years ago

The playback stuff definitely makes use of many range requests, but what we found was that they were being used in a way that vaguely made sense in other versions (i.e. downloading a few hundred kb to continue buffering, pausing the download, and picking up again). With 6.0.0, we're seeing actual storms of these requests happening many times per second.

michael_miller• 14 years ago

To play devil's advocate, why would it matter? Is there any fix app developers could implement, short of blocking requests from iOS 6 users?

untog2• 14 years ago

It would at least communicate to users that the upgrade to 6.0.1 is very important if they use the Podcasts app. Many may have put off installing the update because they thought their phone was working fine. A bandwidth eating bug has to be along the least visible.. in the short term.

fernandezpablo2• 14 years ago

Makes you think twice about those graphics where iOS devices take a big piece of the pie of the "internet usage"

eridius• 14 years ago

No it doesn't. Those aren't tracking bytes streaming over the internet, but rather web browsers visiting pages.

fernandezpablo2• 14 years ago

Yes it does.

smackfu• 14 years ago

Even ignoring the bytes, it looks like one client may show up in the web logs as downloading the file a dozen times. If you aren't careful about cleaning up the stats, that may overrepresent iOS.

robomartin• 14 years ago

I have never considered Apple to be a solid software company. They are a marketing company in the tech sector. If they made washing machines they'd be sexy as hell. In my years of exposure to Apple tech I have always been in awe of their industrial design, mechanical design and manufacturing prowess. Their software, however, often leaves much to be desired. I say this both as a developer and a user. From OSX to iOS there are amazing patches of software manure in plain sight. And it isn't getting any better.

zizee2• 14 years ago

addlepate wrote 37 minutes ago | link [dead]

OK. I bet you use a WebKit browser.

addlepate: you were hell-banned 100 days ago (unfairly in my mind).

pretoriusB7• 14 years ago

Here's an easy clue: if your comments never get above or below 1 point for days, that means you are hell-banned.

pretoriusB7• 14 years ago

>I have never considered Apple to be a solid software company.

What does that even mean? That they make bad software? For one, they make the most popular certified UNIX OS in the world, and one of the two most popular mobile OSs. Their professional offerings are great too: Logic Pro, Aperture, Final Cut Pro, etc. In fact, FCPX aside, they are professional staples, with few competitors.

And huge number of desktop and the majority of mobile users use a browser they developed to the best game in town starting from humble open source origins (khtml -> webkit).

And that Clang thing, that FreeBSD recently adopted? Their work too, along with other LLVM infrastructure.

>Their software, however, often leaves much to be desired. I say this both as a developer and a user.

Care to mention any substantially better mobile OS than iOS?

(One might argue that Android is better. But substantially better, no way in hell).

devcpp• 14 years ago

Popularity means good software? Windows must be the best OS by very far then!

Regarding mobile OS, we all know this now depends on the number of "apps" available. Many may have made a superior OS, but adoption has been too low because of the lack of apps.

cageface2• 14 years ago

They bought Logic and most Logic fans will tell you that they've mostly just dumbed it down and dropped the price since then, turning it into "GarageBand Pro".

Not really much of a feather in their cap, particularly if you compare it to what the competition has been able to accomplish in the time timeframe.

Nikaido• 14 years ago

Aperture a professional staple? This is funny because I can see Adobe Lightroom installed everywhere but no aperture in sight. I'd say, the vast majority of pro photographers favor Lightroom.

huhtenberg• 14 years ago

You don't say.

All that effort that went into humanizing BSD is absolutely laughable.

kevingadd• 14 years ago

The iOS 6.0 podcast bug apparently caused huge overages for GiantBomb as well. They're discussing it right now on one of their live streams; it apparently was so bad that their corporate/finance people are upset with them.

Ensorceled2• 14 years ago

This is the reason Forstall has left the building, iOS 6 is a disaster:

    * Maps was not ready for prime time (so bad it required a public apology)
    * Podcast was not ready for prime time.
    * Passbook was not ready for prime time.
    * App store is a step backwards.
    * Battery and WIFI issues.
    * Soooo many bugs.

There are usually problems but this is the first time I've seen so much utter crap shipped including tons of minor bugs and issues (well Mobile Me was pretty bad :-)

I'm using Audible, Downcast and the built in Music app where I used to just use the Music app. iOS 6 was a huge step backwards for me.

Rather than waiting until it's ready (think copy/paste on iOS) Forstall was responsible for shipping a mess. Hopefully this will fix itself with new leadership.

gurkendoktor2• 14 years ago

How do you know that Forstall was the person advocating a yearly OS upgrade cycle? It seems just as likely to me that this rigid schedule was forced onto him and he couldn't deliver. Things like OS X Lion (a huge change under the hood) and the Maps places database just aren't things that can be built to a deadline.

Ensorceled2• 14 years ago

Well, a mess was shipped and he was the guy in charge. So I'm assuming he was the guy responsible. If a mess really was forced upon him he should have resigned instead of sticking around to ship the mess.

I have in that exact same situation, "This will not be associated with me, I'm out"

rachelbythebay2• 14 years ago

This just seems to be how they handle audio media. It's not even anything specific to podcasts. My own scanner stuff is just a web page, and it gets hit by this from time to time. Each call is a single file, it's static, and there's nothing funny going on with it. It looks like the playback stuff is jumping around in some attempt to figure out exactly what it is first. It seems like it would be far better if they always downloaded at least some reasonable quantity and then decided to bail out later if it seemed overly huge, but this is where we are.

To me, it feels like someone who's trying to be fancy and do things like seek() in a stream. Just because HTTP has progressed to the point where you can effectively do that with Range: requests doesn't mean it's a good idea. Offering seek() type behavior for a HTTP resource seems like a great way to get people to use it and not even realize what sort of messes they may be creating. Leaky abstractions, eh?

Here's just one example from last week. Note: the file is 7001 bytes. There's no reason for this insanity, and yet here it is.

    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"

chrisrhoden3• 14 years ago

Yep, as I mentioned in the article this is the AV Foundation framework - presumably used in all Apple apps since it's the official API they recommend. Doesn't surprise me that the browser would be affected.

The weird thing is that, when we first started doing the tests, it would happen immediately on pressing play. After some period of time running the tests, it started happening later and later in the process, indicating - what? No clue.

orcadk• 14 years ago

It's not just audio files; we've been seeing the same pattern (though actually even worse, as they request not only overlapping ranges, but the complete file, multiple times) on iOS5 for MP4 files: http://stackoverflow.com/questions/6094556/mobile-safari-mak...

erickhill• 14 years ago

I believe this is directly related to phone data usage overages as well. Having used an iPhone 4 on ATT for 2 years, I could almost guess my data usage perfectly each month.

By the 2nd day of using my new Verizon iP5, I was receiving warning texts from Verizon that I was about to exceed my data limits.

Thinking it was me, I increased my plan. That only stopped the bleeding for another few days.

Oh, and I installed the "fix" that was sent out to supposedly repair the known issue. (I was always on Wi-Fi networks but still being charged for data.)

I recently called Verizon to dispute the data. They quickly erased all of my data overages. Snap, like that. The man I spoke with called a technician on my behalf.

The people who can erase overages do not officially have this issue in their database as something to be aware of. But the technician he spoke with said there was absolutely a problem, and both Verizon and Apple knew about it and were working on it.

Then, I was given a "diamond ticket" to absolve me of any future overages until a fix is found.

andrewcastmate• 14 years ago

I don't have anything wildly new to add, but I'm part of a podcast hosting site (http://Castmate.fm) and we've seen this bigtime, it's crazy.

We host some high-traffic podcasts with a huge range of listeners, and I've gone through the logs and regularily see stuff like NUMEROUS listeners making 4000+ http requests in 30 minutes, all the 206 calls everyone here is reporting, it's just nuts, so bizarre.

I feel terrible for all the people getting hit with overages, it's so gross. I did some gorilla math the other day and feel pretty confident that this has boosted our network usage by ~25%, I should have narrowed it down to just the affected iOS 6.0 devices, must be astronomical.

kokey• 14 years ago

I am anxiously awaiting an update to the Podcast app which fixes the issues that I am having, but perhaps I'm too optimistic. I listen to some podcasts from the BBC and the app doesn't recognise that a new daily show with the same name is new, the only way around it I have found is to delete my subscription and manually find the podcast every morning. Apart from that the thing crashes and runs very slowly at times.

iOS 6.0 Causes CDN Overages

Related Stories