Facebook's Realtime Updates API has been down for 12 days

https://developers.facebook.com/bugs/738043236218053/

by jobeirne • 12 years ago

100 27 12 years ago

developers.facebook.com

striking• 12 years ago

The tone of the comments is actually really striking. It almost sounds like people are pleading with Facebook to get their act together.

masterleep• 12 years ago

New to FB platform work, eh?

mjn• 12 years ago

"Like the catcall of 'whore' or 'crook,' the Facebook Platform passes judgement before you even signed up for it. ... If you were innocent, why did you start using the Facebook Platform in the first place? ... Maybe it's me, maybe it's me, Facebook devs whisper quietly, alone, every Tuesday, before heaving the deep, lumbering sighs of resignation beyond sorrow."

http://www.bogost.com/blog/oauth_of_fealty.shtml

sheetjs• 12 years ago

Apparently you must have a facebook account to read this. Does the link point to a status page or a discussion?

bri3d2• 12 years ago

It links to a bug in Facebook's developer-facing/public bug reporting tool. Lots of angry comments about various products (mostly marketing related) depending on the API for updates, suggested workarounds (rewrite to query/pull rather than accept pushes), and an indication from Facebook that the fix will go out with this week's standard Facebook API iteration (they're released on Thursdays).

This is mostly of interest because at the recent F8 conference Facebook announced a "48 hour SLA" for "major bugs" and it appears this bug wasn't "major" enough to be fast-tracked.

bri3d2• 12 years ago

As noted in the comments, this is in stark contrast to Facebook's new SLA policy announced at F8, with the patch seemingly slated for release in Facebook's standard iteration rather than fast-tracked to "within 48 hours."

http://thenextweb.com/facebook/2014/04/30/facebook-announces...

It seems as though they're already playing the "it's not a major bug" triage game that most companies find themselves trapped in upon agreeing on an SLA.

Obviously depending on Facebook's APIs for your well-being has never been a good move, but this API seems particularly egregious to break as it's mostly used by branding/marketing managers, the very people who give Facebook their revenue stream.

zenogais• 12 years ago

You would be surprised. I worked on a marketing product for a major agency. This product heavily utilized the Facebook Ads API, and I can tell you that it is extremely fraught with problems. Often times we were left to work around issues in the API as fast-tracking a fix was near impossible even being a PMD.

CaveTech• 12 years ago

Ads API was and still is full of bugs and undocumented features.

psychometry2• 12 years ago

Depending on Facebook APIs for your well-being is only a good move if you're a total masochist. Easily the worst web API I've had the displeasure of having to code around.

weixiyen• 12 years ago

Move fast break things. It's both cliche and true. I wish they would treat their public API differently.

smacktoward• 12 years ago

In fairness, their new slogan is "move fast with stable infra," and an API that's down 100% of the time is certainly stable...

mattyohe• 12 years ago

Of course it is. “Move fast and break things”

rhizome• 12 years ago

They're just moving so fast it's impossible to see all of the moments where it's working.

psychometry2• 12 years ago

Don't forget the third and fourth tenets of Facebook API design: "Don't document anything, but if you must, do it poorly."

Curmudgel• 12 years ago

"Don't document anything, but if you must, do it poorly" is also sometimes called the "Unix Philosophy".

sergiotapia• 12 years ago

I -knew- something was off! My timeline hasn't been updating on facebook.com for a long time now. I have to refresh the page for it to load new items. This is a bummer, 12 days is a long time, especially for a company the size of Facebook.

rohandhruva• 12 years ago

This bug has nothing to do with timeline freshness: the bug affected developers and not users. If you're having issues with the timeline, please report an issue here: https://www.facebook.com/help/181495968648557

rlu• 12 years ago

That link is for reporting abusive content, not reporting bugs.

sicross2• 12 years ago

I'm the Product Manager on Facebook's API team.

On April 24th, it was first reported to us that the Realtime API was not sending updates when people comment or post on Pages. As of now, a fix is live and in production.

This was not an easy issue to diagnose the root cause of, and thus to determine how many people, developers and apps were affected. But while it took some time for us to diagnose, as soon as we did, we merged and pushed a fix within a few hours.

At f8, we announced an SLA of 48-hours for the highest priority bugs. We announced this because we believe stability matters. It matters to us because it matters to the people who build on our platform.

In this specific case, the bug with the Realtime API didn't prevent apps from accessing data about likes, posts and comments on pages - those were always queryable via the API on demand. That, along with our understanding that this issue affected only apps subscribed to updates on page content, meant we didn't, on this occasion, classify the bug as within the 48-hour SLA. However, we're constantly listening to our community, and may revisit how we classify and prioritize issues like this going forward.

Every bug matters. The very fact that a developer has reported an issue to us means it's affecting someone. It's on us to respond quickly, gauge how many users and apps are affected, and get to work fixing it.

We apologize for the disruption this issue has caused the affected apps, and are doing everything we can to reduce the chance of this happening again.

kungfooguru• 12 years ago

But while it took some time for us to diagnose, as soon as we did, we merged and pushed a fix within a few hours.

The comment in the thread says on May 1, "We have a fix for this issue and it will be pushed on or before our next weekly push next Tuesday"

But it didn't go out until May 5.

sicross2• 12 years ago

We initially thought we had a fix, but upon further investigation, we discovered a deeper issue. I agree, we could have been better at updating the bug with this information.

In the meantime, the severity of the issue became more obvious, which is why we accelerated the fix and deployed it today (May 5th) ahead of the regular weekly push scheduled for tomorrow (May 6th).

alttab• 12 years ago

On April 24th, we got one ticket out of a thousand like we do any other day. We fixed this bug like we do all bugs.

Shits fucked here, ok? We had no idea if you were some asshole who can't read a tutorial, or if you have some phantom dating app with 1,000s of users. When we finally got around to it, we fixed it quickly because we are boss.

When we lay down our new rules, we said we would get to the important things sometime this week. We want to demonstrate we have a process for our madness. You continually bitch about it so we know how important you feel it is for us to move slower and stabilize things.

You act like the API wasn't responding with data, or that it was all data all the time. It was some obscure content edge case that doesn't provide us any real direct revenue. How would you have expected us to react in a situation like this?

We know bugs are important. We don't open them on the weekends for fun anymore either. We get to decide whats important and how, and then fix it when we want to or can. We fixed the bug, and are going to find someone to blame about it so you can all stop worrying.

yeukhon2• 12 years ago

Not sure if this is satire parody or real negative FBer. Also not sure if such parody is needed on HN. I am not trying to defend FB, but they do have a lot of bugs. As I said in my comment below, I want to hear a postmortem.

kamobit• 12 years ago

Can't tell if this is a serious reply, but it seems to go against the conclusion of F8 where Mark said, "My goal for our culture over the next 10 years is to build a culture of loving the people that we serve that is as strong if not stronger than our culture of hacking at Facebook." [source: end of the article here http://goo.gl/iwWlck]

yeukhon2• 12 years ago

I am having trouble understanding what the problem was. I don't even think he explain anything...

In this specific case, the bug with the Realtime API didn't prevent apps from accessing data about likes, posts and comments on pages - those were always query-able via the API on demand. That, along with our understanding that this issue affected only apps subscribed to updates on page content, meant we didn't, on this occasion, classify the bug as within the 48-hour SLA. We're constantly listening to our community, and may revisit how we classify and prioritize issues like this going forward.

I'd like to hear a detail postmortem.

lurkinggrue• 12 years ago

Nice, At least it's down in realtime.

Facebook's Realtime Updates API has been down for 12 days

Related Stories