Facebook's e-mail debacle: One 'bug' fix, but rollback impossible

http://news.cnet.com/8301-1023_3-57465433-93/facebooks-e-mail-debacle-one-bug-fix-but-rollback-impossible/?part=rss&subj=news&tag=title

by wyclif • 14 years ago

175 111 14 years ago

news.cnet.com

51Cards2• 14 years ago

Sometimes it seems to me that FB doesn't have an impact assessment process. This is a very important data change you're making to 900-ish million records on a system that is inter-woven with and syncing to... everything. I understand bugs happening, but really, the cascade effect of this email change had to have been foreseen by someone... no? Bueller? Bueller?

I really don't mean to trash FB as a lot of good people work there. However... when I read about something like this, and then I see a company like Google figuring out a time smear function to seamlessly account for a leap second in their servers to avoid cascading system changes... it's hard to feel the two companies are functioning on the same level. (I know, a mis-balanced and irrelevant comparison, but it's just for a 'gut feeling' sense)

rhizome10• 14 years ago

They knew what they were doing. Not having a complete rollback was intentional.

eli2• 14 years ago

I believe Facebook is capable of doing sneaky things to advance their business goals, but Hanlon's razor [1] comes to mind here.

[1] http://www.jargon.net/jargonfile/h/HanlonsRazor.html

j_s9• 14 years ago

Only Violet Blue has said that they don't have a complete rollback as best I can tell from the CNET article.

I would guess the only thing they can't undo would be the time that has passed. If any data Facebook ever had on any of their users (such as their primary email address [edit: at any point in time]) was ever lost, they wouldn't have as much to sell to their customers!

Read me blathering on further about this: http://news.ycombinator.com/item?id=4192299

tedunangst• 14 years ago

google buzz?

dasil003• 14 years ago

> and then I see a company like Google figuring out a time smear function to seamlessly account for a leap second in their servers to avoid cascading system changes... it's hard to feel the two companies are functioning on the same level.

Sorta makes you wish Google would stop chasing Facebook and just do their thing.

bgilroy26• 14 years ago

Connecting site recommendations to user profiles has the potential to produce better, unspammable Google search results.

Google cannot allow other companies to get ahead of them on social search and remain competitive in their most important vertical.

ChuckMcM2• 14 years ago

Ouch. Talk about not thinking it through. Relating back to a common topic here, one of the things that distinguishes between senior engineers and not senior engineers in my experience is the ability to visualize both secondary and tertiary consequences of a particular change to the system. They stop and say "Ok, we change this, that means ..." and then walk forward past the change in their mind exploring the ramifications. In order to do that they have to have a broad understanding of the system involved from both a technical and operational perspective. They come back with "Oh if we do that then we'll get this effect and that would be bad, how can we prevent that from happening?" and if the answer was "We can't." then they spike the feature.

One of the most frustrating things for me as an engineer at Sun early on was bouncing some whiz bang idea off Bill Joy only to get one of his one line zinger 'but that breaks X' type replies. Always annoying but often right as he would take what was proposed, extrapolate two, three, or a half dozen steps and then point out the now 'obvious' flaw.

rhubarbquid3• 14 years ago

"move fast and break things"

Schwolop• 14 years ago

Let's hope that changes to "hurry up and fix things".

jmspring• 14 years ago

Facebook certainly has a reputation in engineering circles as following the "move fast and break things" model or touting "eventual consistency" or similar. That works for certain areas, as others have pointed out, but once you start manipulating customer data that they have set up, it crosses a line.

I'm a strong believer in confining Windows to a fixed set of resources -- i.e., a VM with fixed memory and disk space. This incident, though I don't seem to be directly impacted (checked contacts, account settings, etc), Facebook has crossed that line into Windows equivalency for me. It needs to be confined where possible.

Mobile app deleted on all my iOS devices and FB account disassociated on my WP7 device (most used for testing).

codelust• 14 years ago

This.

The problem is not limited to engineering. I have often had to push back with product guys responding to feature requests from users to address edge cases.

The trouble with doing so in social networks is that the potential for the law of untended consequences to strike is much greater in it as it is nearly impossible to test for every permutation and combination available.

rhizome10• 14 years ago

If, as described elsewhere, the overwriting is due to using a specific and common API, the only unintended consequence is user reaction. No need to test for every possibility there.

SimHacker• 14 years ago

If only somebody's bounced the idea of Solaris off of him... ;(

ChuckMcM2• 14 years ago

Sadly Solaris, aka the 'Lulu' project, aka the merging of SunOS and System V, was in fact the brain child of Eric Schmidt and Bill Joy. They stood on the podium and announced it together in October 1989. Folks would have noticed if the stock market hadn't gone into free fall :-)

dmethvin• 14 years ago

"Move fast and break things" should still require that you think through what you're doing. The problem seems to be that the complexity of Facebook's system is beyond the understanding of anyone who reviews these changes.

jerrya• 14 years ago

"Move fast and break things" should refer to your servers and your data, not your customers and users devices and their data.

tobtoh• 14 years ago

I'm pretty sure that FB knew exactly what they were doing. They did think things through and decided the the short term PR loss was worth the benefits of gaining access to the email of 90% of their user base.

alan_cx• 14 years ago

Exactly. Users are stupid. Not because they cant read minds and find hidden features, but because they take it.

Its the same thing with things like internet censorship. Every one bleats on and on, and they just do it any way.

dserodio2• 14 years ago

"Move fast and break things" applies to their website. They may break facebook.com all they want, but breaking MY contact list is unacceptable.

pixelcort• 14 years ago

Quoting:

> Today Facebook also admitted that its API for contact sync on phone and device apps was set -- on which devices and systems we're not told -- to take an individual's most recently added e-mail address and overwrite their correct contact e-mail in everyone's address books with the new e-mail.

and

>> Since Facebook changed the email addresses on contacts, I've actually lost every single one of my email addresses including those for work.

From this article it sounds like actual data loss occurred on some address books, where non-facebook.com email addresses were permanently deleted from contacts. For those people who don't have a backup of their address books, it might now be impossible to recover those lost email addresses.

Isn't this more worrisome than any issues around how Facebook handles incoming messages?

TeMPOraL2• 14 years ago

It is. Everything else seemed to me to be whining about how someone who saw you on Facebook will maybe not be able to contact you because your default mail is a Facebook one (but why would anyone contact you by mail if Facebook messages are better? They both a) show notifications on FB and b) send notification e-mails to your real address).

But this, this is real, and worrisome, and it has done real damage, and it will be damn hard if not impossible to reverse.

s_henry_paulson5• 14 years ago

So Facebook messages are better because instead of getting your e-mail, you now have to jump through hoops to read it?

Got it.

reneherse2• 14 years ago

More good reasons to never:

a) Associate your phone number with your FB account.

b) Download the FB app.

The more you can keep Facebook sandboxed on your devices (and in your life) the better.

JaggedJax• 14 years ago

I've been using "Tinfoil for Facebook" on Android for this very reason. https://play.google.com/store/apps/details?id=com.danvelazco...

bsg75• 14 years ago

c) Use FB

Ogre• 14 years ago

c) Associate your real email address with your FB account

Facebook gets an email address I don't use anywhere else. No one has me listed in their contacts under the only email address facebook knows for me (well, I presume. It's not hard to guess the address I use for facebook if you know my real address, but there's no good reason anyone but facebook would use it).

I've done that on general principle for more than a decade any time I have to give my email address to anyone other than an individual. This instance is the first time in a while where it's actually felt worthwhile. Originally I did it more for spam tracking. GMail's excellent spam filter fixed that problem for me years ago. But I never stopped doing it.

reneherse2• 14 years ago

Indeed, I do the same.

And I'll also add: d) Never associate your real name with your FB account.

I use my first name plus a family name that matches my middle initial. It's generally recognizable and accepted by friends.

When I first set up my account this way (items A through D) it did seem a little "tinfoilly", and this last measure still does to some extent. But since Facebook doesn't actually let you delete your account, I'd rather err on the side of prudence. It will be nice to at least approximate a hole in their databases if their policies ever get too onerous.

petitmiam2• 14 years ago

I wish I could go back in time and do this.

marcuspovey• 14 years ago

If you use the FB app, it will associate your mobile phone number with your account (and make it publicly visible unless you change this default). Remove it, and then a few weeks/months later it'll magically appear.

Basically, just don't use the app.

ams6110• 14 years ago

Here's what to do. Open /etc/hosts (or %WINDOWS%/system32/etc) and add the following lines:

  127.0.0.1  www.facebook.com
  127.0.0.1  facebook.com
  127.0.0.1  connect.facebook.net
  127.0.0.1  facebook.net
  127.0.0.1  fbcdn.net
  127.0.0.1  www.fbcdn.net

j_s9• 14 years ago

Can anyone point to a Facebook representative saying that rollback is impossible? (I don't care at all what the CNET author says.) It seems like they should just put everything back to the email address they had before it was auto-updated to @ facebook.com. Then their apps would put them back to what they were on people's phones.

Lastly, if they were truly going to fix the problem, they would pump through all the emails they got a chance to spy on, this time to the correct address, maybe with an attachment or sending the email as an attachment with their official apology.

Can't rollback the delay and that impacts a lot, but email is theoretically unreliable anyway. Hurry up guys and gals... show me someone at Facebook actually cares about the product (their users) they are selling to their customers (their advertisers).

51Cards2• 14 years ago

A couple issues pop to mind. My understanding was that this update took a week(+?) to roll out and may still be in progress. It could take them awhile to even start to roll back. Then if you do roll back what do you do about all the people that have changed their settings in the meantime? They have updated their email addresses, perhaps with different ones, so do those get over-written with the old data in all cases? I'm sure it could be figured out but how timely it would be is questionable... not to mention that FB may still be trying to salvage the bump in use for @Facebook.com they were aiming for in the beginning.

j_s9• 14 years ago

I agree it's not a simple fix. Since they have everything stored, they could act as though the address had never been changed. (Primary email should be most recent non-auto-change. Email should be sent to value of primary email address [without auto-change] as configured when email arrived.)

This mistake/bug/zuck-style enhancement has had a colossal negative impact on many people. If Facebook tries to salvage anything from it then it's even more clearly time to walk away. I am not cheerleading for Facebook at all on this, I just want people's contacts back to normal and their emails delivered to where they want them. I think we agree this is possible, in fairly direct contradiction to the title of this article.

rhubarbquid3• 14 years ago

It's a metaphor. She's saying they can't undo the damage they've already done, not that they can't literally change the code (or email addresses) back.

j_s9• 14 years ago

Thanks. I don't want them to get away with not trying.

jaylevitt• 14 years ago

There's no reason to assume that the old addresses in people's address books were the same ones Facebook had as primaries. If my co-worker was in my phone as tom@patientslikeme.com, but signed up for Facebook as tom@gmail.com, and Facebook tries to roll back? Data loss still, and possibly even a privacy leak.

j_s9• 14 years ago

I was basing this on the fact that the facebook app/sync would have pushed tom@gmail.com into the contact info in the same way that it is now in trouble for overwriting with @facebook.com.

Thanks for bringing this up as it would not work if they only push changes, and no change had been made facebook-side prior to the 'bug'. In that case, they should only rollback those that had been pushed out and the majority of email addresses would be lost. However, since users can go into the settings and restore their primary email address, it seems like it would be best for Facebook to encourage them to do so in the same way they've nagged me to switch to timeline forever.

antidoh2• 14 years ago

"Apple might be able to salvage the relationship with Facebook, but it's pretty clear that Facebook's relationship with users has hit the rocks."

I very much doubt that. Statistically no one will leave FBook over this.

pdonis• 14 years ago

>>> Statistically no one will leave FBook over this.

If that's really true, it's very unsettling. If this doesn't trip a significant number of users' "not worth it" detectors, what would?

alttab• 14 years ago

Maybe we see that 38 dollar stock price now.

tomrod• 14 years ago

And thus I feel compelled to start a discussion of the German zeitgeist in the mid 1930s, thus causing myself to bring about Godwin's law.

blhack• 14 years ago

Um. Hi. I use facebook, this pissed me off.

Then I changed my email address back to what I want it to be, and I have ceased caring. I will continue using facebook to coordinate bar meetups with my friends.

karlmdavis• 14 years ago

No, I won't leave Facebook entirely over this, but I did just uninstall their application on my Android device. I simply don't trust them to have access to my contact list.

Frankly, I didn't trust them much before: I'm sure they were mining it, which is not something I would explicitly consent to. However, now I no longer trust them to not break it, so I've removed their access to it entirely.

catenate• 14 years ago

I see they actually created a way to remove the FB app from an Android device. Back when they hoovered up phone contacts into their database, I tried to break the connection, but all I could do at the time was delete my entire facebook account. Been pretty happy not wasting my time there, as I see it devolving into pseudonymous games, meme one-upmanship, and picture albums kids will mostly later regret.

mcherm• 14 years ago

That assumes you CAN uninstall it. Many devices come with Facebook preinstalled and not removable (because Facebook worked a deal with the device manufacturer or the cell company).

mbreese• 14 years ago

It will be are for people to completely leave Facebook. What Facebook needs to worry about though is that people will stop using the service as much. People may be less likely to integrate Facebook apps on their phones.

All of this could lead people to use FB less, which is is exactly the opposite of what they were intending.

petitmiam2• 14 years ago

I don't have the entire Facebook userbase added as a friend, but I haven't seen a single post on my newsfeed about this problem. Yet every second day I see posts about how much someone hates Timeline.

s_henry_paulson5• 14 years ago

While people might not delete their accounts en-masse, this does hurt them.

When you upset your users, you drive them away. They use your product less, and associate with you negatively.

The more this happens, the more people stop posting and sharing, and everyone simply stops using the site.

Remember most people on MySpace didn't delete their accounts, they just stopped using the service.

uptown• 14 years ago

I used to have an Android device, and let the thing sync my Facebook contacts into my phone. What I wound up with was a mix of people I'm actually friends with, and also a big collection on "Facebook friends" ... those people I'll never ever call, but knew from back-in-the-day and wanted to connect with on Facebook. I don't need them in my phone today because the extend of my correspondence with them will be what I consider "two-way read-only". They can look at my FB stuff. I can look at their FB stuff, but that's pretty much the extent of our relationship.

I'm on an iPhone now, and know that iOS6 will give users the same option ... to link your device w/ Facebook ... sucking in all your "FB friends". I won't be enabling that feature, and while the software apparently indicates you can disable this part of the connectivity, I'm reluctant to even link it for fear that it'll immediately hose my contacts in a way that requires me to manually clean out that kid I went to middle-school with many years ago.

leeoniya• 14 years ago

i uninstalled FB app about 8 months ago, never used sync on it anyways since i use FB more like a glorified twitter. been using mobile site since then (which is quite good), posting photos by sending an email to a special status address. so much more privacy. if they ever get rid of the mobile site to force me to install the app, byebye facebook on mobile.

acchow• 14 years ago

I can't stop my @facebook.com address from appearing on my timeline. The option shows up but clicking on it does nothing. Similarly, I can't make my @gmail.com address show up on my timeline. The mouse listener is also missing.

ricardobeat• 14 years ago

I'm still confused. Did the address book trashing affect only iOS 6 beta users?

s_henry_paulson5• 14 years ago

It affected anyone on Android and Blackberry using contact sync, IOS6, and Windows Phone 8

http://www.bbc.com/news/technology-18687044

AznHisoka• 14 years ago

There were so many obvious problems with associating a @facebook with everyone's account. As soon as I heard the news, I thought.. hmm.. someone could easily scrape 1 million facebook usernames, and be able to send 1 million emails to all of them using their facebook email address.

And make it all uber-personal because they know their full name AND all their friend's names AND what they liked AND their hometown AND the college they went to, etc, etc,. That's a spammer's wet dream.

entropy_2• 14 years ago

The @facebook was always there and could always have been used to reach a person's FB messages via email. Also, most of that other information would probably be marked as only available to friends or friends of friends. So you wouldn't be able to scrape it.

Don't get me wrong, this is a horrible move by facebook, but spam isn't the reason why.

bschlinker• 14 years ago

"Facebook is indeed intercepting messages -- the e-mail is clearly passing through Facebook's servers, but this seems to be where it stays, as neither sender nor receiver are getting a copy."

Is this really appropriate use of the word "intercepting"? Typically intercepting carries a negative meaning -- like "the FBI is intercepting my postal mail and opening it before it reaches me". However, with their use of the word, every email I send is "intercepted" by multiple servers before it reaches its destination....

j_s9• 14 years ago

Facebook is intercepting messages by replacing the destination address (which email clients often do not readily display [opting instead for full names, even photos, etc.]), and now claims to have done so inadvertently. Multiple servers relay messages before they reach their intended destination, yes, but that is by design.

To continue stretching your analogy, I could claim that the post office x-ray/bomb scanners are 'intercepting' mail sent to me when in fact it is an understood part of the process to anyone who cares enough to research how things work. Someone moving my mailbox [edit: filling out a 'change of address'] and photocopying everything sent there is pretty clearly a reasonable use of the word 'intercepting'.

melvinmt• 14 years ago

Let's say 1% of all their users is fed up because of this and 0.1% actually closes their account. This leaves Zuckerberg with 98.9% control over e-mail traffic from 900+ million users. Well played, Zuck, well played.

entropy_2• 14 years ago

Couldn't this be construed as an abuse of a monopoly(antitrust)? They have a monopoly on social networking(there are alternatives, but they are very very small in comparison) and they are leveraging that to expand massively in another sphere, namely as email providers.

reidmain2• 14 years ago

"Since Facebook changed the email addresses on contacts, I've actually lost every single one of my email addresses including those for work." -Complaint from iOS 6 user

My suggestion, don't use the very first beta release of a operating system on anything important. Digital darwinism at work there.

kalleboo• 14 years ago

Indeed, Apple have a very explicit message on the download page for all iOS betas to not use it on a device that's not solely dedicated to testing.

s_henry_paulson5• 14 years ago

What about all the Android and Blackberry users who had their addresses replaced?

What's the lesson there?

reidmain2• 14 years ago

Oh that is Facebook screwing up. I'm not trying to pardon that. I just hate people who complain about stuff when they take a well defined risk and it backfires on them.

danso• 14 years ago

I like FB so I won't be doing one of those "hey everyone, look at me I'm quitting FB" acts. But I highly recommend that you don't put FB on your phone...it's amazing how even a five second glance to check for new notifications will ruin your train of thought. And also, not having it mess up your phone account is nice

dm8• 14 years ago

Why don't FB openly admit their mistake on their engineering blogs? At least they can salvage already messy situation

Devilboy2• 14 years ago

They don't see it as a mistake.

rhizome10• 14 years ago

Or a mess.

SoftwareMaven• 14 years ago

Given FB is already having a hard time generating revenue from mobile clients, doing things that make mobile users afraid to use their service seems like a really bad idea.

wutbrodo• 14 years ago

Well, given that they can't monetize mobile usage and they can monetize desktop usage (if somewhat poorly), technically driving users back to desktop usage from mobile would be good for them.

andrewfelix• 14 years ago

I've been using a great work around to the numerous Facebook issues, including the lack of meaningful interaction with Friends and Family; I use a combination of email, telephone and physical interaction to maintain and form social connections.

luminaobscura• 14 years ago

this is why i have never sync'ed to fb or hotmail.

febeling• 14 years ago

There is the product life-cycle which has the phases: introduction, growth, maturity, and decline.

The point is that facebook needs to understand that "Move fast, and break things" is an appropriate approach in the earlier phases, but it is quite incompatible with later phases, when people start to rely on a service.

dsirijus• 14 years ago

Of course, they do thoroughly test their new ad options, limiting them to region first, then measuring impact.

grey-area• 14 years ago

The problem here is not one of competence or failure to think through implications. Facebook know exactly what they are doing, and you can be sure the potential implications of these changes were debated at length within the company at a high level. They were taken precisely because of their implications for non-technical users, not in spite of them - FB has replaced users contact details because they want Facebook to be the only way most people can contact others online.

Facebook have always wanted to replace the internet entirely with a system they control. Hence the walled garden approach with incessant prompts to signup to view content, the use of address book import to spread virally, the use of beacon to track and announce users activities, the use of like button js to track activity, the launch of Facebook apps and single sign-on to attempt to corral web developers within their ecosystem and make them dependent, the launch of a competitor to email/IM etc etc.

This latest move with email is simply the latest in a long line of moves consistently attempting to steer users into spending all of their online time within the confines of Facebook, it is entirely consistent with their past behaviour and we should expect similar moves in future.

So the arguments about how they could have done this differently, or whether Facebook has made some mistakes here are largely irrelevant - they will always push to own your data, email, photos, and digital life, because it's in their DNA, this is not a mistake, it's a pattern of behaviour - they can and will control your digital life if you choose to hand it over to them. Consider the quote from FB on the way their FB email works:

Regarding the "email loss" this may actually just be confusion around the Messages Inbox: By default, messages from friends or friends of friends go into your Inbox. Everything else goes to your Other folder.

All messages which do not come from FB contacts go into a folder which is effectively a junk folder - they want you to use their email for everything (hence the recent changes), they want your address book to contain nothing but FB contacts, they want you to forget email even existed, and they want you to encourage your friends to join FB so that you actually receive their messages. That won't work for all their users of course, but their hope is that they keep enough of the user base on board that they can dominate the internet and replace email with FB and the web with a network of sites where they control login and user info and all roads lead to FB.

So replacing email with FB mail wherever they can, hiding other contacts and trashing user address books is not a mistake, it's part of a broader strategy, one which is not in the best interests of their users, but which they hope most will go along with anyway.

statictype• 14 years ago

I deleted the Facebook app from my phone a long time ago and I only use it from an incognito browser window. So that's how much I trust their reach. But I find it a little incredulous that Facebook, in their quest to control all their users's mail would deliberately overwrite email addresses in users' contact lists. It seems like the kind of dumb plan for world domination that only a caricature of an evil villain would execute.

j_s9• 14 years ago

How many times must the same type of thing occur before Hanlon's razor falls to Occam's? :)

rhizome10• 14 years ago

How can you cast it as a "dumb plan," by "caricatures," when it succeeds?

GBKS• 14 years ago

Now what happens if other sites and apps do the same? Will our address books change every day and contacts fill up with dozens of unused email addresses? Many of Facebook's growth techniques have been imitated by other startups so going into the territory of manipulating data that has always been off-limits is a big step. If another social site uses those techniques against to manipulate data on Facebook profiles, will Facebook change it's mind?

gitarr• 14 years ago

If lawyers are reading this, a question:

Our governments are treating our emails more and more like real mail (aka. letters).

Is there not a law prohibiting third party people or companies to mess with(fake) my mail addresses?

Fact is, I do not want a facebook email address. And I have given no consent for my email address to be changed.

rhizome10• 14 years ago

It's more like someone sending a change-of-address card to the post office to have your mail delivered to their Mailboxes Etc. store.

tfb2• 14 years ago

That's actually a really good analogy. And the GP is right. There should definitely be some laws to prohibit these kinds of things that Facebook is attempting (doing), now that electronic mail/communication is becoming standard.

crazygringo2• 14 years ago

So this whole debacle has taught me that I have an "Other" folder, which I never saw before, and it turns out I have 20+ messages from people from the past few months which I didn't know about!

I'm trying to find a way to get Facebook to send me an e-mail notification when I receive a message that goes into "Other", the same way it sends me an e-mail notification when I receive a message that goes into "Inbox".

But I can't find any option for this anywhere. Does anyone know? Otherwise, how am I ever supposed to know I have a message from someone on Facebook who I'm not already friends with?!

crazygringo2• 14 years ago

Wow, this is infuriating. According to Slate:

So how can you make sure you find your lost laptop, appear on Israeli TV, and respond immediately to job offers? Unfortunately, the Facebook rep informed me, you can’t change your settings to get email notifications for your Other messages, the way you can for your main messages. Your best bet: Make checking the Other tab part of your daily Facebook routine.

So since I never log into Facebook, but use it as a place to receive messages, which I find out about via e-mail... I am doomed to never find out about possibly important messages from people who are not already my Facebook friends.

This is possibly the worst design decision I've ever come across on the Internet. Facebook has created a black hole for messages. Even when you log into Facebook, it doesn't show a number next to "Other", so you can't even find out if you have messages there without navigating to the "Other" page.

I can't wrap my mind around how insanely stupid this is. WTF, Facebook?

Tyr42• 14 years ago

Time to use selenium [1] to automate it and send you emails?

[1] http://seleniumhq.org/

mariuolo• 14 years ago

Why anyone would delegate handling of their address book to Facebook is beyond me.

Facebook's e-mail debacle: One 'bug' fix, but rollback impossible

Related Stories