Flickr Accidentally Deletes a User's 4,000 Photos and Can't Get Them Back

http://www.observer.com/2011/tech/flickr-accidentally-deletes-users-4000-photos-and-cant-get-them-back

by jamesjyu • 15 years ago

141 91 15 years ago

Would someone really trust Yahoo/Flickr to all their images and not have backups? Hopefully they didn't really care much about the images, else they should have copies elsewhere.

pavel_lishin• 15 years ago

"Someone" would, yes - we're not all as smart and paranoid as you.

Anyway, the artist has originals backed up, but the article points out that all of the links to those images, and embeds, are now broken.

va1en0k• 15 years ago

that's a good example of URL being an important part of the data and not just a piece of metadata

If URL is constructed from the data somehow (auto-slug from the title, hash or something), such restoration becomes way simpler

hkuo• 15 years ago

I would imagine it's not so much about the loss of the photos as much as the loss of the social aspects of those 5 years, the comments made, the photos favorited, collections followed. Moreover, there's the uploading which would be a monumental effort to redo, including titles and captions.

scott_s• 15 years ago

Not only is what you said correct, it's from the second sentence of the article: Mirco Wilhelm has the original files saved elsewhere, but the photos from his extensive Flickr collection had been linked to from all over the web, including the official Flickr blog.

I am forced to conclude that few read the article, since most of the comments have focused on the - erroneous - idea that he depended on Flickr to store his data and had no backups.

user242• 15 years ago

> Would someone really trust Yahoo/Flickr to all their images

It's a paid service[1]. So yes.

[1] he had a pro account.

jat850• 15 years ago

I'm not siding with or against you or Flickr or anyone else here, just pointing this out: I see nothing in the FAQ or Upgrade page that indicates a Pro account ensures your files will be backed up.

(Point being, nothing site-wide in my quick glance seems to indicate a promise of the integrity and forever-ness of your data.)

VladRussian• 15 years ago

people belong to either of the 2 groups:

1. people who've been in the situation of lost data when somebody else responsible for its backup hadn't backed the data up.

2. people who don't belong to the first group.

After graduating from the second group into the first, i've always been classifying my data as either "can loose, fnack it" or "i'm backing it up myself, right now"

zipdog• 15 years ago

I know quite a few people who have lost important data, and yet still don't do proper back-ups (which is crazy given a 1 Tb drive is $70)

calloc• 15 years ago

Nope, and neither did the photographer that is being written about.

What isn't backupable, and thus is lost forever is all of the comments he had on the various pictures, all of the groups his pictures belonged to, all of the embed's of pictures on various sites, all of the social aspect of Flickr (such as friends, family, all of those things).

I use Flickr to share my photos with the world, but I keep a local copy of all of my pictures on my local file server and on my laptop and all of the backups I have of that laptop...

ebiester• 15 years ago

Isn't that on google? Or bing, since that's who yahoo uses. That way, you could get back all of the links. It would take some developer time, but that's exactly what's needed for PR.

jamesjyu• 15 years ago

Time for backupify.com?

iwwr2• 15 years ago

Back up your data on clay tablets for ultra-durable (thousand year) reliability.

gwern• 15 years ago

If something like the Rosetta Disk (http://rosettaproject.org/disk/concept/) made of Monel (http://en.wikipedia.org/wiki/Monel) were available at a non-insane price, I'd use it.

there2• 15 years ago

anyone know of anything like that for downloading the data locally? backing up data in the cloud to another place in the cloud just seems stupid.

i've seen some flickr-specific backup scripts that just dump all of the images to a directory, but they don't preserve titles, dates, sets, collections, tags, etc.

mechanical_fish• 15 years ago

backing up data in the cloud to another place in the cloud just seems stupid

Why?

The difference between backing up your Flickr data to an encrypted drive at, say, Amazon and backing it up to an encrypted drive under your bed is: Amazon has better physical security, and far better redundancy, because (assuming you use S3) the "drive" you save it on there is actually a replicated data store that spans multiple regions and datacenters.

The point of backups is to provide redundant storage that is relatively uncorrelated to the original. Once you move the data to different disks at a different company in (if you like) a different country, you've done a lot to solve that problem.

jgarmon2• 15 years ago

Backupify allows you to download the data you backup to their cloud servers onto your local PC, and the base level of the service (2GB online backup) is free.

lwhi3• 15 years ago

I think this is a huge problem with all cloud services.

As we start to rely on cloud providers to look after our data, we need to either become more educated (and proficient) by creating regular backups ourselves (which kind of makes the idea of managed cloud services defunct imo), or be in a position where we can individually sue for damages incurred by negligence on the part of the service company in question.

So many of these companies have liability clauses which negate all responsibility in situations where negligence leads to loss.

I think the current situation is crazy, considering the amount of responsibility and trust that's involved in using web-services which store and manage irreplaceable data.

d2viant• 15 years ago

You're right in your point about backups. These cloud services present quite a problem in the sense that they're structured to accommodate massive amounts of data. Often times this data is generated within the service itself with no means provided to extract it. So even if the user wanted to be responsible for backing up their data it's often times impossible. Moreover, even if the user wanted to AND had the means to do it, it could be a massive amount of data to sync up -- more than your average user is going to take the time to do.

iwwr2• 15 years ago

A single service, even a cloud, is still a single point of failure. Diversify your backup strategies.

Indemnification is likely going to push prices up further, making cloud non-competitive compared to traditional storage.

Still, I feel a lot of old technology is merely being relabeled. Today everything that stores data remotely is called a 'cloud'. We may yet see small NAS units relabeled as 'private micro-clouds".

lwhi3• 15 years ago

I agree - I've recently learned of the importance of diversification first hand.

However, I'd argue the ability to take legal action, would actually encourage service providers to diversify their own backup strategies and procedures. If a mistake becomes too costly to consider - I think less mistakes would be likely to occur.

electromagnetic• 15 years ago

I don't get how in todays day and age the person lost their own copies of their photos. All the photographs I have taken with my current camera are still on the memory stick, on my hard drive and in the cloud. Why? Because data storage is massive compared to your average cameras photo quality.

My camera holds about 2,000 pictures on the highest compression setting on an 8GB card. It's not like that amount of data is precious. I have hundreds of movies and thousands of TV episodes stored, why is it that this person couldn't keep their photos?

I hear a lot of stories like this, and being a writer I can only think "are they stupid?" Until recently everything I'd ever written, if stored in RTF and rar'd would still fit on a floppy disk. I think I've expanded to two.

I now have a flash drive for storage. However my basic method is to simply download and rar all my files from Docs, email it to myself through gmail and load it onto the USB drive. It leaves me backed up in 3 places and the original copy.

xyzzyb2• 15 years ago

The problem here is that the photographer lost data that he had no way of backing up. His photos are fine, his online collections, uris, and social networking are all gone and only existed on flickr's servers.

chaz• 15 years ago

Consequential damages aren't covered in these services' terms of use because nobody can afford to pay for them. Should Amazon be liable for Netflix's profits on EC2? If so, Netflix isn't paying Amazon nearly enough money.

angrycoder• 15 years ago

Well, there's the next startup idea, a backup service for cloud services.

There is a joke in there about putting a cloud in your cloud, but I am not sure how that meme goes.

jgarmon2• 15 years ago

That service exists: http://www.backupify.com/

catshirt2• 15 years ago

"This wasn't much compensation considering Flickr Pro costs $24.95 per year, and Mr. Wilhelm has already received a year's worth of Pro through his participation in some events and competitions.*"

what else could flickr have offered to compensate?

kgermino• 15 years ago

I would have asked for something closer to a full refund and possibly a longer, if not lifetime, term of free usage.

It's obviously expensive for Flickr to offer something like this but I mean they accidentally deleted 5 years of a paying customers work. I think a refund is the bare minimum.

vacri• 15 years ago

They have done that bare minimum. Flicker has already provided the guy with five years of hosting and linking - that doesn't just disappear because it's in the past. They provided the service that was paid for over those years.

Now they've made a mistake, and have refunded this years payment and added future access for free. They have done more than 'the bare minimum'. It might be good PR to refund previous years' fees, but they were at the time provided in good faith and competant manner; there is no 'fair exchange' reason to refund them.

catshirt2• 15 years ago

even a lifetime of flickr wouldn't satisfy the authors request for compensation. a few hundred in flickr dollars is not worth thousands of lost photos.

WestCoastJustin• 15 years ago

Seems like a funny policy to not backup production paying customers photos & metadata. Think of all the photos, comments, followers, etc that just got nuked! Seems like they rely on RAID or having multiple copies of the data spread around for disaster recovery. Sure it protects you from a server/raid/hardware failure but when a user/admin does something dumb it can really hurt!

Doing a quick google search turns up this gem:

Staff - heather says: "When an account is deleted, photos and any metadata associated with account is queued and deleted. If not instantaneous, it's removed within minutes."

[1] http://www.flickr.com/help/forum/47365/

cosmicray• 15 years ago

How about ... RAID isn't backup.

RAID is there for system interruption protection. RAID is there to keep flickr cruising along without interruption.

If they delete content from a volume covered by RAID, then all the mirrored copies go poof simultaneously.

What you are suggesting is a higher level content management system. Apparently (from the drift of the comments) that is something flickr doesn't want to get involved with.

buro9• 15 years ago

Yup, sounds like it.

They need to learn the mantra: "Replication is not backup, DROP DATABASE replicates pretty well.".

gnaffle2• 15 years ago

It seems like a _very_ good policy from a privacy perspective, if I accidentally uploaded some pictures that I really didn't want online (easy to do with an upload tool pointed at a directory), then I'd feel much more assured if Flickr would tell me that they didn't keep any copies of my pictures floating around "just in case".

Of course, deleting a complete account without any means of rectifying it sounds like a very bad idea and shouldn't have been possible. At the very least there should have been red warning signs flashing when the support person deleted a pro account with 4K+ pictures in it.

relix• 15 years ago

The article pointed out that a lot of services "deactivate" an account rather than delete it. I would suspect this common sense methodology is as obvious as hashing a password, but apparently, and unfortunately for the artist, it is not.

That's the biggest point I took from this. Removing the wrong account can always happen, but actually wiping data instantly without an invisible "grace period", or retrieval from backup, or anything else to get it back, that's just poor. Especially if it's a pro account.

It sounds to me like they didn't invest much time in the internal admin section. They probably delete a lot of accounts every day, and they've never had an issue with the wrong account being deleted? Or an after-the-fact clearing up of a misunderstanding? They don't start a dialog, at the very least only with pro-users, to get the other side of the story?

I hope something else went wrong here as well, and it's not the way they work normally.

jasonlotito• 15 years ago

> I would suspect this common sense methodology is as obvious as hashing a password, but apparently, and unfortunately for the artist, it is not.

A few days ago we had a big thread on Undo in software development, and a lot of people were suggesting Undo is very very difficult to implement with little benefit to most users.

Then you have articles like this where people are shocked that undos weren't in place.

gst2• 15 years ago

What a coincidence. My Flickr pro account expires on Feb. 15 and today I've decided to move away from Flickr and not renew my account.

What I've done in the short term is to use FlickrTouchr (https://github.com/tominsam/flickrtouchr) to retrieve my ~7 GB of photos from Flickr and upload them to Dropbox. The Dropbox gallery feature works quite well and has the advantage that you can easily backup/modify the underlying files.

In the long term I'd like to move the pictures to my own server (allowing me to use my own domain, etc.) and hack together a simple frontend similar to Dropbox' gallery.

If you decide to move away from a Flickr pro account (and if you also have non-public photos on your account) don't forget that after your account is switched back to a normal account you can only access the 200 most recent photos. So if you want to delete your photos from Flickr it's best to do so before the pro account expires.

pbhjpbhj• 15 years ago

>I'd like to move the pictures to my own server (allowing me to use my own domain, etc.) and hack together a simple frontend similar to Dropbox' gallery.

I think Opera's personal browser based server ("Unite") is going to have it's day (or something similar will) - http://www.opera.com/press/releases/2009/06/16/; has a "photo server" built in.

bigiain• 15 years ago

This is fine if you're just using Flickr as photo storage and publishing, but (especially for pro photographers) a big part of the reason to be on Flickr is the social networking features. If you've got people linking to your photos and photo stream, you can't easily duplicate that outside the Flickr ecosystem with your dropbox hosted photos.

geuis• 15 years ago

I keep hearing about this coming up, yet no one ask talked about why the data is irretrievable.

What it implies is that the internal tool(s) that the support staff uses was designed to do deletion instead of status changes. I imagine they have a web interface that fires off some backend script which a) runs sql deletes on an account ID and all tables and b) fires off scripts that physically deletes all photos across all servers.

Taking those implications in hand, it points to an incredibly poorly designed system for account management. Some years ago, there was an internal planning meeting where a decision was made to do this. Some people objected and were overruled. The ramifications of that project manager are still sticking it to people to this day.

I have a hard time believing that Flickr would have purposely engineered their systems in the manner I've described. Not impossible, but unlikely.

So I think its more likely that the data is still there, but that customer service has no method to resurrect it. Its probably a mix of internal policies and the lack of devoting a couple of engineers to fix the problem that is causing all of this.

potatolicious• 15 years ago

> "some backend script which a) runs sql deletes on an account ID and all tables and b) fires off scripts that physically deletes all photos across all servers."

Neither should happen. You never actually delete rows from your tables. You mark them as deleted, sure, but the data needs to be able to come back. Space is cheap, there are simply very few cases where outright deletion from a DB table is warranted.

I have a hard time believing Flickr would make a mistake like this. I agree with you - odds are if they actually let an engineer loose on this problem it'll be licked with a DB backfill in 20 minutes.

abscondment• 15 years ago

I bet that in cases of copyright infringement a logical delete isn't legally sufficient. Sure, you can leave the database pointers in place... but you're going to need to actually remove the infringing content. In that case, at least, a true delete tool would be needed. Like any Internet discussion about copyrights or other people's backend scripts, this is all purely speculative.

T-hawk2• 15 years ago

> You never actually delete rows from your tables. You mark them as deleted, sure, but the data needs to be able to come back. Space is cheap, there are simply very few cases where outright deletion from a DB table is warranted.

This comes with a cost, and it's not a small one. Every single query that ever touches this table, in every single piece of code owned by the company, written or maintained by every single developer (consultant or transient or offshore, none of which ever got training sessions on this requirement) needs to be enhanced to respect "where isDeleted <> true".

It's well understandable if the expected cost to recover after the fact from an occasional issue is smaller than the cost of implementing and maintaining that in the first place.

smackfu• 15 years ago

You never delete, and then people bitch that they can't really leave the service. See: Facebook.

rtghnyhjm2• 15 years ago

It's an engineering decision - Flikr has to host a large amount of data for very little cost, since most users aren't paying. Space isn't cheap by the Petabyte!

The design for a system where you had multiple redundant copies of all the data and could roll back any changes to the data set is different - and would cost a lot more.

sdizdar• 15 years ago

Here is what I don't understand. They have multiple redundant copies in case of unlikely scenario of disaster, but they have no copies or protection in case of quite likely scenario of operator error. This seems backward to me.

EDIT: I don't believe that Yahoo! Flickr does not have a way to recover these things. Or I'm completely wrong.

theDoug• 15 years ago

Indeed. I'd have sympathy for Flickr if this story didn't come up almost monthly in the last few years. Between this and the account deletions of those who spoke loudly over Flickr's other account deletions, more and more Flickr is becoming a hostile system.

The lack of a retrieval system could be from a lack of Yahoo wishing to invest in upgrades to the platform in general. The last few years have seen few changes for users beyond geotagging and tagging persons, and competitors (free and otherwise) have stepped far ahead in terms of personalisation and galleries.

I'm very curious to see if Flickr turns into another 'Delicious' situation in a couple years.

InclinedPlane4• 15 years ago

I think that danger is very real. I used to be a Flickr Pro member but I've been taking fewer pictures the last few years (for several reasons) and can't justify the Pro account anymore. However, the side affect of that is that now some of my older pictures are less accessible and my old photo sets aren't browseable. At least to me this seems like a bad business plan for Flickr, especially amidst an abundance of rabid competition.

citadrianne• 15 years ago

It sounds like Flickr is throwing engineers at the problem now--they responded with a statement which is in the updated post:

http://www.observer.com/2011/tech/flickr-accidentally-delete...

mmaunder• 15 years ago

"..but that customer service has no method to resurrect it".

Having seen this kind of effect from a brain drain in the past, I think you're exactly right. Software needs love to survive and this is what it looks like when software and a company becomes unloved.

callmeed• 15 years ago

Are you saying that a freemium service should never delete data—or that a customer service rep shouldn't be able to do it?

I can see why Flickr might engineer things this way. Most of their users don't pay (granted, this guy was a pro user). Deleting images when an account is deleted saves money.

random42• 15 years ago

How should the deletion system be designed?

zachallaun• 15 years ago

One can only hope that this is not quite over yet.

There's been quite a few well constructed and thoughtful apologies circulating HN lately, one of which was Andrew Mason's apology for Groupon's Japan blunder. Hopefully, Flickr will pick up the torch, and issue a more official, and hopefully more well thought out, apology. After all, it's been less than a day since the incident.

Of course, the deletion in itself is inexcusable. I just hope Flickr steps up.

wmboy• 15 years ago

I highly doubt the files are irretrievable. It's more likely it'd cost Yahoo a whole lot of time (or a large cost in getting specialist contractors) and they would rather just risk facing bad PR than coughing up the money.

Maybe they'll do the latter after some rethinking!

jamieb• 15 years ago

The guy has his backups. His problem is not the data. His problem is the human-networking side of things: lots of places linked to his photos (including the flickr blog) and those links now point to dead space. That will reflect badly on him, I'm sure, and not flickr.

If I were Yahoo, I dont care how much it would cost, I would have someone, or someones, recreate his account as it was, even if that meant calling in a data forensics team.

Bitmobrich• 15 years ago

No off site daily back up? I guess we can assume they have no disaster recovery plan?

jcromartie• 15 years ago

http://ascii.textfiles.com/archives/1717

ck2• 15 years ago

Someday Gmail is going to have a catastrophic loss that's beyond parity bit recovery.

leppie• 15 years ago

For $25 a year, I would expect them to have some backup policy!

rm-rf• 15 years ago

I wouldn't. Replication yes. A logical backup? no.

bl4k2• 15 years ago

This is why I never have a DELETE call anywhere in my app or admin code. I always mark as deleted (there are performance improvements in this for some databases as well)

danudey• 15 years ago

I've worked at several places where the strategy is 'never delete, just mark as deleted'. Your database tends to grow pretty fast, but you never have to defragment your index or tablespace.

bl4k2• 15 years ago

I usually setup purge tasks (copy to same table in another db) every week or fortnight in low-traffic periods

marcusEting• 15 years ago

Yahoo seems to be having trouble in general. First there are rumors that they are shutting down Delicious. Now their photo sharing service is starting to show real problems.

(Not to mention the Flickr API has a KNOWN ISSUE with the search that has a fix ETA of several WEEKS! http://www.flickr.com/help/forum/en-us/72157625560721827/)

delackner• 15 years ago

This whole situation points at a market opportunity for a competing service to offer users the ability to backup a real DB dump of their entire user, not just "everything you uploaded", but actually a file that, provided later to the service (or a competing service) contains everything needed to recreate that account.

duke_sam• 15 years ago

This is pretty awful. I have some sympathy for the support staff though, if their tooling allows them to irreversibly delete someones account (and it's history) they need better tooling. The realisation that they just deleted someones 5 year old Pro account and 4,000 photos... oh shiiiiii

gnaffle2• 15 years ago

Although this is a terrible mistake on Flickrs part, and it shouldn't have been possible to do this by accident, am I the only one that is happy that deleting a Flickr account really means _deleting_ it?

Most other services like Facebook, GMail are explicitly designed to keep your data forever, and if you don't want that, well, you should have thought of that before using the service. I would actually be happier and feel safer if we heard more complaints from Facebook users that accidentally deleted their account and couldn't get it back. It would be a clear sign that their system was designed with privacy and "data hygiene" in mind.

RuadhanMc• 15 years ago

4 year subscription to Flickr Pro? That's not even a year for every year of photos he has lost. I would think that if you deleted 5 years of a users photos the least you could do would be to give them a life time subscription to Pro and buy them the best damned camera available so that they can take new photos.

pilif• 15 years ago

The real dilemma is that as a service, you can't do it right. If you keep the data around and just mark it as deleted, the EFF will post articles about you violating the users rights for privacy and control of their data.

And if you delete the data, then $publication will write articles about you being unable to restore it.

Take the Facebook "account disable" feature as an example. In the context of this article here, it was sold as a "good" feature (so as something that Flickr should have too).

But this exact same feature was all over the press a year ago as something really bad and just showing how Facebook just doesn't want to let you go.

So. What is it now?

Personally, just flagging for deleted and living with the privacy concerns might be the better option because there are potentially more people deleting data by accident (or have it accidentally deleted) than there are people too concerned about privacy to use the service.

Also, users concerned about privacy just won't be users whereas users who just lost their data were users and now aren't, so that's a net loss compared to the users not joining.

fedup2• 15 years ago

Thing is, this is what happens to you if you complain at Flickr. They don't like having to actually do work. That's pretty obvious from the fact that after over five years of losing people's photos like this, they still fail to back up anything. The founder even lost all his stuff at that cluster-f of a social network gone wrong. What's funny about those slackers at Yahoo's Flickr is that they think they can pull off hosting a porn site, while pretending it's a family friendly place for your kids' photos. That may be fine for the average user that wacks off at work, but not so good for an advertising platform, as it turns out. You see, major corporations and small businesses alike have an issue with their ads being surreptitiously placed onto hardcore pornographic web pages without being told about that. Yahoo's Flickr can play this game of hide-the-porn while tricking the general public into trusting them. But those lies don't really fly in the advertising world on which Yahoo depends. Everyday, several people get deleted from Yahoo's Flickr as that company desperately attempts to make their porn site not really appear to be one on the surface. The copyright infringement is the same thing on a smaller scale. They count on people stealing your content, that's why they tricked you into placing it all online in an easily accessible catalog of stock images from trusting idiots. Make any kind complaint about the way they are doing anything, and you're booted out mercilessly. That's just the way it goes and Yahoo doesn't care one bit how you feel, because they obviously do whatever they want to. They have the government in their pocket and free reign to push porn into grade schools unlabeled, give your photos away for free to anyone that wants them without liability, and harbor countless sexual predators, pedophiles, and registered sex offenders, whom they cloak so they can be right next to your children without anyone being suspicious. Can't really see anything worthwhile about that website, or Yahoo in general. It's all lies from them, and everyone eats it up with a clueless smile.

Flickr Accidentally Deletes a User's 4,000 Photos and Can't Get Them Back

Related Stories