Newly recovered Ground Zero photos show why you should back up your CD-Rs now

https://www.theverge.com/2019/6/16/18678447/9-11-ground-zero-world-trade-center-pictures-recovered

by Tomte • 7 years ago

69 61 7 years ago

willtim2• 7 years ago

A huge shame. I backup my optical media to S3, but I also use M-DISC "mineral discs" which hopefully will last much longer. Sadly they were not available back then.

lloydde2• 7 years ago

Do you mind sharing your monthly S3 bill and it’s rate of change?

yourfate• 7 years ago

I can recommend wasabi.com, they are a lot less expensive but also offer a full S3 api.

willtim2• 7 years ago

It's around 10 dollars, but then I have less than 1 TB. I recently starting backing up my RAW files remotely, so this number is set to increase significantly (!) I am very interested in S3 alternatives. I haven't looked at the Amazon Glacier options either, but I suspect the retrieval times make it awkward to query/verify backups.

dvfjsdhgfv2• 7 years ago

I used to use S3 but quickly realized how expensive it is and switched to Hetzner. The more data you need to store and transfer, the more pronounced the difference is.

voltagex_• 7 years ago

What's Hetzner's redundancy like?

masklinn2• 7 years ago

Long-term archival / storage is not the use case for S3, that'd be more Glacier or B2's thing.

The storage itself is competitive with Hetzner's storage solutions (nextcloud or storage box), though you'd pay for traffic on top of that.

dvd-guy3• 7 years ago

Anecdotal evidence, but I've recently gone through my ancient CD collection (in so doing, I found Ubuntu 6.04 installers, AOL dial-up CDs and other relics) and archived them with GNU DDrescue. It had no problem reading all bar a handful of CDs from the mid-to-late 90s and early 00's - the failure rate was much lower than I expected. The failures were mostly completely visually identifiable too - the layers had started to separate.

I've also been backing up personal photos to DVD annually for quite a few years. Each disc contains encrypted Duplicity files, and a PAR2 archive to assist with any deep failures of the on-medium parity. I also usually include a disk or two of pure parity archives, to handle a total failure of any disk or two in the middle.

Each year I check the price of DVD vs. Blu-ray and when factoring in the cost of the burner DVD seems to stay just on top for $/GB. It doesn't help that I've still got spare DVDs lying around, and people basically give away the blanks these days.

This is not my only backup strategy, but it is the only one that is cold, read-only, and offline.

mehrdadn• 7 years ago

Each year I check the price of DVD vs. Blu-ray and when factoring in the cost of the burner DVD seems to stay just on top for $/GB. It doesn't help that I've still got spare DVDs lying around, and people basically give away the blanks these days.

Cost is one thing, but does the capacity not bother you too? DVDs are so small in comparison to BDs. Not to mention the time you have to waste switching discs (or reading/writing them, if relevant).

dvd-guy3• 7 years ago

Burning tens of DVDs is a little tiresome, but it is well enough scripted that it is easy to slowly swap them in and out while focusing on another task - the brief interruption is barely noticed.

It also means that the failure of a whole disk has a lesser impact on the whole.

Edit: that said, I do have a spindle of 100 CD-R that were given to me. I haven't had the motivation to use them up yet.

masklinn2• 7 years ago

It had no problem reading all bar a handful of CDs from the mid-to-late 90s and early 00's - the failure rate was much lower than I expected.The failures were mostly completely visually identifiable too - the layers had started to separate.

It was mostly cheap CD-R from that period, and frankly you needn't wait 20 years for them to fall apart, they'd start failing under 5. Quality CD-R from good brands survived just fine.

KozmoNau73• 7 years ago

I have a CD-R of "alternatively obtained software" from December 1995, which I keep around mostly out of curiosity. It's got some forgettable old games and MIDI sequencing software on it, nothing interesting at all.

But I'm wondering when it'll actually start deteriorating and ultimately fail. So far it seems to still be OK.

unimpressive• 7 years ago

I would honestly just copy it as an iso with something like dd.

rolal• 7 years ago

There is also https://en.wikipedia.org/wiki/Dvdisaster which can automatically create par2 parity when you write DVDs or Bluray. The other important thing to do is always write and verify, a lot of disc errors are there from day 1 and not from the passage of time.

calyth2018• 7 years ago

CDs? or CD-Rs?

Your Ubuntu 6.04 installer is probably burnt onto a CDR, unless you bought a pressed copy.

The AOL dial-up CDs are most certainly pressed CDs, and would have a far longer life.

I've seen bit rot in as little as 5 years on cheap CD-Rs.

dvd-guy3• 7 years ago

It was a mix of both - the Ubuntu 6.04 was in fact one of the freely distributed pressed ones, but many were home burned CD-Rs - photos, software, data backups, etc.

rlv-dan• 7 years ago

My most popular blog post is a test I made, reading 400 CDR:s of backed up data. Most interesting is people discussing their own strategies in the lengthy comments.

https://www.rlvision.com/blog/how-long-do-writable-cddvd-las...

pmlnr3• 7 years ago

Archival grade blu-ray or mdisc, those should last surprisingly long. Even normal blu-ray is ok, but the very cheap ones are not, those use organic dye. The rest is usually ~5-8 years.

llao• 7 years ago

Direct link to the source: https://twitter.com/textfiles/status/1138537238537625601

Direct link to the images: https://www.flickr.com/photos/textfiles/albums/7215770899728...

And what a title! The photos do not show the need, but the need is shown by some of the CD-Rs on which the photos are stored being faulty. Ground Zero refers to 9/11 here, not the nuclear bombing of Japan in WW2.

antocv• 7 years ago

I just keep all my archives in zfs mirror disks, which I every 4 years buy 2 new of and copy them to there. The old ones are not deleted, just stored offline.

Then I also have usb-memory sticks which I gave to family.

shellmayr• 7 years ago

I wonder about this: how do people archive their information, especially photo and video data. I use an array of mirrored HDDs in different locations with everything on it, plus parts of it on a cloud hosting service. It isn't very convenient though and it seems there should be a better solution. How do people in here do it? Most solutions (for me) fall short on quick retrieval and possibility to visually browse photos.

WalterBright6• 7 years ago

I buy new hard drives every year and copy all the data to them.

Edit: It's not just the longevity of the drive (though after 20 years they often don't work at all). Drive interfaces change over time, and I have drives where there is no way to hook them up to a newer computer, and my old computers failed due to capacitor problems.

For example, good luck trying to read a 5.25" floppy. You'll need to build your own hardware and write your own device driver to do it.

Just keep copying it all forward every year. Multi-terabyte hard drives are cheap.

westpfelia• 7 years ago

So this isnt a way to actively use a 5.25" floppy for storage... but if you have data on one you want to save this is an option:

http://www.cbmstuff.com/proddetail.php?prod=SCP

walrus01• 7 years ago

For example, good luck trying to read a 5.25" floppy. You'll need to build your own hardware and write your own device driver to do it.

It's really not that hard, assuming some older PC format. Motherboards up to the 486DX4/120 generation had onboard controllers for the cable to interface to a standard AT type, half height 5.25" drive. I had one in my PC at the time. They read 360KB disks fine too.

From that sort of computer, which can run DOS 6.22, it's not too hard to use it to transfer the data to a modern machine.

https://rover.ebay.com/rover/0/0/0?mpre=https%3A%2F%2Fwww.eb...

If you mean good luck trying to read a 5.25" disk due to magnetic degradation on the media,that's another problem.

tialaramex2• 7 years ago

The philosophy of just copying data makes sense, but your justification not so much.

Drive interfaces change over time, and I have drives where there is no way to hook them up to a newer computer

20+ year old PC drives will have IDE interfaces, which you can of course get a USB adaptor for very cheaply and I have several laying about. Really old PCs or non-PC hardware (e.g. the Amstrad CPC 6128 in the 1980s had an after market hard disk option) will need a different adaptor, but it's still just not that hard, and certainly shouldn't be categorised as "there is no way".

For example, good luck trying to read a 5.25" floppy. You'll need to build your own hardware and write your own device driver to do it.

You can easily buy refurbished 5.25" drives, and you can convert the data signal (which for a PC was basically the same as for 3.5" drives, but with a different physical connector) to USB. They're bulky and they need a lot of power at 12 volts, which is probably why off-the-shelf 5.25" USB floppy drives aren't a thing unlike for 3.5", but it won't require that you "build your own hardware" in any real sense.

You're only going to need to go writing device drivers if you've got some crazy disk format, and it'd have to be pretty crazy because the Catweasel already covered a lot of that territory (a Catweasel was a thing for Amigans, whose disks are a crazy format, to let them use PC disk drives to read their disks years ago, the last ones have an FPGA in them so that the uploadable firmware can decide how to control the drive).

WalterBright6• 7 years ago

In the early 80's, I had a PDP-11 with an 8" floppy drive. When I bought an IBM PC, I used Kermit to transfer my files to the PC over the serial cable. Fast forward 30 years, and I had put my PDP-11 Empire game up on github. Someone emailed me that it was missing a file. I was dead in the water, having gotten rid of the 11 long ago.

I recalled my roommate in college might have one. Called him up, he said he was about to discard it, I called him just in time! He said it was unlikely to work (been a long time since it was powered up), and unlikely to be able to read those ancient floppies, but was willing to try.

I had kept the floppies in the attic, and mailed them to him.

They all read perfectly! He sent me all the files on them, and bit images of the disks.

Thank you Shal, and thanks DEC for making machines and floppies that still worked.

https://github.com/DigitalMars/Empire-for-PDP-11

I wish I'd kept my 11.

dharmab• 7 years ago

A few of my co-workers have cross-site backups arranged with siblings or parents. Each house runs a few local servers and mirrors the backups to the other house.

This relies on having a trusted party outside of your natural disasters zone, but it's pretty cost effective if both parties wanted to run home servers anyway (e.g. Plex, homelab)

enriquto2• 7 years ago

Once a year, before christmas, you chose about a hundred of the best photos, print them in a beautiful album, send copies to your grandmothers, and delete the rest.

Fnoord• 7 years ago

I love this approach.

It features (with some overlaps):

1) Less is more.

2) Data hygiene.

3) Present for grandfather/grandmother/family.

The problem is that the more photos you made, the more difficult it becomes to pick a hundred. We leave behind so much data trails every day, including data which is no longer relevant. We constantly have to decide which data we keep or 'consume'. The problem is only becoming worse throughout the years (in history). Data is more easily available, and more easily able to be made (or fabricated!) now than 50 years ago.

tonyedgecombe2• 7 years ago

I've had a policy of keeping one photo a day and discarding the rest for some time now. It's quite freeing to not have to worry about keeping a huge digital archive.

KozmoNau73• 7 years ago

I store my important documents, music collection etc. locally on my PC, regularly synced to a NAS, automatically synced to pCloud, and every week I sync to an external hard drive that I keep in my locker at work.

Maybe it's overkill, but it really isn't much work to bring the disk home, fire off an rsync script and bring it to work again the next morning. And external hard drives are ridiculously cheap.

For photos, my phone uploads them to Google Photos automatically. I've been considering moving that to pCloud as well.

lrem3• 7 years ago

Processed photos and videos: Google Photos and YouTube. As an insider I know that I can't afford to match the availability of those and having the live display copy indestructible is soooo convenient. Downside: I don't think YouTube will give you back verbatim input material. Photos will, but it can be expensive.

Raw media material after processing: single local HDD and a remote raid.

Home directory: two local time machine HDDs and a remote raid copy of head.

Unpublished code: home directory + a private git server.

Published code: as unpublished + GitHub (which I assume to be indestructible too).

y4mi• 7 years ago

I agree that it's hard to beat the availability of Google consumer services... But that is not backup.

A backup serves as a fallback if for any reason the primary source has become unavailable.

In Google's case such an even can and does occur when your account gets banned because of some random event which you had hardly anything to do with or because your user was compromised.

A backup let's you recover from that and if you don't have that, I hope for your sake you don't mind losing the data too much, because the previous situations happen quite frequently if you actually listen on Twitter, Reddit or whatever social network you frequent.

mceachen• 7 years ago

Google Photos is most decidedly not a viable backup for photos and videos.

Google Takeout is broken or flakey, and had been for at least a year. Good luck downloading any archive before it falls with "transfer failed: bad authentication" 30 minutes into the transfer. Of 30 tarballs, only one actually completed successfully. And the images in the tarballs aren't the same bits that I uploaded (via my pixel, so it's supposed to be "original quality," not "high quality").

Pulling your files via the API results in corrupted metadata. Time and GPS location tags are stripped.

YouTube videos are, by design, reencoded with very low bitrates.

Cthulhu_• 7 years ago

It would be neat to have a service that converts digital photos to the equivalent of film or microfilm, an analog medium that can be converted back to full size pictures in a hundred years. It would have to be scalable and cheap though, given how easy it is to take pictures nowadays.

But yeah, spend $100 / year and get a big film roll of your photo archive, that would be neat.

ahje• 7 years ago

I have a small laptop running a few services for home use, incl. a DLNA server and network storage. Media dropped on a specific network drive will be available over DLNA.

Makes it easy to look at photos from computers and DLNA capable TV's, but it doesn't work for archival purposes as the laptop used as a server isn't really redundant.

Our own laptops are backed up using a cloud backup solution.

Tepix• 7 years ago

I use MDISC media for photos. I wonder what everyone's experiences are, especially for the 25GB and 100GB Blu-Ray variants - have you seen them fail? It'd be great if MDISC are indeed as great as they claim to be and I don't have to worry about copying the stuff over to some other media for 50 years or so.

PS: Yes, I have an external optical drive with USB 3.0 (and a USB A and USB C cable) to go with the media to be able to read them later

lrem3• 7 years ago

My USB Blu-ray drive failed after two years and new ones cost about four times more than they used to. I shrugged it off and added one more hard drive of redundancy instead.

pmlnr3• 7 years ago

While hard drive lasts, the trouble of disk based storage lies in the filesystem. I'd recommend running ZFS for long term storage.

Newly recovered Ground Zero photos show why you should back up your CD-Rs now

Related Stories