Optical storage – the future of long term data preservation (2015) [pdf]

https://www.snia.org/sites/default/orig/DSI2015/presentations/ColdStorage/DougFerguson_Optical_Storage_Future%20Long_Term_Data.pdf

by dsego • 5 years ago

44 40 5 years ago

snia.org

quercusa• 5 years ago

I worked in optical storage in the 90s. Then and now, the storage technology of the future!

nayuki2• 5 years ago

When the Compact Disc was released, the data capacity was way more than typical hard drives for many years. But by the time DVD rolled around, HDDs became denser and cheaper than optical storage could ever be.

alliao• 5 years ago

I got some 25GB bluray to record family photos, I do cull rather ruthlessly though.

lehi• 5 years ago

Page 13 prompts some questions. 128GB BDXL is positioned before the document's publishing year of 2015, but doesn't appear to have actually been released until 2017. The multi-TB discs extrapolated out to present day seem to still be vaporware. Why did advances in high-capacity bit-rot-resistant optical media apparently stall?

lastbitwritten2• 5 years ago

Downloading 100gb games over the internet became a thing, and h264 and h265 were efficient enough to prevent consumer video needing anything much larger than blu-ray.

Sony ODA is a thing, and uses 11 600gb discs in a compact cartridge to give around 6tb per cartridge.

rbanffy5• 5 years ago

I do not expect large-capacity optical media to become available for consumer use, but A 100GB BDXL can archive only 6 minutes of 8K video at 60 fps from a RED, so I would assume high-end visual production would want something better than 100GB per disk.

bobsmooth• 5 years ago

BDXL M-Discs are available but I imagine Sony didn't see a need for bigger Blu-Ray disks so there aren't ant M-Disc equivalents.

throw63738• 5 years ago

CDRs are most reliable way to backup important documents for several years.

jeffbee3• 5 years ago

Compared to microfilm? Surely not.

nayuki2• 5 years ago

The joke might be "several" years.

crmrc114• 5 years ago

I had a box of attic junk that had about 50cd-r discs with Various bits of data. Mostly old Linux ISO's - I can tell you that over ten years in an attic wiped Sony discs to nothing, damaged Maxwell discs and made Memorex discs auto eject. Various floppies from 1990 or later stored in the same box read just fine... So that blew my mind.

TheCondor• 5 years ago

What’s the attic like?

My cool dry Colorado basement (high temp is about 67) has been good and many disc from the 1990s are perfectly readable

mrjin• 5 years ago

Cannot agree more. I had around 300 CR-R/DVD-R and a couple dozen of DVD-RW disks, of various brands, mostly Sony, Verbatim, or Maxwell. Although those claimed to be able to sustain 10+ years, some even claimed to endure 50+ years, quite a few started to fade in 3 to 5 years and most of them failed when I last check after around 8 years. I might be able to recover a couple of them, but I would say 95% of them were long gone before I checked. It was quite a pity that some videos I archived on those disks were permanently lost. I knew those optical drives could easily become unreadable if the surface got scratched but I did not realize it could just fade away or I could have just keep those I don't want to lose on HDDs. I did not got a chance to check my floppies as my floppy drive got busted and there was really neither much point to get a new driver nor I could easily get one. Data on my HDDs is just fine, even after almost 15 years not been powered on.

Now I'm getting curios on how long my data can retain on a SSD without powering on it...

Amezarak• 5 years ago

I had a few dozen CDRs from a couple different manufacturers that I used as backup discs that sat in an air-conditioned dark closet for about five years. None of them were readable without errors. Today none of them are readable at all.

Hamuko• 5 years ago

Doesn't CD-Rs use organic materials unlike BDs?

KennyBlanken2• 5 years ago

I think the parent commenter is making a funny. CD-Rs were often unreadable within a few years even under ideal conditions. Put one out on your desk where sun from a window can reach it and you might not be able to read the disc in a few months.

Putting a label on a CD-R kills them extra fast.

jeffbee3• 5 years ago

What is the utility of cloud computing for archivists? Is it too expensive? It seems like the longevity is fairly proven at this point. If you put something in S3 in 2006, it's still there. GCS was launched over ten years ago, too. With a multi-cloud strategy you could probably make the case for durability, certainly as a 2nd or 3rd replica that just happens to be dramatically more convenient than optical discs.

Considering that many government records are now produced in the cloud anyway, it makes a certain sense. USGS elevation models for example are hosted on S3. Are they also on tape somewhere else?

rbanffy5• 5 years ago

Are they also on tape somewhere else?

I can assure you they are and that there must be a number of copies available for use on any respectable supercomputer.

It just may be that the canonical source of the data is, officially, the cloud and that they release the data by uploading it.

KennyBlanken2• 5 years ago

It is far too expensive due to the power. Remember that you're not keeping one drive per unit of data going, but multiple drives for redundancy. Then there's the power used by the fans, power supply losses, and then the datacenter cooling costs...plus all the network infrastructure and so on. Some of those are pretty small but it adds up to a lot to keep, say, 6TB work of data online vs in a LTO cartridge sitting on a shelf.

Drive manufacturers, storage vendors, and cloud providers have been working on lowering opex/TCO for years. Bigger and bigger drives, helium which reduces aerodynamic drag, tiered storage where stuff is moved to slower and slower spindles the longer it goes without being accessed, for example...in some cases automatically getting pushed to tape. But you need systems to manage all that, systems that take power, have support contracts, and introduce a lot of complexity.

Eventually flash memory will get cheap enough - but flash degrades with time, too.

jeffbee3• 5 years ago

Eh, but the whole point of the cloud is if you mix a cold storage workload on part of a disk and an active workload on another part of it then the marginal cost of the cold workload in terms of all that stuff you mentioned is almost zero.

wmf2• 5 years ago

https://blog.dshr.org/2019/02/cloud-for-preservation.html

selfhoster11• 5 years ago

Archivists need access to this data decades, if not a century down the line. I trust the Internet Archive to be around somehow for much longer than S3.

Kye2• 5 years ago

Yep. Companies eventually die. Sometimes they're reborn, like IBM, AOL, and Microsoft, but never with the same priorities. The Internet Archive is almost as old as Amazon and archiving is its entire reason for being. The pendulum will swing back as it always has and send things off big [mainframes, clouds], and S3 will be sunset in favor of serving whatever comes next.

VectorLock2• 5 years ago

Never heard of M-Disc (defunct) til I read this, and from the Wikipedia page on M-Disc about Syylex for "more durable" optical media.

Whats the potentially longest lasting optical media available to consumers today?

joe_the_user• 5 years ago

If your media lasts a million years but all the machines that can read it vanish in twenty, where's the win? I assume the evolved rats or bunny rabbits that follow us may reverse engineer it at some point in their evolution and then just shake their fuzzy heads.

Edit: Note, in the early nineties, a lot of library got sold proprietary optical drives as archiving solutions and when those became pink elephants, they wrote learned papers about "you can never what medium will last"

wmf2• 5 years ago

There have been various ideas about putting decoding instruction in human-readable format at the beginning of the media so that people in the future could use a simple magnifying glass to read the info that would allow them to build a device to read the rest of the data. Nobody cares enough to actually use that stuff though.

VectorLock2• 5 years ago

You can always make new drives or new ways to read media.

mkj• 5 years ago

It looks like the original M-Disc went broke but discs are still being made by Verbatim? That one looked more promising than other formats, at least it's readable by normal drives. Future terapixel cameras or scanners could probably decode a disc from an image, right?

Kye2• 5 years ago

Shiny discs need a laser in a dark, stable space with lots of cache and a purpose-built lens and sensor to read them reliably. I don't see a camera or a scanner having that reliability.

Every new pixel you want to render from a lens reveals more and more flaws. That's why older lenses often perform poorly on newer digital cameras: the designers didn't have to contend with 24, 50, 100... megapixel sensors. Even the 24 baseline already pushes what the film they make 4K remasters from can support and that's twice as big as what most film still cameras use.

Optical storage – the future of long term data preservation (2015) [pdf]

Related Stories