Kevin Kelly recently posted about digital continuity on The Technium. He references David Pogue’s experience of having problems reading DVDs which were only 4 years old.

No problem, I thought: I’ve got all of the original iMovie projects backed up on DVD, in clear cases, neatly arrayed in a drawer next to my desk. (My hard drive wasn’t big enough to hold those 50 videos a year.)

Guess what? On the Mac I use for video editing, most of the DVD’s were unreadable. They’re less than four years old!

Tried them on another machine. About half of them were readable.

This is an issue which effects all digital artists, and visualists in particular, as digital video tends to take up large amounts of space. Traditionally I’ve used optical media to backup and archive my work, but recently I’ve changed my habits and now I’m keeping everything on hard drives, consolidating smaller drives on to bigger ones as often as architecture changes or Moore’s law makes it affordable. 750GB (around 11c/GB) seems to be the current dollars-per-gig sweet spot, (especially when you keep in mind the increased manufacturer ripoff factor for a “1TB” drive). 750GB will fit around 170 DVDs worth of data, which obviates even the financial argument for burning DVDs – even if it only takes 4 minutes to burn each DVD, that’s over 11 hours of burning time.

Backup Stacks
My current stack of drives ready for Moveage

Update: This post has already generated some fantastic feedback, so we decided to expand it slightly, and promoted it from an Aside to Full Article status.

OrT Says:

I used to work for a company that, among other things, specialized in data recovery just up to the clean room.

Their suggestions:
- Tape is still the most safe long-term storage
- CD’s/DVD’s should be re-written every 2 years
- For live usage: two storage servers with redundancy, spread over two seperate buildings (can be cheaper than it sounds) – one at your location and one a few houses/buildings down or on another remote location

For re-writing a CD/DVD archive, there are burning robots that can automate the process. It will, ofcourse, cost you.

Oh – and for CD’s/DVD’s you should use every once in a while: burn them twice and never touch the second copy to avoid chemical reactions and decay.

This is of course a good point. Tape is a very stable storage medium if stored correctly. I keep all of my DV tapes, so I have every moment of video footage I’ve ever shot, filling various cardboard boxes a secure, climate-controlled bunker. However the data on them is very time consuming to format-shift, so the likelihood that I’ll put them on to a more modern, stable, high-capacity tape format is unlikely, and I’m not really holding out hope for my DV-based cameras to still be in daily use 10 years from now, so retrieving that data can become problematic.

previewlounge says:

i have lost more data on disk drives (three 80 gig drives and a 350 gig drive over about 7 years) than i ever did lose from DVD’s

in fact it was the DVD’s that saved the day! :-)

therefore i suggest backing up to hard drive and also toasting your best animations onto DVD’s as data disks.

This brings us to the crux of the matter, and the most important rule of thumb with data storage: If it doesn’t exist in at least two places, then you don’t really want to keep it.

The point of this post isn’t that hard drives are a superior media for long-term storage, but that long-term storage in general is risky, so if you can keep your data moving, it’s much safer. For most people backing up is a chore, so if the process of moving the data is faster and simpler, it’s much more likely to happen regularly enough to keep everything safe.

The good thing about hard drives is that it’s a reasonably trivial task to duplicate them, and 2 years is about the right amount of time for Moore’s law to let you consolidate your previous generation of drives to the newest, cheap drive. For example, the last generation of drives I need to back up are:

320GB x2
160GB x3
80GB x4
40GB x3

Which happens to add up to about 2x 750GB drives. If this data was on DVDs, it would be at absolute minimum 340DVDs, but probably closer to 400 as there’s generally quite a bit of blank space when I’m backing up to DVD. Even giving just 5 minutes per disk burnt, that’s 33 hours worth of DVD burning just to keep them up to date. Realistically, that’s never going to happen, but copying large amounts of data to new hard drives is a relatively hands-off process, and I’m not expecting it to take me more than an afternoon.

  • http://www.dixitort.com OrT

    I used to work for a company that, among other things, specialized in data recovery just up to the clean room.

    Their suggestions:
    - Tape is still the most safe long-term storage
    - CD's/DVD's should be re-written every 2 years
    - For live usage: two storage servers with redundancy, spread over two seperate buildings (can be cheaper than it sounds) – one at your location and one a few houses/buildings down or on another remote location

    For re-writing a CD/DVD archive, there are burning robots that can automate the process. It will, ofcourse, cost you.

  • http://www.dixitort.com OrT

    Oh – and for CD's/DVD's you should use every once in a while: burn them twice and never touch the second copy to avoid chemical reactions and decay.

  • http://previewlounge.tv previewlounge

    i have lost more data on disk drives (three 80 gig drives and a 350 gig drive over about 7 years) than i ever did lose from DVD's

    in fact it was the DVD's that saved the day! :-)

    therefore i suggest backing up to hard drive and also toasting your best animations onto DVD's as data disks.

    also, it is essential to turn had drives on at least once every three months, if not more often. imho.

    basically, i do not trust hard drives.

    sony is also bringing out an optical disk that looks like an uber 3.5 inch disk, with a special drive, each disk holds 27 gigs, 50 year shelf life .. i will go with that when they bring the price down to say $300? haha, that will take a while.

    anyways, cheers to all for the great article and also good info from OrT (cool name!).

  • http://www.jaymis.com Jaymis

    The most important rule of thumb with data storage is this: If it doesn't exist in at least two places, then you don't really want to keep it.

    The point of this isn't that hard drives are a superior media for long-term storage, but that long-term storage in general is risky, so if you can keep your data moving, it's much safer.

    @previewlounge: Interesting that you've had such bad luck with hard drives. I'm lucky enough to have never lost a drive, and there's about a dozen bare backup drives hanging around here at the moment, as well as all of the ones sitting live in various machines in my studio.

    The good thing about hard drives is that it's a reasonably trivial task to duplicate them, and 2 years is about the right amount of time for Moore's law to let you consolidate your previous generation of drives to the newest, cheap drive. For example, the last generation of drives I need to back up are:

    320GB x2
    160GB x3
    80GB x4
    40GB x3

    Which happens to add up to about 2x 750GB drives. If this data was on DVDs, it would be at absolute minimum 340DVDs, but probably closer to 400 as there's generally quite a bit of blank space when I'm backing up to DVD. Even giving just 5 minutes per disk burnt, that's 33 hours worth of DVD burning just to keep them up to date. Realistically, that's never going to happen, but copying large amounts of data to new hard drives is a relatively hands-off process, and I'm not expecting it to take me more than an afternoon.

  • http://previewlounge.tv previewlounge

    cheers for the extra info .. really helpful.

    "long-term storage in general is risky, so if you can keep your data moving, it’s much safer."

    ah yes, perfect.

    as regards my data losses of the past, it was more to do with inappropriate treatment of drives.. living in dusty artist warehouses, all that sort of thing.

    i guess one situation was unexpected .. i dragged and dropped all the contents of an 80 gig usb2 drive onto a firewire 400 drive (using a mac mini at the time) … also, one external was ntfs and the other was fat32. everything froze. i rebooted. data was lost. ah well. paid to have it retrieved.

    got my own studio now, so that is awesome.

    when i am ready and mature enough, i hope to instigate a Raid System with auto backups.. not sure when that will be tho. ummm … tomorrow! :-D

  • http://vade.info vade

    You can be safe backing up to HDD, but the issue is you have to keep the disks on, or at least, keep them moving occasionally, or the drive mechanism and fluid will seize up. If the spindles dont move, then the lubricant will harden and in 5 years when you power on the drive for the first time, you will be in for a bad surprise.

    Fire up your backups occasionally, and use clean, safe power.

    Also, the larger the drives, the denser the storage, and the more chance their is for a read failure :
    http://blogs.zdnet.com/storage/?p=162

    So, as was mentioned, if you care about some data, have it in more than one place!

  • http://www.jaymis.com Jaymis

    Good point about the fire-ups Vade. I guess once every two years isn't often enough.. We'll see if I have any issues tonight with previously functioning drives which haven't been plugged in for a while.

    As to the ZDNet article – there's a little problem of some factual errors invalidating the claims in that article, but it still doesn't change our overall message: Backing up is cool!

  • Chris

    In addition to hard backups, I also do online backups. mozy.com, $4.99 a month for unlimited backup with 30 days of incremental file restores. If you have a fast enough outgoing connection, it's worth it.

  • prevolt

    I'm thinking of putting together a RAID 1 setup first and saving to get tape backup after.
    Maybe a 4 to 8-bay enclosure that supports FW800? I haven't found the right one yet, though.

  • sleepytom

    RAID 5 NAS servers are pretty affordable these days, something like the buffalo terrastation is a good bet.

    If your building raid systems make sure you get hard disks with non-sequential batch numbers. disk failure often happens because of a manufacturing fault with a batch of disks – if all your raid disks are from the same batch then they might all fail at once which would be a total data loss.

    i'm moving to triple redundancy with two separate raid 5 servers which do automated FTP backups / mirroring of each other, via the internet. Hopefully that will safeguard me against fire and theft as well as drive failure.

  • Kevin Hackett

    Sony makes a 24k gold layer disk they claim has a storage life of 300 years. They only come in CD size (800mb) and cost about 2-3.00 each. Anyone used these?

  • http://seguesound.com Dri

    The problem with that product is the word "Sony". I don't think anyone believes their predictions for storage/shelf life anymore.

    I was listening to a BBC Science podcast yesterday that explored meta-materials, which is the next logical (albeit "potential") step for optical media. Basically the combination of manufacturing materials on a ridiculously "thin" surface to control light in some of the ways that animal retinas do (as say, deep sea fish cant "make" mirrors, their eyes have developed clever ways to process light). This control of light, saddled with a greater understanding of ways to approach photon manipulation, is deemed as a progressive area of product development for storage media.

    Until then, Sony can make more ridiculous claims (i have a shelf of their "40 years" CD-R's that have never been touched, and are degrading after 5 years) whilst the logical mind would say "erm, flashdrives are pretty cheap too".

    This hurts A/V of course, but moreso the Audio side. These days we have access to pure audio on magnetic tape from all the classic albums. The future will have a few hard drives, session files from incompatible DAW's and plugin's that dont work anymore. The legacy of our art in the future.

  • xpez2000

    I learned this lesson about two years ago and lost tons of animations because I used 99 cent dvds in the year 1999.

    You need ceramic plated dvds for long term storage.

    I now migrate all of most important data to newer, larger drives.

    I have accepted the strange reality that a closet in my place will become packed with hard drives in 10 years…

    BUT I have a seagate 3.5 – 1 gig from like 1997. It still fires up and loads up my drum samples for my MPC2000…HA!!!!