[Tig] Archiving (was Re: Holographic Drive at NAB)

[Bob Friesenhahn]

As some of you are aware, I am a fan of Sun's ZFS filesystem 
(http://en.wikipedia.org/wiki/ZFS) which includes considerable data 
safeguards including various sorts of RAID, extra block redundancy, 
and checksums of every block.  The holographic drive discussion caused 
me to think about how ZFS could be used to assist with long-term 
archiving.  I had an idea for how it could be done so I brought the 
topic up on a ZFS discussion list.  One of the list members posted 
this little recipe which jived with my own way of thinking:

> It would involve the following steps
> 1. create blank ISO images of the size of your media
> 2. zpool create wormyz raidz2 image1.iso image2.iso image3.iso ...
> 3. Move your data to the pool
> 4. export the pool
> 5. burn the media

So basically you create a temporary ZFS pool based on a set of 
appropriately-sized disk-emulation files, copy your data into the 
pool, and then copy the disk-emulation files onto the actual media.

With this approach you obtain the benifit of quite rigorous 
checksumming combined with automatic repair algorithms which are 
independent of the underlying storage media (which likely includes its 
own block-level error correction).

If the Holographic drives are 300GB, then you can build RAID arrays 
based on an arbitrary number of 300GB drives (or SAN LUNs), put ZFS on 
top (takes a couple of seconds), copy the data to be archived to the 
array, and then just copy the content of each raw disk to a 
holographic disk for storage.  This approach obtains full RAID 
performance.  Since the RAID array is fully usable, it can also act as 
the working storage for the data.

There is an archive of the discussion topic at 
"http://www.opensolaris.org/jive/thread.jspa?threadID=58062&tstart=0".

Bob
======================================
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/