Archiving Large Files

» Articles / Code » Server Configuration » Archiving Large Files

Creating the Archive

Suppose I want to back up the directory "mydir". The following command
creates a compressed archive of the mydir directory and its contents,
split into 1000MB chunks:

tar -cz mydir | split -d -b 1000m - myarchive.tgz

(Note the use of a "pipe" (|) to send the output of the tar command
directly to the split command, thereby avoiding the need to create an
intermediate file and halving the disc space and time required for this
operation.)

The result is a series of files named "myarchive.tgz.00",
"myarchive.tgz.01", etc., each no larger than 1000MB. In your case,
replace "mydir" with the path to the directory or files that you wish to
back up and replace "myarchive" with your own name for the archive to be
created.

I now calculate an MD5 checksum for each chunk and write it to a file
called myarchive.md5:
md5sum myarchive.tgz.* > myarchive.md5

That's it! In two lines we have created a set of archive and checksum
files ready to be copied to the backup storage of your choice. You can
use your favourite DVD writing program (GnomeBaker, K3b, etc.) to write
the files to DVD or, alternatively, plug in a USB drive and simply copy
the files to it.
Restoring the Archive
To restore the files from the archive, I create a temporary directory,
say "restored", copy the archive and checksum files into it and proceed
as follows.

Change to the directory containing the archive files and calculate new
checksums:

cd restored
md5sum myarchive.tgz.* > restored.md5

Compare the original checksums with the ones just calculated:

diff myarchive.md5 restored.md5

This will list any differences between the two sets of checksums (no
news is good news!).

Re-combine the chunks into a single archive and extract the original
files:

cat oldpc.tgz.* | tar -xz

(Again we use a pipe to avoid the need for an intermediate file, saving
time and disc space.)

The directory "restored" now has a sub-directory called "mydir"
containing a perfect copy of the original files.