Monday, June 20, 2011

Copying files between servers with compression

You may have to copy very big files or file systems between servers sometimes, and the more you optimize this copy the faster you will transfer the files with less computer resources usage.

You might think that scp alone would do a good job, but without compression you may be wasting bandwith and time (talking about copying file systems of hundreds of gigabytes between servers far away one of the other). But if you have a lot of disc space available, why not to compress the file system first and then transfer it? Well, because compressing big file systems takes a lot of time and your disc I/O is severely degraded (the server reads a file and tries to write it compressed at the same time); you might be render your server practically unusable while compressing big files, at least in AIX.

But you can make a backup, compress it and send it to the other server, uncompressing it and writing just the files you want to transfer (not writing compressed files on disk at any time), compressing it on the fly with pipes. To this end, you may write this little script and put it in an appropriate path in the source server:

tar -cf - $1 | compress -c

Then, if you named this script mycompressedtar for example, you can transfer files or directories to your target server this way:

otheruser@targetsrv> ssh -l myuser sourcesrv /path/mycompressedtar /some/dir | uncompress -c | tar -xf -

If you issue this command at your target server, you will copy /some/dir from source server to the path where you stand at execution time; of course you have to have access to /some/dir with the myuser account.

But some times you have few but very big files (like Oracle datafiles), and you cannot transfer them with tar without getting an error in AIX; in that case you can just transfer file by file with this little script:

cat $1 | compress -c

Then, if you named this script mycompress for example, you can copy one file to your target server this way:

otheruser@targetsrv> ssh -l myuser sourcesrv /path/mycompress /some/file | uncompress -c > somefile

If you issue this command at your target server, you will copy file from source server to the path where you stand at execution time and name it somefile; you can use the same name as the source file or a different one.

But we're talking about different servers, and the files are so valuable we cannot afford to change a single byte. How can we be sure the files are exactly the same? In AIX, you can use the cksum command:

myuser@sourcesrv> cksum myfile.txt
1656934735 2229 myfile.txt

===================================

otheruser@targetsrv> cksum myfile.txt
1656934735 2229 myfile.txt

This way you'll be sure both files are equal.

More information:

cksum Command

No comments:

Post a Comment