Use rsync to copy files faster, as well as showing progress

One of the commands that I find my self using almost everyday is rsync:

a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host
over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control
every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is
famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending
only the differences between the source files and the existing files in the destination. Rsync is widely
used for backups and mirroring and as an improved copy command for everyday use.

Rsync finds files that need to be transferred using a “quick check” algorithm (by default) that looks for
files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as
requested by options) are made on the destination file directly when the quick check indicates that the
file’s data does not need to be updated.

In general, the following flags work great: -P (Progress), -a (archive mode), -v (be a little verbose).   When copying to an NFS share, it’s also a good idea to bandwidth limit your connection to some reasonable number. I’m copying via wifi and I know from extensive testing that I get around 15 to 25 megabytes a second copies to my NFS server. So to stay on the safe side I limit to just under 15 megabytes (150000 bytes). This can be done with the –bwlimit=BYTES flag.

 

cfaber@sputnik:/home/dump$ rsync -Pav --bwlimit=15000 data /home/hal/OpenNAS/.
sending incremental file list
data
    608,370,688  56%   14.65MB/s    0:00:31 

And so on.

The nice thing about this is that rsync will respect that bandwidth limit. Additionally the file being synced is checksummed for errors, so you can be sure that the destination file is identical to the source file.

Rsync is also available on windows within Cygwin however, it doesn’t respect, and can’t copy file permissions (from within a windows environment).

Leave a Reply

Your email address will not be published. Required fields are marked *