How to transfer files¶

Selecting the Right Method¶

scp: transfer single and unique files
rsync: transfer hierarchical data, such a directory and its member files, It is able to detect what has changed between the source and destination data, and saves time by only transmitting these changes
bbcp: transfer large data file by providing higher speed via multiple steams.

What is NOT Available:

ftp (file transfer protocol, and not to be confused with sftp)
rcp (remote copy)

SCP¶

Using scp is the easiest method to use when transferring single files.

Local File to Remote Host

% scp localfile user@remotehost:/destination/dir/or/filename

Remote Host to Local File

% scp user@remotehost:/remote/filename localfile

SFTP¶

Interactive Mode¶

One may find this mode very similar to the interactive interface offered. A login session may look similar to the following:

% sftp user@remotehost
_(enter in password)_
 ...
sftp>

The commands are similar to those offered by the outmoded ftp client programs: get, put, cd, pwd, lcd, etc. For more information on the available set of commands, one should consult sftp the man page.

% man sftp

Batch Mode¶

One may use sftp interactively in two cases.

Case 1: Pull a remote file to the local host.

% sftp user@remotehost:/remote/filename localfilename

Case 2: Creating a special sftp batch file containing the set of commands one wishes to execute with out any interaction.

% sftp -b batchfile user@remotehost

rsync (preferred)¶

rsync is an extremely powerful program; it can synchronize entire directory trees, only sending data about files that have changed. That said, it is rather picky about the way it is used. The rsync man page has a great deal of useful information, but the basics are explained below. Single File Synchronization

To synchronize a single file via rsync, use the following:

To send a file:

% rsync --archive --stats --progress localfile \
    username@remotehost:/destination/dir/or/filename

To receive a file:

% rsync --archive --stats --progress \
    username@remotehost:/remote/filename localfilename

Directory Synchronization¶

To synchronize an entire directory, use the following:

To send a directory:

% rsync --archive --stats --progress localdir/ \
    username@remotehost:/destination/dir/

or

% rsync --archive --stats --progress localdir \
    username@remotehost:/destination

To receive a directory:

% rsync --archive --stats --progress \
    username@remotehost:/remote/directory/ /some/localdirectory/

or

% rsync --archive --stats --progress \
    username@remotehost:/remote/directory /some/

Note the difference with the slashes. The second command will place the files in the directory /destination/localdir; the fourth will place them in the directory /some/directory. rsync is very particular about the placement of slashes. Before running any significant rsync command, add --dry-run to the parameters. This will let rsync show you what it plans on doing without actually transferring the files.

Synchronization with Deletion¶

This is very dangerous; a single mistyped character may blow away all of your data. Do not synchronize with deletion if you aren't absolutely certain you know what you're doing.

To have directory synchronization delete files on the destination system that don't exist on the source system:

% rsync --archive --stats --dry-run --progress \
    --delete localdir/ username@remotehost:/destination/dir/

Note that the above command will not actually delete (or transfer) anything; the --dry-run must be removed from the list of parameters to actually have it work.

BBCP¶

Using bbcp to transfer large data files without encryption.

% bbcp \[opt] user@source:/path/to/data user@destination:/path/to/store/data

Possible options include:

-P 2

Give a progress report every 2 seconds

- w 2M

TCP window size of 2MB

-s 16

Set the number of streams to 16 (default is 4)

Other options may be necessary if bbcp is not installed in a regular location on either end of the transfer. This can lead to rather complex command lines:

$ bbcp -z -T \
  "ssh -x -a -oFallBackToRsh=no %I -l %U %H /home/user/Custom/bin/bbcp" \
  foobar-5.4.14.tbz "ruser@10.20.30.40:foo.tbz"

Client Software¶

scp and sftp¶

The command-line scp and sftp tools come with any modern distribution of OpenSSH; this is generally installed by default on modern Linux, UNIX, and Mac OS X installs.

Windows Clients¶

Windows clients include:

(puTTY-related command line utilities), and

scp, sftp, & rsync as provided by Cygwin.

*** VERY IMPORTANT *: if you use Filezilla, please use the Site Manager feature (under "File") to manage the profile of the cluster you use. In the "Transfer Settings" tab, make sure that the "Limit number of simultaneous comments" box is checked and the "Maximum number of connections" is set to 1. Failing to do so may result in Filezilla creating excessive ssh connections, which could lead the suspension of your user account.