Contents


Deprecated: please see new documentation site.



This page discusses methods by which you may get files on to and off of the various LONI resources.

What is NOT Available

  • FTP (file transfer protocol, and not to be confused with SFTP)
  • rcp (remote copy)

Selecting the Right Method

Knowing what one want to transfer will help them determine the best tool for the job.

For example, the transferring of single, unique files should be done using scp or sftp. When transferring hierarchical data, such a directory and its member files, one should use rsync. This is especially true when an older copy of this hierarchical data already exists on the destination side. rsync is able to detect what has changed between the source and destination data, and saves time by only transmitting these changes.

Be Careful!

Any tool may clobber (destroy) a source file if it is told to copy a file to the same location. For example, if file A on host 1 is transferred to file A on host 1, file A will be over written. In most circumstances, this results in a source file of 0 bytes - effectively deleting it.

For example, the following will clobber myfile.txt on host1.

user@host1> scp ~/myfile.txt host1:~

Before transferring any file, one needs to note on what host they are so that a file is not accidentally copied to itself.

Also, see the Disk and IO FAQ for information on backing up ones important files.

Useful Tools and Methods

This section describes what tools and methods are available for transferring files, and it outlines the trade offs associated with each method. It will be noted when a particular method is not available for use on a particular LONI resource.

rsync Over SSH (preferred)

rsync is an extremely powerful program; it can synchronize entire directory trees, only sending data about files that have changed. That said, it is rather picky about the way it is used. The rsync man page has a great deal of useful information, but the basics are explained below.

Single File Synchronization

To synchronize a single file via rsync, use the following:

To send a file:

% rsync --rsh=ssh --archive --stats --progress localfile \
        username@remotehost:/destination/dir/or/filename

To receive a file:

% rsync --rsh=ssh --archive --stats --progress \
        username@remotehost:/remote/filename localfilename

Note that --rsh=ssh is not necessary with newer versions of rsync, but older installs will default to using rsh (which is not generally enabled on modern OSes).

Directory Synchronization

To synchronize an entire directory, use the following:

To send a directory:

% rsync --rsh=ssh --archive --stats --progress localdir/ username@remotehost:/destination/dir/

or

% rsync --rsh=ssh --archive --stats --progress localdir username@remotehost:/destination

To receive a directory:

% rsync --rsh=ssh --archive --stats --progress \
        username@remotehost:/remote/directory/ /some/localdirectory/

or

% rsync --rsh=ssh --archive --stats --progress \
        username@remotehost:/remote/directory /some/

Note the difference with the slashes. The second command will place the files in the directory /destination/localdir; the fourth will place them in the directory /some/directory. rsync is very particular about the placement of slashes. Before running any significant rsync command, add --dry-run to the parameters. This will let rsync show you what it plans on doing without actually transferring the files.

Synchronization with Deletion

This is very dangerous; a single mistyped character may blow away all of your data. Do not synchronize with deletion if you aren't absolutely certain you know what you're doing.

To have directory synchronization delete files on the destination system that don't exist on the source system:

% rsync --rsh=ssh --archive --stats --dry-run --progress \
        --delete localdir/ username@remotehost:/destination/dir/

Note that the above command will not actually delete (or transfer) anything; the --dry-run must be removed from the list of parameters to actually have it work.

SCP

Using scp is the easiest method to use when transferring single files.

Local File to Remote Host

% scp localfile user@remotehost:/destination/dir/or/filename

Remote Host to Local File

% scp user@remotehost:/remote/filename localfile

No-Payload-Encryption SCP

This is a modified scp utility that saves time by encrypting only the authentication exchange with the remote host. The actual payload (files) being transferred are not encrypted, as is the case with the traditional scp utility. This will only work when both the sending and receiving system have the HPN-enabled SSH installed.

At this time, this tool is only available on the IBM 575 clusters, and can be used by replacing scp with npescp. Do not use this if the security of the data is critical; it will go over the wire unencrypted.

% npescp localfile user@remotehost:/destination/dir/or/filename

SFTP

Interactive Mode

One may find this mode very similar to the interactive interface offered. A login session may look similar to the following:

% sftp user@remotehost
(enter in password)
 ...
sftp>

The commands are similar to those offered by the outmoded ftp client programs: get, put, cd, pwd, lcd, etc. For more information on the available set of commands, one should consult sftp the man page.

% man sftp

Batch Mode

One may use sftp interactively in two cases.

Case 1: Pull a remote file to the local host.

% sftp user@remotehost:/remote/filename localfilename

Case 2: Creating a special sftp batch file containing the set of commands one wishes to execute with out any interaction.

% sftp -b batchfile user@remotehost

Additional information on constructing a batch file is available in the sftp man page.


Client Software

scp and sftp

Standard Clients

The command-line scp and sftp tools come with any modern distribution of OpenSSH; this is generally installed by default on modern Linux, UNIX, and Mac OS X installs.

Windows Clients

Windows clients include:

(puTTY-related command line utilities), and

  • scp, sftp, & rsync as provided by Cygwin.

For additional clients, please see http://www.openssh.com/windows.html.

Mac OS Clients

See http://www.openssh.com/macos.html.

rsync

The command-line rsync application is the only widely-available client for the protocol. It comes with most modern Linux, Unix, and Mac OS X distributions, and other versions (plus source) can be downloaded from the official website.

Advanced Methods for Raw Speed

While the methods above will suffice for all but the most time-sensitive file transfers, and are considerably simpler than the alternatives, users who have further needs can contact the systems administrators to work out potential alternatives.


Users may direct questions to sys-help@loni.org.

Powered by MediaWiki