Transferring files between computers#
There are many different protocols for transferring files between computers. You may have heard of FTP - File Transfer Protocol - which is a non-secure but commonly used example. A more secure file transfer protocol is SCP - Secure Copy Protocol, and programs such as WinSCP use it. The command scp (Secure Copy Protocol) is an easy way to transfer a file immediately between the server you are working on and another (or two different servers!). Another command to copy files is rsync, which can be used with many options such as preserving the ownership and date of creation of a file (and much more).
Be aware that all the scp commands shown above have to be executed on your local computer. Thus if you are currently connected to a server, you first have to exit this connection to be connected to your local computer again (Windows Command or Mac Terminal).
Remember that with the tab key you can auto complete and see the available options by double pressing. This can make finding a file you want to upload way easier. Note that this only works for the machine you are currently on.
# Secure CoPy
scp /path/to/source user@server:/path/to/destination # local server (where the command is executed) to remote server
scp <user>@server:/path/to/source /path/to/destination # remote server to local
scp <user>@server1:/path/to/source <user>@server2:/path/to/destination # remote server 1 to remote server 2
# Download an E. coli genome from the server to your local computer
# First open Windows Command or Mac Terminal on your local computer
scp <user>@cousteau.ethz.ch:/nfs/teaching/551-0132-00L/1_Unix/genomes/bacteria/escherichia/GCF_000005845.2_ASM584v2/GCF_000005845.2_ASM584v2_genomic.fna .
# Note that the local path is just "." - this means "here", i.e.: your working directory
# Copy the E.coli genome (or any file) from your local computer to the home folder on the server
# Again, on your local system, run the following commands in Windows Command or Mac Terminal
scp GCF_000005845.2_ASM584v2_genomic.fna <user>@cousteau.ethz.ch:~/
# Copy the E.coli genome from cousteau to euler, executed on cousteau
scp GCF_000005845.2_ASM584v2_genomic.fna <user>@euler.eth.ch:~/
# Copy the E.coli genome from cousteau to euler, executed on your own computer
scp <user>@cousteau.ethz.ch:GCF_000005845.2_ASM584v2_genomic.fna <user>@euler.ethz.ch:~/
Warning
Windows Paths
Windows terminal is not unix-like - instead it is based on MS-DOS and uses some different commands and conventions. Paths on Windows terminal use backslashes instead of forward slashes, and the root is the letter assigned to the drive where the path is. For instance a file in my downloads folder might have the path:
C:\Users\fieldc\Downloads\file.txt
Sometimes you want to download a file directly from the internet to the server, rather than going via your local machine. For using this command, you have to connect to the server first and then wget allows you to download files in this way.
# Download from the internet
wget source-URL
wget https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/482/265/GCF_000482265.1_EC_K12_MG1655_Broad_SNP/GCF_000482265.1_EC_K12_MG1655_Broad_SNP_genomic.fna.gz
Up/downloading files to/from Cousteau in R-Studio#
In the section above you learned how to transfer files between the server Cousteau and your local computer through the command line interface.
As the R-Studio environment which we setup for this course also runs on Cousteau, alternatively to transferring files through the command line interface with the command scp, you can actually up/download files from within R-Studio. When you login to a new R-Studio session, you can look at the bottom right panel showing your home directory on cousteau. You can then click on the button ‘Upload’ to upload a file from your local computer or download a file to your local computer by ticking the ‘Checkbox’ beside a file and then clicking ‘More’ -> ‘Export’.
Note that this alternative of transferring files only works for file transfer between Cousteau and your local computer because we setup the R-Studio session on Cousteau. Therefore, if you want to transfer files between another server and your local computer, you would have to use the command line interface with scp, f.ex. when transferring files between two servers.
Exercise 1.9#
Exercise 1.9
On your local machine (Windows Command/Mac Terminal), download the mystery image
/nfs/teaching/551-0132-00L/1_Unix/mystery_image.jpgfrom the server to your local machine. What is it?
# First either disconnect from the server that you are logged in by typing exit or if you are connected through Mobaxterm, open the Windows Command/Mac Terminal instead
scp <your eth username>@cousteau.ethz.ch:/nfs/teaching/551-0132-00L/1_Unix/mystery_image.jpg .
# On the Windows Command or Mac Terminal, check your current location on your local computer that the file was downloaded to, copy/paste this path into the file explorer, look for the downloaded .jpg file in that directory
A cat
On the server, download the E. coli file from the example above to your home folder on the server.
# Reconnect to the server
ssh <your eth username>@cousteau.ethz.ch
# Make sure I am in my home directory
cd ~
# Download the file
wget https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/482/265/GCF_000482265.1_EC_K12_MG1655_Broad_SNP/GCF_000482265.1_EC_K12_MG1655_Broad_SNP_genomic.fna.gz
# Check that the file was correctly downloaded to your home directory
ls
From within R-Studio, download the E. coli file from your home folder on the server to your local computer.
# Open a new R-Studio session
# Check the home folder in the bottom right corner for the E. coli file
# Check the tickbox beside the file and click 'More' + 'Export' to download this file to your local computer
# After checking that the download was successful, delete the file from your local computer to save disk space