Ctrl+K

Other useful file operations

Other useful file operations#

Transferring files between computers#

There are many different protocols for transferring files between computers. You may have heard of FTP - File Transfer Protocol - which is a non-secure but commonly used example. A more secure file transfer protocol is SCP - Secure Copy Protocol, and programs such as WinSCP use it. The command scp is an easy way to transfer a file immediately between the server you are working on and another (or two different servers!). Another command to copy files is rsync, which can be used with many options such as preserving the ownership and date of creation of a file (and much more).

Be aware that all the scp commands shown below have to be executed on your local computer. Thus if you are currently connected to a server, you first have to exit this connection to be connected to your local computer again (Windows Command or Mac Terminal).

../../../_images/scp_ms.gif

Tip: Remember that with the ‘tab ‘ key you can auto complete and see the available options by double pressing. This can make finding a file you want to upload way easier. (Note: This works only for the machine you are currently on.)

# Secure CoPy
man scp
scp /path/to/source user@server:/path/to/destination # local server (where the command is executed) to remote server
scp <user>@server:/path/to/source /path/to/destination # remote server to local
scp <user>@server1:/path/to/source <user>@server2:/path/to/destination # remote server 1 to remote server 2

# Download an E. coli genome from the server to your local computer
# First open Windows Command or Mac Terminal on your local computer
scp <user>@cousteau.ethz.ch:/nfs/teaching/551-0132-00L/1_Unix1/genomes/bacteria/escherichia/GCF_000005845.2_ASM584v2/GCF_000005845.2_ASM584v2_genomic.fna .
# Note that the local path is just "." - this means "here", i.e.: your working directory

# Copy the E.coli genome (or any file) from your local computer to the home folder on the server
# Again, on your local system, run the following commands in Windows Command or Mac Terminal
scp GCF_000005845.2_ASM584v2_genomic.fna <user>@cousteau.ethz.ch:~/

# Copy the E.coli genome from cousteau to euler, executed on cousteau
scp GCF_000005845.2_ASM584v2_genomic.fna <user>@euler.eth.ch:~/

# Copy the E.coli genome from cousteau to euler, executed on your own computer
scp <user>@cousteau.ethz.ch:GCF_000005845.2_ASM584v2_genomic.fna <user>@euler.ethz.ch:~/

Warning

Windows Paths

Windows terminal is not unix-like - instead it is based on MS-DOS and uses some different commands and conventions. Paths on Windows terminal use backslashes instead of forward slashes, and the root is the letter assigned to the drive where the path is. For instance a file in my downloads folder might have the path:

C:\Users\fieldc\Downloads\file.txt

Sometimes you want to download a file directly from the internet to the server, rather than going via your local machine. For using this command, you have to connect to the server first and then wget allows you to download files in this way.

# Download from the internet
wget source-URL
wget https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/482/265/GCF_000482265.1_EC_K12_MG1655_Broad_SNP/GCF_000482265.1_EC_K12_MG1655_Broad_SNP_genomic.fna.gz

Compressing and decompressing files#

Files can be compressed to take up less space on the hard drive (disk), or for transfer over the internet. The file you downloaded is an example, and we can decompress it using the gunzip command:

# Decompress a file
gunzip GCF_000482265.1_EC_K12_MG1655_Broad_SNP_genomic.fna.gz

If you ever need to compress a file, for instance to send it to someone, you can use the gzip command:

# Compress a file
gzip GCF_000482265.1_EC_K12_MG1655_Broad_SNP_genomic.fna

Exercise 0.4

  • Download the mystery image /nfs/teaching/551-0132-00L/1_Unix/mystery_image.jpg from the server to your local machine. What is it?

scp <your eth name>@cousteau.ethz.ch:/nfs/teaching/551-0132-00L/1_Unix/mystery_image.jpg .

A cat

../../../_images/mystery_file.jpg

  • On the server, download the E. coli file from the example above to your home folder.

# Make sure I am in my home directory
cd ~

# Download the file
wget https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/482/265/GCF_000482265.1_EC_K12_MG1655_Broad_SNP/GCF_000482265.1_EC_K12_MG1655_Broad_SNP_genomic.fna.gz

  • Decompress the file.

#Decompress it
gunzip GCF_000482265.1_EC_K12_MG1655_Broad_SNP_genomic.fna.gz
Contents