Secure Remote Access

This section is the about setting up secure, easy to use, terminal (command-line) access from your laptop to the EOAS Ocean Linux computers, and the Compute Canada graham High Performance Computing (HPC) cluster.

The software technology we use to do that is called ssh (sometime lowercase, sometimes uppercase). ssh is also the command that makes terminal/command-line connections between computers. ssh stands for “secure shell”. The first s is for secure, and the sh is in tribute to the original Unix command-line interface called “shell”. That’s why people often say or write “open a shell” or “go to shell” when they mean open or switch to your Terminal program.

This brief Introduction to SSH explains some of the concepts and terminology about the software that we use to accomplish easy, secure access.

Specifically, we use key-based authentication. The digitalocean.com ssh tutorial has a good explanation of how key-based authentication works in comparison to password authentication:

An SSH server can authenticate clients using a variety of different methods. The most basic of these is password authentication, which is easy to use, but not the most secure.

Although passwords are sent to the server in a secure manner, they are generally not complex or long enough to be resistant to repeated, persistent attackers. Modern processing power combined with automated scripts make brute forcing a password-protected account very possible. Although there are other methods of adding additional security (fail2ban, etc.), SSH keys prove to be a reliable and secure alternative.

SSH key pairs are two cryptographically secure keys that can be used to authenticate a client to an SSH server. Each key pair consists of a public key and a private key.

The private key is retained by the client and should be kept absolutely secret. Any compromise of the private key will allow the attacker to log into servers that are configured with the associated public key without additional authentication. As an additional precaution, the key can be encrypted on disk with a passphrase.

The associated public key can be shared freely without any negative consequences. The public key can be used to encrypt messages that only the private key can decrypt. This property is employed as a way of authenticating using the key pair.

The public key is uploaded to a remote server that you want to be able to log into with SSH. The key is added to a special file within the user account you will be logging into called ~/.ssh/authorized_keys.

When a client attempts to authenticate using SSH keys, the server can test the client on whether they are in possession of the private key. If the client can prove that it owns the private key, a shell session is spawned or the requested command is executed.

The security of key-based authentication comes from:

  • the cryptographic strength of key pair

  • the length and complexity of the passphrase that you choose to use to encrypt your private key when it is stored on disk

  • the fact that you remember that passphrase and don’t write it down or share it with anyone

The thing that makes key-based authentication easy to use is a program called an “ssh agent” that runs on your laptop and manages using your keys to authenticate your access to remote systems. Once you have your ssh keys set up, the only time you should need to type in your passphrase is when you connect to a remote computer for the first time after your laptop has been started up after a shutdown or reboot. When you connect to a remote computer that first time, your ssh agent asks you for your passphrase so that it can decrypt your private key that is stored on disk. After that, it can use that key to do any authentications you need done, until you shutdown or reboot your computer, or explicitly tell the ssh agent to forget about a key.

It’s not just authentications that are encrypted by ssh keys. Once you sigh-in to a remote computer using ssh key-based authentication, all of the information sent both ways over the Internet between your laptop and that computer is also encrypted.

The instructions below assume that your are working on a Mac or a Linux laptop or desktop. They may also work on a Windows computer that is running Windows Subsystem for Linux (WSL), but that has not been tested. If you are using Windows 10, you will need to install the Open SSH Client app via Settings > Apps > Apps > Features > Optional Features. Please see these instructions, but only install the “Client” part – don’t try to do any of the “server” parts.

Note

Help Wanted

If you know about setting up ssh on Windows, please feel free to add to these docs, or contact Doug to help him do so.

Generate ssh Keys

The first step is to generate an ssh key pair on you local computer. That is done using the ssh-keygen command. Here we tell it to use the Ed25519 algorithm to create the keys.

Open your Terminal program to get a command-line interface, and type the command:

$ ssh-keygen -t ed25519

The output should look like:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_ed25519):

but with with your laptop user id instead of username in the key file path in the parentheses.

Hit enter to accept the default key file path and name.

Warning

If you get a message like:

/home/username/.ssh/id_ed25519 already exists.
Overwrite (y/n)?

it means that you already have a key pair with the default name. Please use Ctrl-c to exit from ssh-keygen and contact Doug for advice on how to proceed.

You might see the message:

Created directory '/home/username/.ssh'.

next, then, for sure, you should see the message:

Enter passphrase (empty for no passphrase):

This is where you enter the passphrase that encrypts your private key that will be stored on disk. Please use a long passphrase that you can easily remember; something like:

  • a line or two from a favourite song, poem, or story

  • a nonsense rhyme

  • the names of some or all of your cousins

  • a list of place names that are significant to you but that would be hard for someone else to guess

  • a statement that someone in your family often makes

  • a famous or inspiring quotation

  • the names of the 12 goldfish you had growing up

You can use all the spaces and special characters that you want. Remember that a longer passphrase is more secure, and that you won’t have to type it very often…

But you will have to type it again after the next prompt :-)

Enter same passphrase again:

When the key pair generation is finished you should see output like:

Your identification has been saved in /home/username/.ssh/id_ed25519
Your public key has been saved in /home/username/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:8lYuN0DcZra83nBgsElzsP6EYqZYEt7zzslgKhuuxT8 username@host
The key's randomart image is:
+---[RSA 4096]----+
|        .        |
|       . +       |
|  .     B *      |
| . o   + % .     |
|  o + = S B      |
| . + * + B o     |
|. + + . + B .    |
|oo +E= o + =     |
|++. ..=   . .    |
+----[SHA256]-----+

except that you will see:

  • your laptop user id instead of username in the key files paths

  • a different key fingerprint, ending with your user id and computer name instead of username@host

  • a different “randomart image”

Congratulations! You are now the owner of a shiny new ssh key pair!

Set Up ssh Configuration

The next step to making ssh keys easy to use is to tell the ssh command (and other related commands) how we want them to behave. We do that by putting directives in the file /home/username/.ssh/config. You can use any text editor you want to do this: vim, emacs, VSCode, SublimeText, Notepad, or nano.

Please don’t use a word processing program like Microsoft Word or LibreOffice Write.

We’ll use nano here because it is available almost everywhere.

Open the file with nano:

$ nano ~/.ssh/config

This should create a new, empty file. If there is already some text in the file, please contact Doug for advise on how to proceed.

By the way, ~ is a shorthand way of typing /home/username with your user id in place of username.

Directives for All Hosts

Put the following lines in your ~/.ssh/config file:

Host *
  ForwardAgent yes
  ServerAliveInterval 60

The first line, Host *, means that the directives that come next, indented beneath it, apply to all remote computers that you connect to.

The indentation can be spaces or tabs; 2 or 4 spaces are conventional.

The 2nd line, ForwardAgent yes, means that when you connect to a remote computer, your ssh agent should set things up so that, if you connect to another computer from that one, the authentication handling is passed back to your laptop. That means that you don’t have to store copies of your private key on a bunch of different computers, it can stay safe on your laptop.

The 3rd line, ServerAliveInterval 60, tells ssh to send a keep-alive message every 60 seconds to any computer that you are connected to. That helps prevent you connection from getting broken if you stop typing for a few minutes to think, answer the phone, check Slack, or go make tea. Some Internet service providers (notably Shaw) are really aggressive about shutting down idle network connections. This directive helps a little to defend against that annoyance.

If you are working on a Mac with the Sierra or later version of the operating system, or if you are on Windows 11, you should add another line to the stanza that you have already typed so that it looks like:

Host *
  ForwardAgent yes
  ServerAliveInterval 60
  AddKeysToAgent yes

That 4th line, AddKeysToAgent yes, tells the ssh agent to remember the keys that you give the passphrases for. Apple decided to make Sierra and later versions of their OS super annoying (though, admittedly, more secure - to the level of paranoia :-) by making the ssh agent forgetful by default.

Host Aliases

To make it easier to connect to remote systems that you use often you can add stanzas to your ~/.ssh/config that:

  • give the remote computer a shorter name

  • tell ssh and friends what user id you use on that computer

  • provide directives to override or augment those in the Host * stanza, or other defaults

Add a host alias stanza for graham.computecanada.ca by adding these lines:

Host graham
  HostName graham.computecanada.ca
  User username

Note

Please be sure to replace username with your Compute Canada user id.

It is conventional to separate the stanzas in ~/.ssh/config with empty lines. You can also add comment lines if you want by starting them with the # character. Your file should now look like:

Host *
  ForwardAgent yes
  ServerAliveInterval 60

Host graham
  HostName graham.computecanada.ca
  User username

Now, instead of having to type:

ssh username@graham.computecanada.ca

you will be able to type:

ssh graham

(after we complete 1 more step of setup).

Add another stanza for the MOAD workstation that Susan told you to work on (using chum here as an example), and one for our compute server, salish:

Host chum
  HostName chum.eos.ubc.ca
  User username

Host salish
  HostName salish.eos.ubc.ca
  User username

Note

Please be sure to replace username with your EOAS user id.

Save your file with Ctrl-o (then hit enter), and exit nano with Ctrl-x.

Copy Your Public ssh Key to Remote Computers

The final step to make ssh key pair authentication work is to copy your public key to each remote system that you want to connect to. The command to do that is ssh-copy-id.

Copy your public key to graham with:

$ ssh-copy-id graham

That command will use the information you put into ~/.ssh/config to expand graham to username@graham.computecanada.ca. It should produce output like:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/username/.ssh/id_ed25519.pub"
The authenticity of host 'graham.computecanada.ca (199.241.166.2)' can't be established.
ED25519 key fingerprint is SHA256:mf1jJ3ndpXhpo0k38xVxjH8Kjtq3o1+ZtTVbeM0xeCk.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Type yes to proceed. (You can confirm graham’s host key fingerprint at https://docs.alliancecan.ca/wiki/SSH_security_improvements#SSH_host_key_fingerprints if you want to).

The output from ssh-copy-id should continue with:

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
username@graham.computecanada.ca's password:

Type in your Compute Canada password, and the output from ssh-copy-id should finish with:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh graham"
and check to make sure that only the key(s) you wanted were added.

Now, as the output says, test the authentication with:

$ ssh graham

Your ssh agent should ask you for your passphrase so that it can decrypt your private key, then you should find yourself at the command-line prompt on graham:

*********************************************************************

Welcome to the ComputeCanada/SHARCNET cluster Graham.

 Documentation: https://docs.computecanada.ca/wiki/Graham
Current issues: https://status.computecanada.ca/
       Support: support@computecanada.ca

*********************************************************************

NEWS

If you are conducting research related to COVID-19, please email
support@computecanada.ca to get additional support.

Apr 2: We try to ensure everyone access to get their computational work
       completed in a timely manner but there may be some longer wait
       times while we push through urgent, time critical COVID-19 related
       jobs. Your patience is appreciated

Apr 13: Need GPU resources? Have you tried the new T4 cards?
        Add --gres=gpu:t4:1 to request a T4 class card for your GPU work.
        More details: https://docs.computecanada.ca/wiki/Graham#GPUs_on_Graham
[username@gra-login1 ~]$

Disconnect from graham with exit, and connect again with ssh graham. This time you should connect without being asked for your password or your passphrase.

Now, repeat the ssh-copy-id process to put your public key on the EOAS Ocean machines including the MOAD workstations via salish:

$ ssh-copy-id salish

This time you will need to use the EOAS Linux systems password sent to you by EOAS IT. You shouldn’t have to type in your ssh passphrase again though, because you have already decrypted your key for the ssh agent.

Note

You only have to copy your public key to one of the EOAS Ocean machines or MOAD workstations, not all of them. They all use the same authentication system, so what one knows, they all know.

If you are curious about what ssh-copy-id is doing, it is automating a bunch of steps to store your public key in a file called ~/.ssh/authorized_keys. We used to have to do those steps one by one. Life is much better with ssh-copy-id…

Copy Your Public ssh Key to GitHub

You can, and should, also use key-based authentication to access our Version Control Repositories and your personal repositories on GitHub. Doing so lets your ssh agent handle authentication when you do git pull and git push commands to copy committed changes between your local repository clones and GitHub. Please follow the instructions provided by GitHub to put your public key into your account settings on GitHub.

SSH Commands

There is a collection of commands associated with SSH:

The ones you used above to get things set up:

  • ssh-keygen - generate and work with authentication key pairs for SSH

  • ssh-copy-id - copy a local public key to a remote computer so that SSH key pair authentication can be used to log in

The one you will most often use:

ssh - log into a remote machine, or execute commands on a remote machine

Commands for copying files to/from/among your local computer and remote machines:

  • scp - an SSH-secured version of the cp command that lets you copy files from one machine to another; local to/from remote, or remote to remote

  • sftp - an SSH-secured version of the FTP file transfer protocol that provides a command-line interface for doing things navigating and creating directories on remote machines, and copying files between your local file system and that of a remote machine

To get a short reminder of the option flags for any of these commands (or most any Linux command), use the --help option; e.g.

$ ssh --help

To get a detailed description of a command (again, this works for most any Linux command) use the man command; e.g.

$ man scp

or use Google or another search engine in your browser. Searching for “man scp” should give you hits for nicely formatted versions of the same information that man scp gives you; e.g. https://linux.die.net/man/1/scp

The man command is short for “manual”, and the information it shows you is known as a “manpage”.

ssh Command

Most often you will use ssh to open a terminal session (also knowns command-line interface, CLI, or shell) on a remote computer; e.g.

$ ssh graham

You can also use ssh to execute a command on a remote computer without actually opening the terminal session; e.g.

$ ssh salish ls -lh /results2/SalishSea/nowcast-green.201905/09sep20/

That command means: “Use ssh to connect to salish and show me the long listing (including permissions, ownership, human-readable sizes, and date/time stamps) of the files in the directory there called /results2/SalishSea/nowcast-green.201905/09sep20/”.

If you get too fancy with the command that you want to execute remotely you may have to enclose it in quotes to prevent your local shell from messing it up; e.g.

$ ssh salish "find /results/forcing/atmospheric/GEM2.5/GRIB/20200909/12 -type f | wc -l"

If you have trouble with ssh not making a connection, you can tell it to output debugging messages its progress by using the -v option; e.g.

$ ssh -v graham

This is helpful in debugging connection, authentication, and configuration problems. Adding more v``s (up to 3) e.g. ``-vv, or -vvv, increases the verbosity of the messages. If you need help interpreting the output of ssh -v, paste it into a message on the SalishSeaCast #general Slack channel.

Please see ssh --help, man ssh, ask on the SalishSeaCast #general Slack channel, or Google for more information about how to use ssh.

scp Command

scp lets you copy one or more files between your local machine and a remove machine, or between two remote machines without bringing the file to your local machine.

Note

If you need to copy files between two Compute Canada cluster you should use Globus File Transfer because it is at least 4 times faster than scp on the high performance network connections among the Compute Canada clusters.

To copy a file from your current directory on your local computer to your home directory on graham use:

$ scp my-local-file graham:

You can also include a path in place of my-local-file, and/or after the colon in graham:. If you give several paths/files in place of my-local-file, all of those files will get copied to graham.

To copy a file from your current directory on your local computer to your scratch space on graham use:

$ scp my-local-file "graham:$SCRATCH/"

The quotes around "graham:$SCRATCH/" are necessary to prevent your local shell from expanding the SCRATCH environment variable.

To copy a file from your scratch space on graham to your current directory on your local computer use:

$ scp "graham:$SCRATCH/my-remote-file" ./

The ./ at the end means, this directory. It could also be some other directory path. Unlike cp, scp always has to have a destination directory for the file. Including a file name in the destination is an easy way to combine copying and renaming the copied file in one operation.

To copy a file from your scratch space on graham to your MOAD /data/ space via salish use:

$ scp "graham:$SCRATCH/my-remote-file" "salish:/data/$USER/"

The quotes around "salish:/data/$USER/" are necessary to prevent your local shell from expanding the USER environment variable.

If you have trouble with scp not making a connection, you can tell it to output debugging messages its progress by using the -v option; e.g.

$ scp -v my-local-file graham:

This is helpful in debugging connection, authentication, and configuration problems. If you need help interpreting the output of scp -v, paste it into a message on the SalishSeaCast #general Slack channel.

Please see scp --help, man scp, ask on the SalishSeaCast #general Slack channel, or Google for more information about how to use scp.

sftp Command

sftp can be used to do the same job of copying files to/form remote machines as scp. But it also provides a command-line interface to do other operations on the remote and local file systems, such as navigating directories (cd and lcd), and creating directories on the remote file system (mkdir). The command for uploading a file is put, and for downloading get. help or ? will give you a list of the available command.

Please see sftp --help, man sftp, ask on the SalishSeaCast #general Slack channel, or Google for more information about how to use sftp.

Here is a sample sftp session to copy a file from your scratch space on graham to your current directory on your local computer:

$ sftp graham
Connected to graham.
sftp> cd /scratch/username/
sftp> get my-remote-file
get my-remote-file
Fetching /home/username/my-remote-file to my-remote-file
/home/username/my-remote-file                 100%  954     7.0KB/s   00:00
sftp> quit

If you have trouble with sftp not making a connection, you can tell it to output debugging messages its progress by using the -v option; e.g.

$ sftp -v graham

This is helpful in debugging connection, authentication, and configuration problems. If you need help interpreting the output of sftp -v, paste it into a message on the SalishSeaCast #general Slack channel.

X2Go Remote Desktop

A last resort for remote access to MOAD workstations is to use X2Go (https://wiki.x2go.org/doku.php). It is a last resort because it is very bandwidth-hungry, so it can be painfully slow to use on home WiFi. It also relies on password authentication rather than key-based authentication, so it is less secure. Once authentication is completed X2Go uses SSH to secure all of the data between your local machine and the remote one.

Warning

Please be particularly cautious of X2Go’s weaker authentication security on public Wifi connections such as coffee shops, or public libraries. Since bandwidth on those types of connections is often very limited, X2Go is likely an all-round bad choice on public WiFi.

One use case that X2Go is helpful for is accessing UBC sites like the payroll system that require VPN for access from outside the UBC network. Connecting to a MOAD workstation desktop puts you inside the UBC network, so you can use Firefox on that desktop to connect to UBC the payroll system without the need for VPN on your laptop.

Please see the X2Go documentation for instructions on how to install the X2Go client for your operating system.

Once you have the client installed and running, use the Session menu to create a New Session for a specific workstation; we’ll use chum as an example.

In the Session preferences dialog enter:

  • Session name: chum

  • Host: chum.eos.ubc.ca

  • Login: Your EOAS user id

  • Session type drop-down: choose Mate

After you click the Okay button you should see a new session tile called chum on the right side of the X2Go window.

Note

Finding a Session type that works can take some trial and error. Most MOAD workstations have the Mate window manager installed, but you may have to try others. Feel free to ask for help on the SalishSeaCast #general Slack channel,

To connect to chum, click the chum session tile on the right side of the X2Go window, enter your password, and click the Okay button. If it is your first time connecting to chum you will get a Host Key verification failed alert that shows chum’s public host key hash: 85:41:ab:c4:a4:f3:7f:7b:c0:4c:8b:48:da:66:c9:5c:ec:8a:2b:69; click the Yes button to add chum’s public key to your ~/.ssh/known_hosts file.

After some time, and some logging messages appearing in a session status dialog in the main part of the X2Go window, a new window showing chum’s desktop should appear.

When you are finished with your desktop session, close the remote desktop window, and click the Cancel button in the session login dialog that replaces the session status dialog in the main part of the X2Go window.