This document describes how to configure your lemuria.cis.vermontstate.edu account for effective access to the cluster. You will need to do the steps here to make MPI development easier (for example).
Lemuria is your access point to the cluster nodes. Those nodes are located on a private network that connects them to each other and to Lemuria's secondary network interface. The cluster nodes are named node1, node2, node3, and node4. They use NIS to obtain user account information from Lemuria; you can use the same credentials to log into a cluster node as you use for Lemuria access. Similarly, the cluster nodes use NFS to mount your home directory, so any files you create on Lemuria (or on any node) are visible on all the other nodes. This is a convenient way to distribute files among the nodes.
Because the cluster nodes are also running Ubuntu Linux 22.04 on the x86_64 architecture, any programs compiled on Lemuria should also run on the nodes without a need for recompilation (the CPU architectures and system libraries are compatible). This is typically the situation in a cluster configuration, although sometimes collections of heterogeneous computers are used for parallel computing where this is not true. In those cases, it is necessary to compile a different version of the software for each node, which adds extra complexity to the deployment process.
You can easily SSH from Lemuria to any cluster node, or from one cluster node to another. Use the node name (listed above) as the name of the host and use your Lemuria username and password to log into that node. Once you log in, you will find that all your Lemuria files in your home directory are available to you as usual.
Note that, since your username is the same on all nodes, you don't have to specify your username when using SSH. For example:
jjones@lemuria:~$ ssh node1
Here, a hypothetical user 'jjones' on Lemuria uses the SSH client to log into node1. She does not need to say ssh jjones@node1 since the usernames on both systems are the same.
It is also possible to execute a program on a remote node with SSH without first getting a shell on that node. For example:
jjones@lemuria:~$ ssh node1 'ls -l'
This executes the ls -l command on node1 with the output displayed on the Lemuria terminal. The quotation marks are needed because the command contains a space (or other special character).
To use the nodes easily, for example, with MPI programs, you will want to configure your account so you can log into the cluster nodes without supplying a password. This will allow the mpirun tool to automatically launch MPI programs on other nodes on your behalf (it uses SSH to remotely execute commands in a way similar to above). Without this extra configuration, you will be prompted for a password for each node that you involve in your program. That is highly undesirable in a large cluster with hundreds (or even thousands) of nodes.
The first step is to set up public-key authentication. You only need to do this only once. Log into Lemuria and proceed as follows:
Execute ssh-keygen. This command will generate an RSA public/private key pair that OpenSSH can use. The private key will be stored in ~/.ssh/id_rsa and the public key will be stored in ~/.ssh/id_rsa.pub. When you generate the key pair, you will be prompted for a passphrase. Normally, this passphrase is used to encrypt your private key. In our case, just type "Enter" when prompted for a password to create an empty passphrase! This is not as severe a security problem as it sounds, since to use your OpenSSH key someone needs to access the ~/.ssh/id_rsa file which should be safely stored in your personal home directory (you might verify there is no public read access to that file).
Ordinarily, you would copy the public key to the machines where you want to log in. However, in this case, those machines share files with Lemuria, so you can "distribute" the public key by just copying ~/.ssh/id_rsa.pub to the file ~/.ssh/authorized_keys.
Check that public-key authentication works. From Lemuria SSH to one of the nodes. You should be immediately logged in. However, you may be prompted about the remote host key and asked if you want to continue connecting. Answer "yes" to the prompt to save the remote host key to the ~/.ssh/known_hosts file. Type exit to log out of a node. Log into all four nodes to save all the host keys.
If you are prompted for your ordinary password, it means the SSH server on the remote host doesn't realize you've created a public/private key pair. In that case, check the permissions on your home directory and on the ~/.ssh directory. Your home directory should not be writable by other users, and the ~/.ssh directory should have 700 permissions. The OpenSSH server will not use a key if it is stored in what it feels to be an insecure way.
If you are asked for a phassphrase instead of a password, that means you must have specified a passphrase when you generated the public/private key pair. For our purposes, you should use an empty passphrase instead. Delete the key files and generate a fresh key pair.
Normally, people do use passphrases on the private key file. It is possible to still use password-less logins in that case with the help of the SSH agent. For completeness, I describe the steps for doing that below. However! The OpenMPI process launcher doesn't, by default, work properly in this situation. The problem is that the launcher will try to launch processes in a tree like structure... starting at Lemuria as the root it will launch processes on "neighboring" nodes as child nodes. Then those child nodes will launch processes on their neighbors in parallel, etc. This approach greatly speeds up process launching in a large cluster where there might be thousands of nodes. But the OpenMPI process launcher doesn't know about setting up the SSH agent on the other nodes, so launching fails. There are ways to work around this, but they are complicated and not necessary for our purposes. This is why I suggest using empty passphrases rather than the SSH agent, despite the slight security risk.
If you want to play with the SSH-agent, do the following:
At a shell prompt on Lemuria execute the command eval $(ssh-agent) followed by the command ssh-add. The first command runs the SSH agent in the background, and the second command loads your private key into the agent. When you execute ssh-add you will be prompted for your passphrase (which no longer needs to be empty).
The reason for the use of eval is to allow the SSH agent to set an environment variable in your shell holding its process ID number. This is how other programs can locate the agent. You will need to execute a fresh agent process, and load it with the keys you want to use, every time you create a new shell.
You can now SSH to any other node without supplying a password. The SSH agent will take care of authenticating you automatically. Don't forget that you must start the agent, etc., each time you log in, and before you attempt to run an MPI program!
To test your configuration, log into Lemuria and build and run the HelloMPI sample program as follows:
$ make
$ mpirun --mca btl_tcp_if_include 10.0.0.0/24 -np 4 \
--host node1,node2,node3,node4 main
The mpirun command launches MPI programs. The -np option specifies how many processes to create. The --host option lists the machines on which to create those processes. The name of the program is main.
At the time of this writing, there is an issue with the command waiting for a timeout before proceeding. Thus, the startup time is very long, which is annoying. Based on information in the OpenMPI FAQ, I believe the problem is due to the virtual network interfaces on Lemuria. The mpirun command is attempting to communicate with nodes on those interfaces and timing out. At least that is my working hypothesis. According to the OpenMPI FAQ, this issue can be addressed by specifying which network(s) to use for MPI communication. That is the point of the -mca btl_tcp_if_include 10.0.0.0/24 option. It configures the Module Component Architecture (MCA) Byte Transfer Layer (BTL) to use TCP only on the specified network. Unfortunately, this doesn't seem to help. The matter is under investigation.
The problem can be worked around by running mpirun on one of the nodes. This is only slightly less convenient and doesn't impact the execution performance. The program runs on only the nodes anyway, as listed in the --host option; mpirun program just serves as a gathering point for the output of the entire job.
Note that you should use mpic++ to compile C++ programs that use MPI.
Last Revised: 2024-01-05
© Copyright 2024 by Peter Chapin <peter.chapin@vermontstate.edu>