Quick Start

This brief document shows how to run the cluster default application that is written in the ​​C, C++ or Fortran languages and uses MPI.

IMPORTANT Be sure to read the full version of this documentation!

To run the task on a cluster you must:

  1. Download the source code and input data to cluster.

  2. Select and load environment modules: compiler, implementation of MPI, additional libraries (if needed).

  3. Compile the program.

  4. Create a job script.

  5. Add a task to the run queue.

Download the source code and data to the cluster

For this purpose, you can use PuTTY and pscp or WinSCP if you have Microsoft Windows OS installed, or ssh, scp and sftp if you have Linux OS installed.

It is recommended to create separate directories for different applications. For example, if your login user and you want to create a directory example, run the command mkdir (Make Directory):

[User @ n001 ~] $ mkdir /home/user/example

Put the source code of the program to directory just created.

Go to the directory created by the command cd (Change Directory):

[User @ n001 ~] $ cd /home/user/example

Select and load environment modules

There is a number of different compilers for C, C++, Fortran, several implementations of MPI, several implementations of Java and Python, and other software components installed on cluster. Of course, you can not simultaneously use two different implementations of MPI. Therefore, in order to have an ability to use different software components the environment modules system is used.

Environment module is a software component, such as compiler or implementation of MPI. To control modules the command module is used. All of the changes made by module command apply only for the current session. After the session has ended all changes are forgotten.

Immediately after the start of the new session there are only HPCC system module and resource management system module are loaded:

[User @ n001 ~] $ module list
Currently Loaded Modulefiles:
1) HPCC / January 2) SLURM/2.6.0

Now you will not even have a compiler available:

[User @ n001 ~] $ cc
bash: cc: command not found

[User @ n001 ~] $ gcc
bash: gcc: command not found

In order to get a default set of applications, load the environment module cgmvolf 1.1.11rc1, which includes Clang 3.2, GFortran 4.7.2, MVAPICH2 1.9rc1, and some other libraries:

[User @ n001 ~] $ module load cgoolf/1.1.11rc1

Note that loading of environment module is meant to connect software components that are already installed on the cluster. You do not need to download any programs from your local computer or the Internet yourself.

After loading the module cgmvolf you can check that Clang compiler became available:

[User @ n001 ~] $ clang - version
clang version 3.2 (tags/RELEASE_32/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix

WARNING cgmvolf program set is just one of the possible combinations of compiler + MPI. Your program can work better with other compilers or sale MPI, established on the cluster. Read the full documentation and make reasonable choices. Unfortunately, for various applications the optimal combination is different. You should try different options and experiment to discover which option is best for your application.

Compiling Programs

If the program is written by third-party developer, the instructions for compiling can be supplied along with the program.

Program in C, which is written using MPI, is compiled as follows:

[User @ n001 example] $ mpicc -W -Wall -O2 -std=c99 file_name_1.c file_name_2.c ... -o program

Where file_name_1.c, file_name_2.c - the names of all the files that make up the program; program - name for the executable which is created by compiler.

Do not forget to go to the directory with the source code before compiling.

If the program is written in C++, you must use mpic++ compiler.

For programs in Fortran mpif77 or mpif90 compilers should be used.

Create a job script

Create the file job.slurm in the directory /home/user/example with the following text:

#! /bin/bash -l
# SBATCH -J example
# SBATCH -o example.out.%j
# SBATCH -e example.out.%j
# SBATCH -n 8
# SBATCH -p debug
# SBATCH --qos=ku-debug
# SBATCH -t 00:05:00

set -x
set -e
module list 2>&1

mpirun ./a.out

You must pass -l option to command interpreter in the first line, otherwise the environment modules system will not function correctly.

WARNING Do not send command mpirun or mpiexec option -n or -np to specify the number of cores. OpenMPI and MVAPICH2 integrated with SLURM resource manager and this information will be passed automatically.

Lines starting with #SLURM are comments for command interpreter. They set the job options. These options are described in the sbatch man-page:

[User @ n001 ~] $ man sbatch

The most important parameters are:

  • -J example - specifies the name of the task. It is not mandatory, but helps distinguish jobs in the queue.

  • -o example.out.%j and -e example.out.%j - set the file name to save the output and error program (names may be the same or different). Text % j is replaced by a job identifier. This parameter is optional, the default will create a file name that begins with slurm.

  • -n 8 - specifies the number of cores.

Other parameters are described in the full version of the documentation.

After #SBATCH lines commands to run your application are set. These commands will be run in the directory from which the task was added to the queue. To explicitly specify a working directory, add cd /home/user/example before actual program invocation. You can also add commands for loading environment modules.

Add a task to the queue:

[User @ n001 example] $ sbatch job.slurm
Submitted batch job 88

88 - identifier of your job.

Use the command squeue to view the status of your job. Output format is described in the full documentation.

After job has completed review its output files (example.out.%j) and results files that were created by your program.