This brief document shows how to run the cluster default application that is written in the C, C++ or Fortran languages and uses MPI.
IMPORTANT Be sure to read the full version of this documentation!
To run the task on a cluster you must:
Download the source code and input data to cluster.
Select and load environment modules: compiler, implementation of MPI, additional libraries (if needed).
Compile the program.
Create a job script.
Add a task to the run queue.
For this purpose, you can use
WinSCP if you have Microsoft Windows OS installed, or
sftp if you have Linux OS installed.
It is recommended to create separate directories for different applications. For example, if your login
user and you want to create a directory
example, run the command mkdir (Make Directory):
[User @ n001 ~] $ mkdir /home/user/example
Put the source code of the program to directory just created.
Go to the directory created by the command
cd (Change Directory):
[User @ n001 ~] $ cd /home/user/example
There is a number of different compilers for C, C++, Fortran, several implementations of MPI, several implementations of Java and Python, and other software components installed on cluster. Of course, you can not simultaneously use two different implementations of MPI. Therefore, in order to have an ability to use different software components the environment modules system is used.
Environment module is a software component, such as compiler or implementation of MPI. To control modules the command
module is used. All of the changes made by
module command apply only for the current session. After the session has ended all changes are forgotten.
Immediately after the start of the new session there are only HPCC system module and resource management system module are loaded:
[User @ n001 ~] $ module list Currently Loaded Modulefiles: 1) HPCC / January 2) SLURM/2.6.0
Now you will not even have a compiler available:
[User @ n001 ~] $ cc bash: cc: command not found [User @ n001 ~] $ gcc bash: gcc: command not found
In order to get a default set of applications, load the environment module cgmvolf 1.1.11rc1, which includes Clang 3.2, GFortran 4.7.2, MVAPICH2 1.9rc1, and some other libraries:
[User @ n001 ~] $ module load cgoolf/1.1.11rc1
Note that loading of environment module is meant to connect software components that are already installed on the cluster. You do not need to download any programs from your local computer or the Internet yourself.
After loading the module cgmvolf you can check that Clang compiler became available:
[User @ n001 ~] $ clang - version clang version 3.2 (tags/RELEASE_32/final) Target: x86_64-unknown-linux-gnu Thread model: posix
WARNING cgmvolf program set is just one of the possible combinations of compiler + MPI. Your program can work better with other compilers or sale MPI, established on the cluster. Read the full documentation and make reasonable choices. Unfortunately, for various applications the optimal combination is different. You should try different options and experiment to discover which option is best for your application.
If the program is written by third-party developer, the instructions for compiling can be supplied along with the program.
Program in C, which is written using MPI, is compiled as follows:
[User @ n001 example] $ mpicc -W -Wall -O2 -std=c99 file_name_1.c file_name_2.c ... -o program
file_name_2.c - the names of all the files that make up the program;
program - name for the executable which is created by compiler.
Do not forget to go to the directory with the source code before compiling.
If the program is written in C++, you must use
For programs in Fortran
mpif90 compilers should be used.
Create the file
job.slurm in the directory
/home/user/example with the following text:
#! /bin/bash -l # SBATCH -J example # SBATCH -o example.out.%j # SBATCH -e example.out.%j # SBATCH -n 8 # SBATCH -p debug # SBATCH --qos=ku-debug # SBATCH -t 00:05:00 set -x set -e module list 2>&amp;1 mpirun ./a.out
You must pass
-l option to command interpreter in the first line, otherwise the environment modules system will not function correctly.
WARNING Do not send command
-np to specify the number of cores. OpenMPI and MVAPICH2 integrated with SLURM resource manager and this information will be passed automatically.
Lines starting with
#SLURM are comments for command interpreter. They set the job options. These options are described in the
[User @ n001 ~] $ man sbatch
The most important parameters are:
-J example - specifies the name of the task. It is not mandatory, but helps distinguish jobs in the queue.
-o example.out.%j and
-e example.out.%j - set the file name to save the output and error program (names may be the same or different). Text
% j is replaced by a job identifier. This parameter is optional, the default will create a file name that begins with
-n 8 - specifies the number of cores.
Other parameters are described in the full version of the documentation.
#SBATCH lines commands to run your application are set. These commands will be run in the directory from which the task was added to the queue. To explicitly specify a working directory, add
cd /home/user/example before actual program invocation. You can also add commands for loading environment modules.
Add a task to the queue:
[User @ n001 example] $ sbatch job.slurm Submitted batch job 88
88 - identifier of your job.
Use the command
squeue to view the status of your job. Output format is described in the full documentation.
After job has completed review its output files (
example.out.%j) and results files that were created by your program.