Contents
Jobs submission from a non shared directory
The LIP home directiries are not shared between the submit nodes and the execution nodes, the input files or directories must be transfered into the worker nodes, and at job completion the output files or directories should be transfered out of the execution node. The job runs on a temporary local directory which is cleaned at job completion. The information about the transfers into and out of worker nodes is provided with the directives INPUT and OUTPUT as shown bellow.
Prepare the submit script
For the sake of this example lets suppose we need to execute a root macro. The simulation input file is MyMacro.c and this macro execution will produce an output file named graph_with_law.pdf.
The working directory will start with two files, MyMacro.c and MyMacro.sh:
1 [uname@pauli01 wdir]$ ls -l
2 -rw-r--r-- 1 uname ugroup 1822 Feb 19 12:08 MyMacro.c
3 -rw-r--r-- 1 uname ugroup 310 Feb 19 12:07 MyMacro.sh
4
5 [uname@pauli01 mdir]$ cat MyMacro.sh
6 #!/bin/bash
7
8 # --------------------------------------------------------------------- #
9 # Select partition, or queue. The LIP users should use "lipq" partition
10 #SBATCH -p lipq
11
12 # Transfer input files to the execution machine
13 # INPUT = MyMacro.c
14
15 # Transfer output files back to submission machine
16 # OUTPUT= graph_with_law.pdf
17 # --------------------------------------------------------------------- #
18
19 # Load environment
20 module load root/6.18.04
21
22 # Execute macro
23 root -b -q MyMacro.c
Submit the job
The SLURM submission command is sbatch, check the manual page for detais, in this case we would execute:
[uname@pauli01 wdir]$ sbatch MyMacro.sh Submitted batch job 350327
The job was submitted to the cluster with job identifiel 350327.
Check job status
There are two commands to check the job status, squeue and scontrol, on most cases we will use the first, the second is more detailed and used when something is wrong on the job placement.
We will execute squeue multiple times to check the job status as it moves throught the scheculer cycle, when the job disapears or presente the status C then it is finished.
[uname@pauli01 mdir]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 350327 lipq MyMacro. uname PD 0:00 1 (None) [uname@pauli01 mdir]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 350327 lipq MyMacro. uname R 0:05 1 wn168 [uname@pauli01 mdir]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
As we can see the job wait a bit on pending state, PD, then it was executed, state R, on workernode wn168, and finally it finish.
The second command would be run as following and would give you a detail very prolix result:
[uname@pauli01 mdir]$ scontrol show job 350327 UserId=uname(5800002) GroupId=ugroup(5800000) MCS_label=N/A Priority=4578449 Nice=0 Account=ugroup QOS=normal JobState=PENDING Reason=None Dependency=(null) ....
Cancel job
If for any reason you need to cancel the job we would use the command scancel:
[martinsj@pauli01 mdir]$ scancel 320327
Check job output
If the standard output file name is not specified then the job standard output will be written to a file name of form:
slurm-<JOB_ID>.out
where JOB_ID is the job identifier.
For the time being the standard error is merged with the standard output, please expect out of order standard error messages mixed with the standard output.
Check your working directory on submit node and you will find new files:
[uname@pauli01 mdir]$ ls -l -rw-r--r-- 1 uname ugroup 15297 Feb 19 13:34 graph_with_law.pdf -rw-r--r-- 1 uname ugroup 1822 Feb 19 12:08 MyMacro.c -rw-r--r-- 1 uname ugroup 484 Feb 19 12:15 MyMacro.sh -rw-r--r-- 1 uname ugroup 2100 Feb 19 12:15 slurm-350327.out
One of the files is the standard output file, slurm-350327.out, and the other is the provided output file on submission script, # OUTPUT=graph_with_law.pdf.
The file slurm-350327.out have three sections. The first section is the prolog section with some generic informations about the job, including the job identifier, the used partition, the workernode, the stagein files transfered, the time the job started, and some other details. The following section is the job output. Finally, the last section is the epilog section with the time of job completion and the stageout files transfered.
Job script details
The submission script is comprehend by two sections, the directives section, normally at script top with, and the working section where the user will place his work.
Directive section
SLURM directives start with '#SBATCH' and encode the sbatch command line options, this is a more convenient way to configure jobs as needed. This is a simple example, check the manual page to get familiar with more complex options.
The INPUT and OUTPUT parameters are in reality environment variables that could be supplied on the command line interface as
sbatch --export=INPUT=MyMacro.c,OUTPUT=graph_with_law.pdf
this encoded on the submit script is more convenient for a daily operation.
1 # --------------------------------------------------------------------- #
2 # Select partition, or queue. The LIP users should use "lipq" partition
3 #SBATCH -p lipq
4
5 # Transfer input files to the execution machine
6 # INPUT = MyMacro.c
7
8 # Transfer output files back to submission machine
9 # OUTPUT= graph_with_law.pdf
10 # --------------------------------------------------------------------- #
11
#SBATCH -p lipq
This directive select the running partition, or queue, for LIP users it will be always lipq, any other partition name will fail.
# INPUT = MyMacro.c
This select the input file for the job, on the old sge cluster we used SGEIN. Multiple lines for other input files it is accepted, it also accept directory names instead of file names.
The sintaxe rules for this parameter is
# INPUT = sub_name[:exe_name]
where sub_name is the name of input file or directory on the submission node and exe_name is the file or directory name on the execution node. This name on the execution node is optional and if not provided then it assumed to be equal to sub_name.
The INPUT parameter name have the following alias: IN, SGEIN and SLURMIN. Furthermore, accept any number of digits follwoing the name, e.g. INPUT2 ou IN1034.
# OUTPUT = graph_with_law.pdf
This directive defines the output file or directory the user need to recover, it is equivalent to SGEOUT on the old sge cluster. As for the INPUT parameter, the OUTPUT may be provided multiple times and also accept any number of digits following the name. The following alias are accepted: OUT, SGEOUT and SLURMOUT.
The sintaxe rules for this parameter is similar to INPUT but with a little difference or the order of files or directory names, the name on execution node came's first.
# OUTPUT = exe_name[:sub_name]
This parameter is mandatory because the working directory on execution nodes is clean on job completion, if the output files or directories are not retrieved then they will be lost! |
Working section
This is the section where the user will insert the job procedures, in this case load the root environment in order to access the application installation and run the macro.