Advanced job submission options

• Whenever possible, the user should try to declare in advance the resource share that it will need to execute his task. This will allow a better matching between resource and user needs.
• Generally, users know in advance the time his application takes to execute, the size of files it generated or the memory the application requires. Declaring these types of variables prevents the job to be executed in a resource which does not fulfils the user requirement.

• These requirements can be issued at submission time ('qsub -l <requirement>' my_scrupt.sh) or by directly adding '#$ -l <requirement>' to the submitted script.

• The following examples demonstrate generic use cases on how to issue such requirements:

# Request for 1G of free virtual memory. The system will schedule the task in a machine with 1 GB of virtual memory free but the job will be killed if it uses more than on 1G.
$ qsub -l h_vmem=1G <your script>

# Request for 1G of virtual memory and 20G of free space. The job will be killed if it needs more resources that it requested.
$ qsub -l h_vmem=1G,h_fsize=20G <your script>

# Request for 30m of wallclock time. The job will be killed if it uses more than 30m.
$ qsub -l s_rt=00:30:00 <your script>

Job Wrappers

• Some users use a personal wrapper script to generate a arrays of submission scripts.
• On this case, the user may loose some aliases definitions because, by default, only an interactive bash inherits the aliases definitions. That problem can be bypassed if the user adds the following settings to his personal wrapper.

# ! /bin/bash -l
shopt -s expand_aliases

• Here is an example of a very simple job wrapper that should be executed as my_job_wrapper.sh 1 10

  • Generates and submits 10 different scripts with 10 different input files and output files.

$ cat my_job_wrapper.sh
# ! /bin/bash
shopt -s expand_aliases

i=$1
while [ $i -le $2 ]; do
   cat << EOF > job.sh.$i
#\$ -v SGEIN1=input_file$i.txt
#\$ -v SGEOUT1=output_file$i.txt
cat input_file$i.txt >> output_file$i.txt
EOF
   qsub job.sh.$i
   rm -f job.sh.$i
   i=`expr $i + 1`
done

exit 0