High Performance Computing

 #!/bin/bash
 
#PBS -A your_allocation
#PBS -q single
#    Note: a single queue is not present on Queen Bee,
#    use workq or checkpt, but you will be charged for all cores.
#PBS -M your_email
# Change ppn to match cluster (4, 8 or 16) if no single queue.
#PBS -l nodes=1:ppn=1
#PBS -l walltime=06:00:00
#PBS -V
#PBS -o AutoGrid_test.out
#PBS -e AutoGrid_test.err
#PBS -N autogridtest
 
export EXEC=autogrid4 
export INPUT=hsg1.gpf
export OUTPUT=hsg1.glg
export WORK_DIR=$PBS_O_WORKDIR
 
cd $WORK_DIR
 
$EXEC -p $INPUT -l $OUTPUT

Portable Batch System: qsub

qsub

All HPC@LSU clusters use the Portable Batch System (PBS) for production processing. Jobs are submitted to PBS using the qsub command. A PBS job file is basically a shell script which also contains directives for PBS.

Usage

$ qsub job_script

Where job_script is the name of the file containing the script.

PBS Directives

PBS directives take the form:

#PBS -X value

Where X is one of many single letter options, and value is the desired setting. All PBS directives must appear before any active shell statement.

Example Job Script

 #!/bin/bash
 #
 # Use "workq" as the job queue, and specify the allocation code.
 #
 #PBS -q workq
 #PBS -A your_allocation_code
 # 
 # Assuming you want to run 16 processes, and each node supports 4 processes, 
 # you need to ask for a total of 4 nodes. The number of processes per node 
 # will vary from machine to machine, so double-check that your have the right 
 # values before submitting the job.
 #
 #PBS -l nodes=4:ppn=4
 # 
 # Set the maximum wall-clock time. In this case, 10 minutes.
 #
 #PBS -l walltime=00:10:00
 # 
 # Specify the name of a file which will receive all standard output,
 # and merge standard error with standard output.
 #
 #PBS -o /scratch/myName/parallel/output
 #PBS -j oe
 # 
 # Give the job a name so it can be easily tracked with qstat.
 #
 #PBS -N MyParJob
 #
 # That is it for PBS instructions. The rest of the file is a shell script.
 # 
 # PLEASE ADOPT THE EXECUTION SCHEME USED HERE IN YOUR OWN PBS SCRIPTS:
 #
 #   1. Copy the necessary files from your home directory to your scratch directory.
 #   2. Execute in your scratch directory.
 #   3. Copy any necessary files back to your home directory.

 # Let's mark the time things get started.

 date

 # Set some handy environment variables.

 export HOME_DIR=/home/$USER/parallel
 export WORK_DIR=/scratch/myName/parallel
 
 # Set a variable that will be used to tell MPI how many processes will be run.
 # This makes sure MPI gets the same information provided to PBS above.

 export NPROCS=`wc -l $PBS_NODEFILE |gawk '//{print $1}'`

 # Copy the files, jump to WORK_DIR, and execute! The program is named "hydro".

 cp $HOME_DIR/hydro $WORK_DIR
 cd $WORK_DIR
 mpirun -machinefile $PBS_NODEFILE -np $NPROCS $WORK_DIR/hydro

 # Mark the time processing ends.

 date
 
 # And we're out'a here!

 exit 0

 #!/bin/bash
 
#PBS -A your_allocation
#PBS -q single
#    Note: a single queue is not present on Queen Bee,
#    use workq or checkpt, but you will be charged for all cores.
#PBS -M your_email
# Change ppn to match cluster (4, 8 or 16) if no single queue.
#PBS -l nodes=1:ppn=1
#PBS -l walltime=06:00:00
#PBS -V
#PBS -o AutoDock_test.out
#PBS -e AutoDock_test.err
#PBS -N autogridtest
 
export EXEC=autodock4 
export INPUT=ind.dpf
export OUTPUT=ind.dlg
export WORK_DIR=$PBS_O_WORKDIR
 
cd $WORK_DIR
 
$EXEC -p $INPUT -l $OUTPUT

PBS Job Chains

Quite often, a single simulation requires multiple long runs which must be processed in sequence. One method for creating a sequence of batch jobs is to execute the qsub to submit its successor. We strongly discourage recursive, or "self-submitting," scripts since for some jobs, chaining isn't an option. When your job hits the time limit, the batch system kills them and the command to submit a subsequent job is not processed.

PBS allows users to move the logic for chaining from the script and into the scheduler. This is done with a command line option:

$ qsub -W depend=afterok:<jobid> <job_script>

This tells the job scheduler that the script being submitted should not start until jobid completes successfully. The following conditions are supported:

afterok:<jobid>

Job is scheduled if the job <jobid> exits without errors or is successfully completed.

afternotok:<jobid>

Job is scheduled if job <jobid> exited with errors.

afterany:<jobid>

Job is scheduled if the job <jobid> exits with or without errors.

One method to simplify this process is to write multiple batch scripts, job1.pbs, job2.pbs, job3.pbs etc and submit them using the following script:

#!/bin/bash
 
FIRST=$(qsub job1.pbs)
echo $FIRST
SECOND=$(qsub -W depend=afterany:$FIRST job2.pbs)
echo $SECOND
THIRD=$(qsub -W depend=afterany:$SECOND job3.pbs)
echo $THIRD

Modify the script according to number of job chained jobs required. The Job <$FIRST> will be placed in queue while the jobs <$SECOND> and <$THIRD> will be placed in queue with the "Not Queued" (NQ) flag in Batch Hold. When <$FIRST> is completed, the NQ flag will be replaced with the "Queued" (Q) flag and will be moved to the active queue.

A few words of caution: If you list the dependency as "afterok"/"afternotok" and your job exits with/without errors then your subsequent jobs will be killed due to "dependency not met".