gnuparallel
Table of Content
Versions and Availability
h4
h5
Module Names for gnuparallel on smic
| Machine | Version | Module Name |
|---|---|---|
| smic | 20190222 | gnuparallel/20190222/intel-19.0.5 |
▶ Module FAQ?
▶ Did not find the version you want to use??
About the Software
GNU parallel is a shell tool for executing jobs in parallel using one or more computers
Usage
Parallel typical serial and MPI-based applications.
(1) Parallel serial jobs
Example of a blast job on Mike:
#!/bin/bash
#PBS -A hpc_smictest3
#PBS -l nodes=2:ppn=16
#PBS -l walltime=1:00:00
#PBS -q workq
cd $PBS_O_WORKDIR
export JOBS_PER_NODE=16
export WDIR=$PBS_O_WORKDIR
parallel --progress \ # shows progres
--joblog logfile \ # job logfile
-j $JOBS_PER_NODE \ # jobs per node
--slf $PBS_NODEFILE \ # nodes assigned to your job
--workdir $WDIR \
./cmd_blast.sh {} {/.} :::: input.lst #script_to_parallize input output joblist
where: input.lst contains job input list:
/work/$USER/blast/data/input1.faa /work/$USER/blast/data/input2.faa .... /work/$USER/blast/data/input200.faa
where: cmd_blast.sh is the script for running a serial blast job
e.g.: ./cmd_blast.sh input1.faa input1 -- how to run single serial job
#!/bin/bash export WDIR=/xxx/xxx cd $WDIR blastp -query $1 -db db/img_v400_PROT.00 -out output/$2.out -outfmt 7 -max_target_seqs 100 -num_threads 2
(2) Parallel MPI jobs
Use "mpirun" to run a laplace
#!/bin/bash
#PBS -A your_allocation_name
#PBS -l walltime=2:00:00
#PBS -l nodes=4:ppn=16
#PBS -q checkpt
export JOBS_PER_NODE=8
export NPROCS=2
export WDIR=$PBS_O_WORKDIR
cd $WDIR
parallel --progress \
-j $JOBS_PER_NODE \
--slf $PBS_NODEFILE \
--workdir $WDIR \
./cmd_mpi.sh {} $NPROCS :::: input.lst
where: cmd_mpi.sh is the script to run one MPI job
#!/bin/bash
export WDIR=$PBS_O_WORKDIR
FILE=$(eval echo $1)
param=`cat ${FILE}`
mpirun -ppn $2 $WDIR/lap_mpi $param
where: input.lst contains job input list:
/work/$USER/laplace/data/input1 /work/$USER/laplace/data/input2 .... /work/$USER/laplace/data/input200 cat input1: 4096 4096 2 2 0.08 20000 0 0
Resources
Last modified: September 10 2020 11:58:50.