Login to PAWS
Baton Rouge, Louisiana | Monday, November 23rd, 2009
LSU Header
HPC@LSU

About Philip

Philip is a 3.5 TFlops Peak Performance 37 compute node cluster running the RHEL 5 operating system. Each node contains two Quad Core Nehalem Xeon 64-bit processors operating at a core frequency of 2.93 GHz. 3 compute nodes have 96GB 1066MHz Ram, 2 compute nodes have 48GB 1066MHz Ram, and 32 compute nodes have 24GB 1333MHz Ram.

Philip's intended use is for single processor / SMP jobs that require a large amount of memory. Although the cluster is connected with a 1 Gb ethernet interconnection, MPI jobs would be better served by either Tezpur or Pelican.

Access to phililp

*nix and Mac Users - One would issue a command similar to the following: 

$ ssh -X username@philip.hpc.lsu.edu
The user would then be prompted for his password. The -X flags allow for X11 Forwarding to be set up automatically.

Windows Users - For a Windows client please use the PuTTY utility.

If you have forgotten your password, or you wish to reset it, see here (click "Forgot your password?").

User Environment.

The LSU HPC systems make use of softenv to allow for adding software to the user's environment. Executing softenv on philip will display the available software: 

$ softenv
  +ImageMagick-6.4.6.9-intel-11.1
  +ParMetis-3.1.1-intel-11.1-mpich-1.2.7p1
  +R-2.8.1-gcc-4.3.2
  ...
In order to add software to your environment, you'll need to add the appropriate key to your ~/.soft file. For example, to add the package ImageMagick to your user environment, you would need to add the following: 
$ cat ~/.soft
  +ImageMagick-6.4.6.9-intel-11.1
  @default
The order in which you add keys to ~/.soft is important. The first occurrence of a setting takes presedence.

Once the entries are to your liking, you must then execute the command resoft, i.e.: 

$ resoft
If your code needs to link to a library of given package, you will find all software installed under /usr/local/packages/, e.g.: 
$ ls /usr/local/packages/
  apache_ant  boost     fuse     gold      hdf5         ...
  arpack      boostjam  gamess   graphviz  hypre        
  atlas       condor    git      gromacs   ImageMagick  
  blacs       fftw      gnuplot  gsl       iozone

File Storage

Home Directory

The /home file system quota is 5 GB. Files can be stored on /home permanently, which makes it an ideal place for your source code and executables. The /home file system is meant for interactive use such as editing and active code development. Do not use /home for batch job I/O.

Work (Scratch) Directory

The /work volume meant for the input and output of executing batch jobs and not for long term storage. We expect files to be copied to other locations or deleted in a timely manner, usually within 30-120 days. For performance reasons on all volumes, our policy is to limit the number of files per directory to around 10,000 and total files to about 500,000.

There is no quota is enforced on /work but if it becomes over utilized we will enforce a 30 days purging policy, which means that any files that have not been accessed for the last 30 days will be permanently deleted. An email message will be sent out weekly to users targeted for a purge informing them of their /work utilization.

Please do not try to circumvent the removal process by date changing methods. We expect most files over 30 days old to disappear. If you try to circumvent the purge process, this may lead to access restrictions to the /work volume or the cluster.

Please note that the /work volume is not unlimited. Please limit your usage rate to a reasonable amount. When the utilization of /work is over 80%, a 14 day purge may be performed on users using more than 2 TB or having more than 500,000 files. Should disk space become critically low, all files not accessed in 14 days will be purged or even more drastic measures if needed. Users using the largest portions of the /work volume will be contacted when problems arise and they will be expected to take action to help resolve issues.

Programming/Compiling

Version 11.1 of the Intel compilers are loaded by default, codes can be compiled according to the following chart:

Serial Codes MPI Codes OpenMP Codes Hybrid Codes
Fortran ifort mpif90 ifort -openmp mpif90 -openmp
C icc mpicc icc -openmp mpicc -openmp
C++ icpc mpiCC icpc -openmp mpiCC -openmp

Running Jobs

There are 3 possible job queues to choose from:

All nodes have a limit of 72 hours of wallclock time.

Single Queue Job Script Template

$ cat ~/script

#!/bin/bash
#PBS -q single
#PBS -l nodes=1:ppn=1 
#PBS -l walltime=HH:MM:SS
#PBS -o desired_output_file_name
#PBS -N NAME_OF_JOB

/path/to/your/executable

Workq Queue Job Script Template

$ cat ~/script

#!/bin/bash
#PBS -q workq
#PBS -l nodes=1:ppn=8
#PBS -l walltime=HH:MM:SS
#PBS -o desired_output_file_name
#PBS -j oe 
#PBS -N NAME_OF_JOB

# mpi jobs would execute: mpirun -np 8 -machinefile $PBS_NODEFILE /path/to/your/executable
# OpenMP jobs would execute: export OMP_NUM_THREADS=8; /path/to/your/executable

Bigmem Queue Job Script Template

$ cat ~/script

#!/bin/bash
#PBS -q bigmem
# Request to use a 48 GB node, similarly could request mem96
#PBS -l nodes=1:ppn=8:mem48
#PBS -l walltime=HH:MM:SS
#PBS -o desired_output_file_name
#PBS -j oe 
#PBS -N NAME_OF_JOB

# mpi jobs would execute: mpirun -np 8 -machinefile $PBS_NODEFILE /path/to/your/executable
# OpenMP jobs would execute: export OMP_NUM_THREADS=8; /path/to/your/executable
Submit the job by executing: 
$ qsub script

Monitoring Jobs

The following commands can be used to view/modify the queue