High Performance Computing

The information here is applicable to LSU HPC and LONI systems.

h4

Shells

A user may choose between using /bin/bash and /bin/tcsh. Details about each shell follows.

/bin/bash

System resource file: /etc/profile

When one access the shell, the following user files are read in if they exist (in order):

1. ~/.bash_profile (anything sent to STDOUT or STDERR will cause things like rsync to break)
2. ~/.bashrc (interactive login only)
3. ~/.profile

When a user logs out of an interactive session, the file ~/.bash_logout is executed if it exists.

The default value of the environmental variable, PATH, is set automatically using Modules. See below for more information.

/bin/tcsh

The file ~/.cshrc is used to customize the user's environment if his login shell is /bin/tcsh.

Modules

Modules is a utility which helps users manage the complex business of setting up their shell environment in the face of potentially conflicting application versions and libraries.

Default Setup

When a user logs in, the system looks for a file named .modules in their home directory. This file contains module commands to set up the initial shell environment.

Viewing Available Modules

The command

$ module avail

displays a list of all the modules available. The list will look something like:

--- some stuff deleted ---
velvet/1.2.10/INTEL-14.0.2
vmatch/2.2.2

---------------- /usr/local/packages/Modules/modulefiles/admin -----------------
EasyBuild/1.11.1       GCC/4.9.0              INTEL-140-MPICH/3.1.1
EasyBuild/1.13.0       INTEL/14.0.2           INTEL-140-MVAPICH2/2.0
--- some stuff deleted ---

The module names take the form appname/version/compiler, providing the application name, the version, and information about how it was compiled (if needed).

Managing Modules

Besides avail, there are other basic module commands to use for manipulating the environment. These include:

add/load mod1 mod2 ... modn . . . Add modules
rm/unload mod1 mod2 ... modn  . . Remove modules
switch/swap mod . . . . . . . . . Switch or swap one module for another
display/show  . . . . . . . . . . List modules loaded in the environment
avail . . . . . . . . . . . . . . List available module names
whatis mod1 mod2 ... modn . . . . Describe listed modules

The -h option to module will list all available commands.

If a software package you would like to use for your research is not available on a cluster, you can request it to be installed. The software requests are evaluated by the HPC staff on a case-by-case basis. Before you send in a software request, please go through the information below.

h3

Types of request

Depending on how many users need to use the software, software requests are divided into three types, each of which corresponds to the location where the software is installed:

• The user's home directory
• Software packages installed here will be accessible only to the user.
• It is suitable for software packages that will be used by a single user.
• Python, Perl and R modules should be installed here.
• /project
• Software packages installed in /project can be accessed by a group of users.
• It is suitable for software packages that
  • Need to be shared by users from the same research group, or
  • are bigger than the quota on the home file syste.
• This type of request must be sent by the PI of the research group, who may be asked to apply for a storage allocation.
• /usr/local/packages
• Software packages installed under /usr/local/packages can be accessed by all users.
• It is suitable for software packages that will be used by users from multiple research groups.
• This type of request must be sent by the PI of a research group.

h3

How to request

Please send an email to sys-help@loni.org with the following information:

• Your user name
• The name of cluster where you want to use the requested software
• The name, version and download link of the software
• Specific installation instructions if any (e.g. compiler flags, variants and flavor, etc.)
• Why the software is needed
• Where the software should be installed (locally, /project, or /usr/local/packages) and justification explaining how many users are expected.

Please note that, once the software is installed, testing and validation are users' responsibility.

INSTRUCTIONS FOR QUANTUM ESPRESSO TUTORIAL, LSU DFT WORKSHOP

Instructions prepared by Shobhana Narasimhan, with the help of Alex Pacheco.

Note that the following color code has been used in this instruction sheet:

Broad headings are in red.

File names are in magenta.

Phrases to be typed into the command line are in blue.

Input parameters are in dark green.

A very brief reminder of useful linux commands:

• cp A B copies file A to file B
• mv A B moves file A to file B
• mkdir C creates a directory named C
• ls C lists the names of the files and sub-directories contained in directory C
• vi A or gedit A are some of the many ways in which you can open the file A for reading and/or editing.
• Adding an ampersand & at the end of your command will make it run in the background, so that you can continue to issue other commands while the original one is still running.

This exercise is meant as an introduction to how to perform simple scf calculations using the pw package of Quantum ESPRESSO. You will perform calculations on Si (an insulator – well, actually a semiconductor) and Al (a metal).

First, you will first perform simple scf (self-consistent field) calculations on silicon.

• First switch over to an interactive process on the machine eric:
• qsub -X -I -l walltime=3:00:00,nodes=1:ppn=4 -q priority -A loni_train2011
• Find out which processor you have been assigned:
• echo $HOSTNAME > $HOME/hostfile

(Note: the above two steps are particular to running interactive jobs on the eric cluster at LSU, if you are running, for example on a PC back home, you can skip steps like these.)

• Go to the directory /home/etrain00/qetutorial
• You will see the following files:
• qetut_instructions.html this file!
• Si.sample.in this is a sample input file, for a primitive Si cell containing 2 atoms.
• Si.sample.sh this is a sample shell script that you can use (optionally) to run a series of calculations on Si.
• Si.pbe-rrkj.UPF this is a pseudopotential file for Si.
• Al.sample.in this is a sample input file, for a primitive Al cell.
• Al.sample.sh this is a sample shell script that you can use (optionally) to run a series of calculations on Al.
• Al.pz-vbc.UPF this is pseudopotential file for Al.

(Note that usually when you have downloaded and compiled QE your pseudopotentials will be in a separate directory called pseudo; but for simplicity, for this exercise I have put the pseudopotentials in the same working directory).

• Create the qetutorial directory
• mkdir -p /work/$USER/qetutorial
• Copy all the files that you see here into /work/$USER/qetutorial on the machine you are working on.
• cp /home/etrain00/qetutorial/* /work/$USER/qetutorial
• Change to this directory and from now on work only in /work/$USER/qetutorial!!!
• cd /work/$USER/qetutorial
• Make a temporary subdirectory tmp in which you will store large files (wavefunctions, etc:)
• mkdir tmp

• Open and read the sample file Si.sample.in
• For each calculation that you will run, you can use Si.sample.in as a template that you can copy to another file and then edit. Alternatively, you can use shell scripts to automate this process.
• Note that we have specified that large files will be written in the subdirectory tmp
• Note also that we have specified that the pseudopotential is in the same working directory (usually it is another directory whose name is supplied).
• Note that in the &controlnamelist, we have specified that we want to run an scf calculation.
• Silicon has the diamond structure. Note that we are using a (primitive) unit cell that corresponds to an fcc lattice, with 2 Si atoms in the atomic basis. Notice the values given for ibrav, nat, ntyp and ATOMIC_POSITIONS.
• In this file, the lattice constant celldm(1) has been set equal to 10.26 bohr = 5.43 Å, which is the experimental value.
• A 6×6×6 shifted Monkhorst-Pack k-point mesh has been used.
• The plane-wave cut-off for wavefunctions, ecutwfc has been set to 20 Ry. Since we are not using an ultrasoft pseudopotential, we are leaving ecutrho at the default value.

• Use Xcrysden to view the structure in the sample input file:
• xcrysden --pwi Si.sample.in
• Play around with xcrysden! Rotate the view, display the boundaries of the conventional cubic cell, etc.
• You can also use xcrysden to help answer questions like:
• How many atoms are contained in the conventional cubic cell?
• What is the coordination number of each atom?
• What is the nearest-neighbor distance?

• Run an scf calculation:
• The way to do this is:
• export espresso=/usr/local/packages/espresso/4.3.1/intel-11.1-mvapich-1.1
• mpirun -np 1 -machinefile $HOME/hostfile $espresso/bin/pw.x < input_filename > output_filename
• (The syntax of the above two lines is particular to running on eric. When you run a simple one-processor interactive job in general, you often just have to type $espresso/bin/pw.x < input_filename > output_filename , where $espresso is the directory where you have installed Quantum Espresso)
• So in this particular case, you will type:
• mpirun -np 1 -machinefile $HOME/hostfile $espresso/bin/pw.x < Si.sample.in > Si.sample.out

• Read the output, and answer, for example, the following questions:
• Was symmetry used to reduce the number of k-points from 6×6×6=216?
• How many scf iterations were performed before convergence was achieved?
• What is the final (converged) value for the total energy? (Note: you can find this by looking for an exclamation mark in the output file). For example, you can type:
• grep ! Si.sample.out
• Note: If you set verbosity = 'high' in the &control namelist, you will get additional information in your output file, e.g., about symmetries and occupation numbers.

• Now do convergence tests with respect to plane-wave cut-off:
• Find out how the total energy changes when you vary the plane wave cut-off ecutwfc between 10 and 30 Ry, in steps of 5 Ry. You can either do this by copying the sample file Si.sample.in to other input files and editing them, and running pw.x using each of these input files in turn, or by running the sample shell script Si.sample.sh
• Before running this shell script, do edit it to supply the information about the directory in which you have installed Quantum Espresso. Then:
• ./Si.sample.sh
• Now make a data file that contains the value of ecutwfc in the first column, and total energy in the second column. Remember that you can easily find the values of the total energy by searching for an exclamation mark, e.g., using the grep command
• Plot a graph of this data using your favorite plotting program (e.g., xmgrace or gnuplot).
• Does the total energy decrease monotonically as ecutwfc is increased?
• Decide what value of ecutwfc you are "happy" with. Fix the value of at this value of ecutwfc and now check convergence wrt BZ sampling....

• Now examine convergence with respect to Brillouin zone sampling:
• Fix the value of ecutwfc at whatever value you have decided is good enough. (Remember: differences in energy will converge at lower values of ecutwfc than the absolute value of the total energy!) I chose ecutwfc to be 20 Ry, but you can make whatever choice you want!
• Let the lattice constant celldm(1) emain at the experimental value.
• Vary the Monkhorst-Pack grid parameters nk1, nk2 and nk3.
• You can do this either by manually editing input files, or by modifying the script Si.sample.sh
• Use nk1=nk2=nk3= 1, 2, 3, 4, 5, 6.
• Leave the offset (shift) at k1=k2=k3 = 1.
• Run pw.x for each of the six input files you have made (either one by one, or using a shellscript)
• Remember: don"t overwrite output files before you extract data from them!
• Assemble a data file that contains how the total energy varies with nk1=nk2=nk3. The first column should contain the value of nk1 (or, if you prefer, the number of k-points generated and used), and the second column should contain the value of the total energy).
• Plot a graph of this data. Do you think that the total energy is converging as you increase the number of k-points?
• Note that the convergence wrt the number of k points need not necessarily be monotonic (but is it, in this case?)

• Now obtain the equilibrium lattice constant of Si:
• Fix values of ecutwfc and nk1=nk2=nk3.
• Make input files in which you vary the lattice constant, i.e., celldm(1) from 9.8 to 11.0 bohr, in steps of 0.2 bohr.
• Run these seven scf calculations.
• Assemble the results for the dependence of total energy on lattice constant in a data file, e.g., one called etotvsalat.dat (2 columns, the first containing the lattice constant in bohr, and the second containing total energy in Ry).
• Plot this data; compare your calculated value for lattice constant (obtained by just eyeballing the data or by doing a rough fit) with experiment.
• To do later today (in the tons of spare time you will undoubtedly have) – or you can do it now if you have the time:
• Fit this data to an equation of state by running
• mpirun -np 1 -machinefile $HOME/hostfile $espresso/bin/ev.x

When you run this, you will be prompted to supply:

• the units in which you have the lattice constants
• the type of Bravais lattice (note: using the name and not the ibrav number)
• which particular equation of state you want to use

(I usually use the 2^nd Order Birch eq., but you can experiment with other choices too.)

• the name of the input file containing the data of lattice constant and total energy
• the name of the output file where you want the results of the fitting procedure
• From the output file you can get the values of the equilibrium lattice constant a0 and bulk modulus k0 as well as information about the quality of fit.

(At this stage, you might get some error messages on the screen, but ignore them, the ev.x program seems to run just fine....)

More optional stuff to do later on:

Repeat the calculations for Si using a conventional cubic cell instead of an fcc primitive cell.

Now switch over from Si to Al

• Open and read the sample file Al.sample.in
• Compared to your input files for Si (exercise above) there are two main differences:
• (i) Now nat = 1 instead of nat= 2, and accordingly there is only one Al atom at 0.0 0.0 0.0
• (ii) There are additional lines, saying that the occupation should be smeared, which type of smearing to use, and what value of degauss (smearing width) to use.
• Note that now we have set verbosity= 'high' in the &control namelist ; so that the code wll print out occupation numbers, etc.
• Use Xcrysden to view the structure in the sample file:
• xcrysden --pwi Al.sample.in &
• Run an scf calculation:
• mpirun -np 1 -machinefile $HOME/hostfile $espresso/bin/pw.x < Al.sample.in > Al.sample.out
• Read the output, and answer the following questions:
• How many bands were computed? How does this compare with the number of electrons?
• What is the Fermi energy?
• Now do convergence tests with respect to plane-wave cut-off:
• In principle, you should now run a series of calcuations with different values of ecutwfc (as you did for Si) and see at what value of ecutwfc your total energy has converged. However, to save time, let us assume that this has been done. Keep working with ecutwfc=12.
• Now do convergence tests with respect to Brillouin zone sampling and smearing width:
• We will simultaneously decide what k-point mesh (values of nk1,nk2, nk3) and the value of the smearing width (degauss) to use. For simplicity, let us assume now that we will only use the Marzari-Vanderbilt smearing.
• Loop over values of nk1,nk2, nk3 = 6 6 6, 8 8 8, 12 12 12, 16 16 16
• For each of these k-point meshes, loop over degauss ranging from 0.02 to 0.10 (at intervals chosen by you).
• Make input files of the form Al.degauss.*_kdiv.*.in (inserting the appropriate values of degauss and nk). In order to do this, you can either edit your input files manually, or use the shell script Al.sample.sh (If doing the latter, read the shell script carefully and see whether you need to insert additional values or lines).
• Run pw.x for each value of nk and degauss, either using the shell script :
• ./Al.sample.sh
• or a command of the form
• mpirun -np 1 -machinefile $HOME/hostfile $espresso/bin/pw.x < Al.degauss.<val>_kdiv.<val>.in > Al_degauss.<val>_kdiv.<val>.out

(of course, you should substitute the appropriate values).

• Plot a graph showing how the total energy varies with the number of k-points and the smearing width, using a plotting program of your choice.
• To use xmgrace, source /work/jalupo/grace/env.rc

The "true" converged result is obtained in the limit of zero smearing and an infinite number of k-points.

Based upon this, decide what values of nk1=nk2=nk3 and degauss you consider satisfactory.

(Note: you will have to achieve some sort of compromise between accuracy and computational time!)

• Stuff for you to do later on (or you can do it now if you have the time):
• Obtain the equilibrium lattice constant of Al:

Proceed as you did for Si, look for a lattice constant that lies in the range of 7.0 to 8.0 bohr.

• Do non-self-consistent calculations, and play with various post-processing tools
• Examples of stuff you can do: plot charge densities, band structures, densities of states.
• To try this stuff you can either work through some of the examples that come with the QE distribution, or, for example, work through week1/day1/exercise2 on http://www.pwscf.org/wiki/index.php/QESB09