globus
Table of Content
Versions and Availability
h4
h5
Module Names for globus on smic
Machine | Version | Module Name |
---|---|---|
None Available | N/A | N/A |
▶ Module FAQ?
The information here is applicable to LSU HPC and LONI systems.
h4
Shells
A user may choose between using /bin/bash and /bin/tcsh. Details about each shell follows.
/bin/bash
System resource file: /etc/profile
When one access the shell, the following user files are read in if they exist (in order):
- ~/.bash_profile (anything sent to STDOUT or STDERR will cause things like rsync to break)
- ~/.bashrc (interactive login only)
- ~/.profile
When a user logs out of an interactive session, the file ~/.bash_logout is executed if it exists.
The default value of the environmental variable, PATH, is set automatically using Modules. See below for more information.
/bin/tcsh
The file ~/.cshrc is used to customize the user's environment if his login shell is /bin/tcsh.
Modules
Modules is a utility which helps users manage the complex business of setting up their shell environment in the face of potentially conflicting application versions and libraries.
Default Setup
When a user logs in, the system looks for a file named .modules in their home directory. This file contains module commands to set up the initial shell environment.
Viewing Available Modules
The command
$ module avail
displays a list of all the modules available. The list will look something like:
--- some stuff deleted --- velvet/1.2.10/INTEL-14.0.2 vmatch/2.2.2 ---------------- /usr/local/packages/Modules/modulefiles/admin ----------------- EasyBuild/1.11.1 GCC/4.9.0 INTEL-140-MPICH/3.1.1 EasyBuild/1.13.0 INTEL/14.0.2 INTEL-140-MVAPICH2/2.0 --- some stuff deleted ---
The module names take the form appname/version/compiler, providing the application name, the version, and information about how it was compiled (if needed).
Managing Modules
Besides avail, there are other basic module commands to use for manipulating the environment. These include:
add/load mod1 mod2 ... modn . . . Add modules rm/unload mod1 mod2 ... modn . . Remove modules switch/swap mod . . . . . . . . . Switch or swap one module for another display/show . . . . . . . . . . List modules loaded in the environment avail . . . . . . . . . . . . . . List available module names whatis mod1 mod2 ... modn . . . . Describe listed modules
The -h option to module will list all available commands.
▶ Did not find the version you want to use??
If a software package you would like to use for your research is not available on a cluster, you can request it to be installed. The software requests are evaluated by the HPC staff on a case-by-case basis. Before you send in a software request, please go through the information below.
h3
Types of request
Depending on how many users need to use the software, software requests are divided into three types, each of which corresponds to the location where the software is installed:
- The user's home directory
- Software packages installed here will be accessible only to the user.
- It is suitable for software packages that will be used by a single user.
- Python, Perl and R modules should be installed here.
- /project
- Software packages installed in /project can be accessed by a group of users.
- It is suitable for software packages that
- Need to be shared by users from the same research group, or
- are bigger than the quota on the home file syste.
- This type of request must be sent by the PI of the research group, who may be asked to apply for a storage allocation.
- /usr/local/packages
- Software packages installed under /usr/local/packages can be accessed by all users.
- It is suitable for software packages that will be used by users from multiple research groups.
- This type of request must be sent by the PI of a research group.
h3
How to request
Please send an email to sys-help@loni.org with the following information:
- Your user name
- The name of cluster where you want to use the requested software
- The name, version and download link of the software
- Specific installation instructions if any (e.g. compiler flags, variants and flavor, etc.)
- Why the software is needed
- Where the software should be installed (locally, /project, or /usr/local/packages) and justification explaining how many users are expected.
Please note that, once the software is installed, testing and validation are users' responsibility.
About the Software
Description Path: key[@id="globus"]/whatis Not Found!
Usage
▶ Globus Tutorial?
== What do I need? == * Access to LONI Account. * Active LONI Grid Certificate associated with your account. * Active LONI Allocation for submitting and running compute jobs. (optional during testing and development of workflows) == Where are the Globus services on LONI? == They are only available on1 (i.e. qb1, eric1, etc) == Do I need to setup Globus Environment on LONI? == Your Globus environment on LONI machines is already setup by default. == Why Do I need a Globus Certificate? == Every user and service on the Grid is identified via a certificate, which contains information vital to identifying and authenticating the user or service. A GSI certificate includes four primary pieces of information: * A subject name, which identifies the person or object that the certificate represents. * The public key belonging to the subject. * The identity of a Certificate Authority (CA) that has signed the certificate to certify that the public key and the identity both belong to the subject. * The digital signature of the named CA. GSI certificates are encoded in the X.509 certificate format, a standard data format for certificates established by the Internet Engineering Task Force (IETF). == How do I submit a Job using Globus? == Lets first visit the job submission command 'globus-job-run' and know how it is run: $ globus-job-run -help Read over the different options it presents for our target usage later. Run a simple command to be executed on the target grid computing service node: $ globus-job-run louie1.loni.org /bin/date Mon Mar 2 21:32:56 CST 2009 Now, lets create a script using our favorite editor with the following contents: $ cat random-generator.sh #! /bin/sh NLINES=$1 NDIGITS=$2 awk "BEGIN { for (i = 0 ; i < $NLINES; i++ ) { print int(rand() * 10^$NDIGITS ) } }" first lets change the script so that it can be used as an executable using the command: $ chmod u+x random-generator.sh now lets test the script using: $ ./random-generator.sh 4 7 5138706 1757255 3086336 5345316 Now, we are ready to submit this as our executable to be run remotely on a target grid computing service node. $ globus-job-run louie1.loni.org -s random-generator.sh 4 7 2377875 2910657 8458138 1522082 the -s switch is used to stage in the executable. Note the arguments at the end of the command. Now, lets try staging in the input file for an executable like 'cat' be used to print out the contents. $ globus-job-run louie1.loni.org -stdin -s /etc/hosts -l /bin/cat 127.0.0.1 loopback localhost # loopback (lo0) name/address ############################################################ # # Begin LONI Infrastructure # Here we are staging in the input file '/etc/hosts' from the submit machine to the target machine where the local executable '/bin/cat' is used to execute upon. Now, if we want to get the output written into a file and brought back to us, we can add some more options to the above command. $ globus-job-run louie1.loni.org -stdout -s hellohosts -stdin \ -s /etc/hosts -l /bin/cat $ ls hellohosts hellohosts $ rm hellohosts $ globus-job-run louie1.loni.org -stdout -l hellohosts -stdin \ -s /etc/hosts -l /bin/cat using the '-stdout' we can obtain the output in the form of a written file. We can further specify whether you want to leave the output file at the submission site or at the target compute site. == Can I submit Parallel Jobs too? == $ globus-job-run qb1.loni.org/jobmanager-pbs -np 8 -m 10 \ -p loni_cybertools -q checkpt -stdout \ -s helloparallel /bin/hostname In the above command, we are running a serial executable in parallel over Queenbee. We are requesting for 8 processors for a wall clock time of 10 minutes using the allocation of 'loni_cybertools' to be run in the 'checkpt' queue. We are specifying options to dump the output in the file to be named as 'helloparallel'. $ cat helloworld.c #include #include int main(int argc, char *argv[]) { int id, nprocs; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &id); MPI_Comm_size(MPI_COMM_WORLD, &nprocs); printf("Hello world from %d out of %d\n", id, nprocs); MPI_Finalize(); } On compiling the parallel executable: $ mpcc helloworld.c -o helloworld Submitting the parallel job from the command line: $ globus-job-run ducky.loni.org/jobmanager-loadleveler \ -env "GBLL_NETWORK_MPI=sn_all,not_shared,US" \ -x "&(jobType=mpi)(count=8)(maxWallTime=10)(queue=checkpt)\ (stdout=/work/default/sirish/hello.out)\ (stderr=/work/default/sirish/hello.err)(hostCount=1)" \ -l /work/default/sirish/hello On submitting the parallel job asynchronously: $ globus-job-submit ducky.loni.org/jobmanager-loadleveler \ -env "GBLL_NETWORK_MPI=sn_all,not_shared,US" \ -x "&(jobType=mpi)(count=8)(maxWallTime=10)(queue=checkpt)\ (stdout=/work/default/sirish/hello.out)\ (stderr=/work/default/sirish/hello.err) (hostCount=1)" \ -l /work/default/sirish/hello \ https://l2f1g03.sys.loni.org:50004/66178/1272437929/ On checking for the job status later: $ globus-job-status https://l2f1g03.sys.loni.org:50004/66178/1272437929/ DONE == How do I move files using Globus? == Globus provides ''globus-url-copy'' which is a basic command to do file transfer from source to destinations by urls. $ globus-url-copy \ -vb gsiftp://portal1.loni.org/home/sirish/portal/gridftp-tutorial/100mb.file \ gsiftp://qb1.loni.org/home/sirish/100mb.file Source: gsiftp://portal1.loni.org/home/sirish/portal/gridftp-tutorial/ Dest: gsiftp://qb1.loni.org/home/sirish/ 100mb.file 104857600 bytes 52.63 MB/sec avg 52.63 MB/sec inst In the above command, we are transferring a file from one machine to another while using the switch ''-vb'' that will show us more output. For complete list of available options: $ globus-url-copy -help Lets say we test the transfers using files of different sizes, say 100mb and 1gb each. Creating each of the files: 100mb file: $ /opt/freeware/bin/dd count=100 bs=1024k if=/dev/zero \ of=/work/nfs101/sirish/100mbfile 100+0 records in 100+0 records out 1gb file: $ /opt/freeware/bin/dd count=100 bs=1024k if=/dev/zero \ of=/work/nfs101/sirish/1gbfile 100+0 records in 100+0 records out $ ls -lh * -rw-r--r-- 1 sirish sys 100M 2010-04-20 09:40 100mbfile -rw-r--r-- 1 sirish sys 100M 2010-04-20 09:40 1gbfile Transferring the files on command line: $ globus-url-copy -vb file:///work/default/sirish/1gbfile \ gsiftp://ducky.loni.org/work/default/sirish/1gbfile Source: file:///work/default/sirish/ Dest: gsiftp://ducky.loni.org/work/default/sirish/ 1gbfile 101711872 bytes 13.22 MB/sec avg 16.00 MB/sec inst Improving performance using parallel streams: $ globus-url-copy -p 4 -vb file:///work/default/sirish/1gbfile \ gsiftp://ducky.loni.org/work/default/sirish/1gbfile Source: file:///work/default/sirish/ Dest: gsiftp://ducky.loni.org/work/default/sirish/ 1gbfile 104857600 bytes 24.39 MB/sec avg 24.39 MB/sec inst You can even batch transfers together. Using a file containing the list of source and destination files. For example: On aix machines: $ cat Xfertest.urls.1gb.aix gsiftp://bluedawg.loni.org/work/default/sirish/1gbfile \ gsiftp://ducky.loni.org/work/default/sirish/1gbfile gsiftp://ducky.loni.org/work/default/sirish/1gbfile \ gsiftp://zeke.loni.org/work/default/sirish/1gbfile gsiftp://zeke.loni.org/work/default/sirish/1gbfile \ gsiftp://neptune.loni.org/work/default/sirish/1gbfile gsiftp://neptune.loni.org/work/default/sirish/1gbfile \ gsiftp://lacumba.loni.org/work/default/sirish/1gbfile On linux machines: $ cat Xfertest.urls.1gb.linux gsiftp://qb1.loni.org/work/sirish/1gbfile \ gsiftp://eric1.loni.org/work/sirish/1gbfile gsiftp://eric1.loni.org/work/sirish/1gbfile \ gsiftp://louie1.loni.org/work/sirish/1gbfile gsiftp://louie1.loni.org/work/sirish/1gbfile \ gsiftp://oliver1.loni.org/work/sirish/1gbfile gsiftp://oliver1.loni.org/work/sirish/1gbfile \ gsiftp://poseidon1.loni.org/work/sirish/1gbfile gsiftp://oliver1.loni.org/work/sirish/1gbfile \ gsiftp://painter1.loni.org/work/sirish/1gbfile On using without parallel streams: $ globus-url-copy -vb -f Xfertest.urls.1gb Source: gsiftp://bluedawg.loni.org/work/default/sirish/ Dest: gsiftp://ducky.loni.org/work/default/sirish/ 1gbfile 104857600 bytes 18.87 MB/sec avg 10.83 MB/sec inst Source: gsiftp://ducky.loni.org/work/default/sirish/ Dest: gsiftp://zeke.loni.org/work/default/sirish/ 1gbfile 104857600 bytes 30.30 MB/sec avg 30.30 MB/sec inst Source: gsiftp://zeke.loni.org/work/default/sirish/ Dest: gsiftp://neptune.loni.org/work/default/sirish/ 1gbfile 104857600 bytes 22.73 MB/sec avg 22.73 MB/sec inst Source: gsiftp://neptune.loni.org/work/default/sirish/ Dest: gsiftp://lacumba.loni.org/work/default/sirish/ 1gbfile 104857600 bytes 6.76 MB/sec avg 6.77 MB/sec inst On using the parallel streams: $ globus-url-copy -p 4 -vb -f Xfertest.urls.1gb Source: gsiftp://bluedawg.loni.org/work/default/sirish/ Dest: gsiftp://ducky.loni.org/work/default/sirish/ 1gbfile 104857600 bytes 27.03 MB/sec avg 27.03 MB/sec inst Source: gsiftp://ducky.loni.org/work/default/sirish/ Dest: gsiftp://zeke.loni.org/work/default/sirish/ 1gbfile 104857600 bytes 47.62 MB/sec avg 47.62 MB/sec inst Source: gsiftp://zeke.loni.org/work/default/sirish/ Dest: gsiftp://neptune.loni.org/work/default/sirish/ 1gbfile 104857600 bytes 40.00 MB/sec avg 40.00 MB/sec inst Source: gsiftp://neptune.loni.org/work/default/sirish/ Dest: gsiftp://lacumba.loni.org/work/default/sirish/ 1gbfile 104857600 bytes 62.50 MB/sec avg 62.50 MB/sec inst See the performance improvement!! ===How do I use globus online to transfer files=== Globus Online(GO) manages file transfer for you, monitoring performance, retrying failures, auto-tuning and recovering from faults automatically where possible, and reporting status. And there’s no custom infrastructure or software to install, so you don’t have to be an IT wizard to use Globus Online. As a user you need to follow the following steps to use globus online to transfer files from LONI clusters # Obtain a LONI Grid Certificate # Sign Up for a Globus Online Account # Install GridFTP or Globus on your server. ** LONI clusters already have Globus tools installed, so you can skip this step for transferring files between LONI clusters or between LONI and other resources which have globus installed ** To install GridFTP on your local machine, globus online provide a software [https://www.globusonline.org/globus_connect Globus Connect] which will install gridftp and create a local endpoint for use with GO. # Login to the head node where globus is installed e.g. qb1.loni.org, and create and store your grid credentials. # Login to your GO account # Click on "Start Transfer". You should see two windows side by side # Select you endpoint for the machines that you want to transfer files between e.g. loni#qb on one window and your local machine on another # Enter the password/passphrase requested # Select files/directories that you want to transfer and hit the transfer button (the triangle). # You will receive an email when the transfer is complete. To learn more about GO, look through the documentation on the GO website.
▶ LONI Certificates?
The LONI Certification Authority
LONI CA Certificates for the Users/Other Certification Authorities (CAs)
LONI Certification Authority (CA) provides the CA public certificates intended for Browser and Grid related use by the users. For example:
- Users can have their LONI certificates trusted by other Certification Authorities(CAs).
- Users can import LONI CA public certificate into their browser to validate the server certificates of the LONI websites to ensure security.
- Other Certification Authorities can use the LONI CA revocation list to maintain an updated version of active LONI CA issued certificates
LONI CA (Root) Certificate
Following are the different formatted versions of LONI CA (Root) Certificate:- Privacy Enhanced Mail (pem)
- Distinguished Encoding Rules (der)
- Browser Importable (crt)
- Hashed File format
Usage Scenarios
Importing LONI CA certificates into your Browser
For Internet Explorer
- Click on link.
- Click Open.
- Click "Install Certificate..."
- Click Next twice.
For Firefox:
- Click on the browser importable format link for the LONI CA public/root certificate.
- Check the boxes depending on your needs.
- Click Ok.
Importing LONI CA Revocation list into your Browser
For Internet Explorer:
- Click on link.
- Click Open.
- Click "Install Certificate..."
- Click Next twice.
For Firefox:
- Click on the browser importable format link for the LONI CA published revocation list.
- You will see a message that LONI CA certificate revocation list has been successfully imported. You will be asked whether to enable automatic updates. Click Yes.
- Check the enable automatic updates option for the certificate revocation list. Add the Url of the certificate revocation list and click ok.
Getting your LONI Grid Certificate mapped onto machines with other CAs
You have to supply your Distinguished Name (DN) of the globus certificate to the CAs of the machines where you want your certificate to be mapped. Provide this DN to the CA of the machine. In case the CA of the machine does not already trust LONI CA, you will have to provide the CA with LONI CA (root) certificate in the Hashed file format and the CA signing policy file in addition to your DN.
Adding LONI CA to your list of Trusted Certification Authorities
This means you want to add LONI CA to be trusted by your Globus installation. After configuring the Globus environment on your machine, add LONI CA Root certificate file in hashed file format and LONI CA Signing Policy file to the trusted certificates directory.
On an uncustomized Globus installation, this trusted certificates
directory is usually /etc/grid-security/certificates
.
Setting up LONI CA Published Certificate Revocation list in your Globus Installation
After adding LONI CA to your list of trusted certificates directory or if you have already done so, place the LONI CA revocation list in hashed file format along with LONI CA Root certificate. For this:
- Download the file at the Url for the hashed file format.
- Rename it as named in the Url.
- move it into globus trusted certificates directory.
On an uncustomized globus installation, this trusted certificates
directory is usually /etc/grid-security/certificates
.
▶ Request LONI Cert?
== Login to one of the Grid Service Nodes == Login to one of the #1 headnodes on a LONI Cluster. == Generating Globus Certificate Request == === If this is your first request for a LONI Grid Certificate: === Then run the command: $ grid-cert-request A certificate request and private key is being created. You will be asked to enter a PEM pass phrase. This pass phrase is akin to your account password, and is used to protect your key file. If you forget your pass phrase, you will need to obtain a new certificate. Generating a 1024 bit RSA private key ..................................++++++ ............++++++ writing new private key to '/home/sirish/.globus/userkey.pem' Enter PEM pass phrase: As the text above says, you need to choose a pass phrase for activating the proxy from your certificate. This pass phrase is used to secure your private key, so DO NOT leave it blank. Also, DO NOT use your account password as the pass phrase. Note: Your signed grid certificate is valid for six months. It is the responsibility of the user to store/manage the pass phrase, and we trust our users to take that responsibility seriously. Once a signed certificate is issued to a user for the first time, no new certificate will be issued to the same user within three months. If the new pass phrase is ever lost again, the user must wait until the current one expires. If a user renews an expired certificate and forgets the pass phrase, then he or she must wait until the expiration as well. For example, if a grid certificate is signed and issued to a user for the first time on January 1, the earliest date for the user to receive a new certificate is April 1. If the pass phrase is lost again within six months, then the user must wait until October 1. So please choose a secure pass phrase which is easy for you to remember or store it at a safe place. === If this is not your first request for a LONI Grid Certificate: === Sometimes the users want to renew an expired certificate or request for a new certificate. In such case use the -force switch in the command: The -force switch is required on the machine where you already have an existing .globus folder in your home directory. $ grid-cert-request -force $ $GLOBUS_LOCATION/bin/grid-cert-request -force === What is happening? === Once grid-cert-request is issued, the directory .globus will be created in your home directory with three files: usercert_request.pem, userkey.pem, and an empty usercert.pem. ==Sending LONI CA your certificate request== You need to email the usercert_request.pem file to ca@loni.org to aquire a user certificate signed by the CA. DO NOT email ca@loni.org for anything else other than sending the generated certificate request. At this point a LONI administrator will take over, issue the certificate, and normally, this is sent it back to the user by email. ==Installing your certificate== After receiving the signed certificate, you need to copy it to ~/.globus/usercert.pem. Then, run the command: $ grid-cert-info -s [[Category:Grid Computing]] [[Category:Globus]] [[Category:Globus Certificates]] To check your grid certificate information.
▶ Manage LONI Cert?
We assume that you already have an active globus certificate with the LONI CA. If you do not have the LONI issued globus certificate yet, please refer to Obtaining a LONI Grid Certificate to obtain one. Once you have a valid LONI grid certificate, using the myproxy server setup at proxy01.hpc.lsu.edu, you can store the globus certificate credentials to be made available for your applications. First, we need to make sure you have myproxy tools available in your path. If not already, login to a number 1 headnode for an already set up globus environment. Second, make sure that you have globus certificates directory on the machine from which you are trying to store the credentials into proxy01.hpc.lsu.edu. $ ls .globus/ job usercert_request.pem usercert.pem userkey.pem Optionally, you can verify if the certificate is valid or not using $ grid-proxy-init Your identity: /C=US/O=Louisiana Optical Network \ Initiative/OU=loni.org/OU=sys.loni.org/CN=Sirish Tummala Enter GRID pass phrase for this identity: Creating proxy ...................................................... Done Your proxy is valid until: Thu Dec 4 01:33:01 2008 $ grid-proxy-info subject : /C=US/O=Louisiana Optical Network \ Initiative/OU=loni.org/OU=sys.loni.org/CN=Sirish Tummala/CN=1150172559 issuer : /C=US/O=Louisiana Optical Network \ Initiative/OU=loni.org/OU=sys.loni.org/CN=Sirish Tummala identity : /C=US/O=Louisiana Optical Network \ Initiative/OU=loni.org/OU=sys.loni.org/CN=Sirish Tummala type : Proxy draft (pre-RFC) compliant impersonation proxy strength : 512 bits path : /tmp/x509up_u26027 timeleft : 11:59:50 Now that you have setup a valid and working certificate, its time to store it using myproxy tools in our credential management server at proxy01.hpc.lsu.edu sirish@l3f1n03$ myproxy-init -s proxy01.hpc.lsu.edu Your identity: /C=US/O=Louisiana Optical Network \ Initiative/OU=loni.org/OU=sys.loni.org/CN=Sirish Tummala Enter GRID pass phrase for this identity: Creating proxy ..................................... Done Proxy Verify OK Your proxy is valid until: Wed Dec 10 13:35:50 2008 Enter MyProxy pass phrase: Verifying - Enter MyProxy pass phrase: A proxy valid for 168 hours (7.0 days) for user sirish now \ exists on proxy01.hpc.lsu.edu. The first password you enter during the above command is your LONI grid certificate password. The second password is the one that 'you will assign' for your applications to use so that they can retrieve the proxy from the server. If you want to verify the retrieval of your proxy from the credential server, first destroy the generated proxy on the machine using grid-proxy-destroy Then use the following command and enter the second password that you assigned for the applications to use to retrieve. $ myproxy-logon -s proxy01.hpc.lsu.edu Enter MyProxy pass phrase: A credential has been received for user sirish in /tmp/x509up_u26027. By default, the proxy will be valid for 7 days on the credential management server. If you would like to customize this behavior or know about any more options that you might want to use, it can be useful to read the immediate help available to the command. myproxy-init -help myproxy-logon -help
▶ Check LONI Cert Info?
== Certificate Information == You can view your grid certificate information using the command: $ grid-cert-info This will display your entire certificate. === Distinguished Name (DN) === You can find your Distinguished Name (DN/Subject) only using: $ grid-cert-info -subject /C=US/O=Louisiana Optical Network \ Initiative/OU=loni.org/OU=sys.loni.org/CN=your name === Issuer Hash === To find the issuer hash of your certificate: $ grid-cert-info -issuerhash a3bf9f3c === More Details === For more interesting details as how to use the above command to obtain your certificate information: $ grid-cert-info -help == Generating a valid proxy == Initializing your certificate into a working proxy can be done using the command: $ grid-proxy-init Your identity: /C=US/O=Louisiana Optical Network \ Initiative/OU=loni.org/OU=sys.loni.org/CN=your name Enter GRID pass phrase for this identity: Creating proxy .................................... Done Your proxy is valid until: Tue Mar 3 09:15:28 2009
Resources
Last modified: September 10 2020 11:58:50.