Skip to main content

globus

About

GLOBUS is a wide-area distributed computing system.

Versions and Availability

Softenv Keys for globus on painter
Machine Version Softenv Key
▶ Softenv FAQ?

The information here is applicable to LSU HPC and LONI systems.

Shells

A user may choose between using /bin/bash and /bin/tcsh. Details about each shell follows.

/bin/bash

System resource file: /etc/profile

When one access the shell, the following user files are read in if they exist (in order):

  1. ~/.bash_profile (anything sent to STDOUT or STDERR will cause things like rsync to break)
  2. ~/.bashrc (interactive login only)
  3. ~/.profile

When a user logs out of an interactive session, the file ~/.bash_logout is executed if it exists.

The default value of the environmental variable, PATH, is set automatically using SoftEnv. See below for more information.

/bin/tcsh

The file ~/.cshrc is used to customize the user's environment if his login shell is /bin/tcsh.

Softenv

SoftEnv is a utility that is supposed to help users manage complex user environments with potentially conflicting application versions and libraries.

System Default Path

When a user logs in, the system /etc/profile or /etc/csh.cshrc (depending on login shell, and mirrored from csm:/cfmroot/etc/profile) calls /usr/local/packages/softenv-1.6.2/bin/use.softenv.sh to set up the default path via the SoftEnv database.

SoftEnv looks for a user's ~/.soft file and updates the variables and paths accordingly.

Viewing Available Packages

The command softenv will provide a list of available packages. The listing will look something like:

$ softenv
These are the macros available:
*   @default
These are the keywords explicitly available:
+amber-8                       Applications: 'Amber', version: 8 Amber is a
+apache-ant-1.6.5              Ant, Java based XML make system version: 1.6.
+charm-5.9                     Applications: 'Charm++', version: 5.9 Charm++
+default                       this is the default environment...nukes /etc/
+essl-4.2                      Libraries: 'ESSL', version: 4.2 ESSL is a sta
+gaussian-03                   Applications: 'Gaussian', version: 03 Gaussia
... some stuff deleted ...
Managing SoftEnv

The file ~/.soft in the user's home directory is where the different packages are managed. Add the +keyword into your .soft file. For instance, ff one wants to add the Amber Molecular Dynamics package into their environment, the end of the .soft file should look like this:

+amber-8

@default

To update the environment after modifying this file, one simply uses the resoft command:

% resoft

The command soft can be used to manipulate the environment from the command line. It takes the form:

$ soft add/delete +keyword

Using this method of adding or removing keywords requires the user to pay attention to possible order dependencies. That is, best results require the user to remove keywords in the reverse order in which they were added. It is handy to test out individual keys, but can lead to trouble if changing multiple keys. Changing the .soft file and issuing the resoft is the recommended way of dealing with multiple changes.

Usage

▶ Globus Tutorial?
== What do I need? ==

* Access to LONI Account.

* Active LONI Grid Certificate associated with your account.

* Active LONI Allocation for submitting and running compute
  jobs. (optional during testing and development of workflows)

== Where are the Globus services on LONI? ==

They are only available on 1 (i.e. qb1, eric1, etc)

== Do I need to setup Globus Environment on LONI? ==

Your Globus environment on LONI machines is already setup by
default. 

== Why Do I need a Globus Certificate? ==

Every user and service on the Grid is identified via a certificate,
which contains information vital to identifying and authenticating the
user or service.

A GSI certificate includes four primary pieces of information:

* A subject name, which identifies the person or object that the
  certificate represents.

* The public key belonging to the subject.

* The identity of a Certificate Authority (CA) that has signed the
  certificate to certify that the public key and the identity both
  belong to the subject.  * The digital signature of the named CA.

GSI certificates are encoded in the X.509 certificate format, a
standard data format for certificates established by the Internet
Engineering Task Force (IETF).

== How do I submit a Job using Globus? ==

Lets first visit the job submission command 'globus-job-run' and know
how it is run:

  $ globus-job-run -help

Read over the different options it presents for our target usage
later.

Run a simple command to be executed on the target grid computing
service node:

 $ globus-job-run louie1.loni.org /bin/date
 Mon Mar  2 21:32:56 CST 2009

Now, lets create a script using our favorite editor with the following
contents:

 $ cat random-generator.sh 
 #! /bin/sh
 NLINES=$1
 NDIGITS=$2
 awk "BEGIN {
   for (i = 0 ; i < $NLINES; i++ ) {
         print int(rand() * 10^$NDIGITS )
           }
         }"

first lets change the script so that it can be used as an executable
using the command:

  $ chmod u+x random-generator.sh 

now lets test the script using:

 $ ./random-generator.sh 4 7
 5138706
 1757255
 3086336
 5345316

Now, we are ready to submit this as our executable to be run remotely
on a target grid computing service node.

 $ globus-job-run louie1.loni.org -s random-generator.sh 4 7
 2377875
 2910657
 8458138
 1522082

the -s switch is used to stage in the executable. Note the arguments
at the end of the command.

Now, lets try staging in the input file for an executable like 'cat'
be used to print out the contents.


 $ globus-job-run louie1.loni.org -stdin -s /etc/hosts -l /bin/cat
 127.0.0.1       loopback        localhost       # loopback (lo0) name/address
 
 ############################################################
 #
 # Begin LONI Infrastructure
 #

Here we are staging in the input file '/etc/hosts' from the submit
machine to the target machine where the local executable '/bin/cat' is
used to execute upon.

Now, if we want to get the output written into a file and brought back
to us, we can add some more options to the above command.

 $ globus-job-run louie1.loni.org -stdout -s hellohosts -stdin \
      -s /etc/hosts -l /bin/cat 
 $ ls hellohosts 
 hellohosts
 $ rm hellohosts 
 $ globus-job-run louie1.loni.org -stdout -l hellohosts -stdin \
      -s /etc/hosts -l /bin/cat 
 
using the '-stdout' we can obtain the output in the form of a written
file. We can further specify whether you want to leave the output file
at the submission site or at the target compute site.

== Can I submit Parallel Jobs too? ==

 $ globus-job-run qb1.loni.org/jobmanager-pbs -np 8 -m 10 \
      -p loni_cybertools -q checkpt -stdout \
      -s helloparallel /bin/hostname

In the above command, we are running a serial executable in parallel
over Queenbee. We are requesting for 8 processors for a wall clock
time of 10 minutes using the allocation of 'loni_cybertools' to be run
in the 'checkpt' queue. We are specifying options to dump the output
in the file to be named as 'helloparallel'.

 $ cat helloworld.c 
 #include 
 #include 
 int main(int argc, char *argv[])
 {
         int id, nprocs;
         MPI_Init(&argc, &argv);
         MPI_Comm_rank(MPI_COMM_WORLD, &id);
         MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
         printf("Hello world from %d out of %d\n", id, nprocs);
         MPI_Finalize();
 }

On compiling the parallel executable:

 $ mpcc helloworld.c -o helloworld

Submitting the parallel job from the command line:

 $ globus-job-run ducky.loni.org/jobmanager-loadleveler \
    -env "GBLL_NETWORK_MPI=sn_all,not_shared,US" \
    -x "&(jobType=mpi)(count=8)(maxWallTime=10)(queue=checkpt)\
    (stdout=/work/default/sirish/hello.out)\
    (stderr=/work/default/sirish/hello.err)(hostCount=1)" \
    -l /work/default/sirish/hello

On submitting the parallel job asynchronously:

 $ globus-job-submit ducky.loni.org/jobmanager-loadleveler \
    -env "GBLL_NETWORK_MPI=sn_all,not_shared,US" \
    -x "&(jobType=mpi)(count=8)(maxWallTime=10)(queue=checkpt)\
    (stdout=/work/default/sirish/hello.out)\
    (stderr=/work/default/sirish/hello.err) (hostCount=1)" \
    -l /work/default/sirish/hello \
    https://l2f1g03.sys.loni.org:50004/66178/1272437929/

On checking for the job status later:

 $ globus-job-status https://l2f1g03.sys.loni.org:50004/66178/1272437929/
 DONE

== How do I move files using Globus? ==

Globus provides ''globus-url-copy'' which is a basic command to do
file transfer from source to destinations by urls.
 
$ globus-url-copy \
 -vb gsiftp://portal1.loni.org/home/sirish/portal/gridftp-tutorial/100mb.file \
 gsiftp://qb1.loni.org/home/sirish/100mb.file
 Source: gsiftp://portal1.loni.org/home/sirish/portal/gridftp-tutorial/
 Dest:   gsiftp://qb1.loni.org/home/sirish/
   100mb.file
     104857600 bytes        52.63 MB/sec avg        52.63 MB/sec inst

In the above command, we are transferring a file from one machine to
another while using the switch ''-vb'' that will show us more
output. For complete list of available options:

  $ globus-url-copy -help

Lets say  we test  the transfers using  files of different  sizes, say
100mb and 1gb each.

Creating each of the files:

100mb file:

 $ /opt/freeware/bin/dd count=100 bs=1024k if=/dev/zero \
   of=/work/nfs101/sirish/100mbfile
 100+0 records in
 100+0 records out

1gb file:

 $ /opt/freeware/bin/dd count=100 bs=1024k if=/dev/zero \
   of=/work/nfs101/sirish/1gbfile
 100+0 records in
 100+0 records out

 $ ls -lh *
 -rw-r--r--    1 sirish   sys          100M 2010-04-20 09:40 100mbfile
 -rw-r--r--    1 sirish   sys          100M 2010-04-20 09:40 1gbfile

Transferring the files on command line:

 $ globus-url-copy -vb file:///work/default/sirish/1gbfile \
   gsiftp://ducky.loni.org/work/default/sirish/1gbfile
 Source: file:///work/default/sirish/
 Dest:   gsiftp://ducky.loni.org/work/default/sirish/
   1gbfile
     101711872 bytes        13.22 MB/sec avg        16.00 MB/sec inst

Improving performance using parallel streams:

 $ globus-url-copy -p 4 -vb file:///work/default/sirish/1gbfile \
     gsiftp://ducky.loni.org/work/default/sirish/1gbfile
 Source: file:///work/default/sirish/
 Dest:   gsiftp://ducky.loni.org/work/default/sirish/
   1gbfile
     104857600 bytes        24.39 MB/sec avg        24.39 MB/sec inst

You can even batch transfers together. Using a file containing the
list of source and destination files. For example:

On aix machines:

 $ cat Xfertest.urls.1gb.aix 
 gsiftp://bluedawg.loni.org/work/default/sirish/1gbfile \
    gsiftp://ducky.loni.org/work/default/sirish/1gbfile
 gsiftp://ducky.loni.org/work/default/sirish/1gbfile \
    gsiftp://zeke.loni.org/work/default/sirish/1gbfile
 gsiftp://zeke.loni.org/work/default/sirish/1gbfile \
    gsiftp://neptune.loni.org/work/default/sirish/1gbfile
 gsiftp://neptune.loni.org/work/default/sirish/1gbfile \
    gsiftp://lacumba.loni.org/work/default/sirish/1gbfile

On linux machines:
 
 $ cat Xfertest.urls.1gb.linux 
 gsiftp://qb1.loni.org/work/sirish/1gbfile \
    gsiftp://eric1.loni.org/work/sirish/1gbfile
 gsiftp://eric1.loni.org/work/sirish/1gbfile \
    gsiftp://louie1.loni.org/work/sirish/1gbfile
 gsiftp://louie1.loni.org/work/sirish/1gbfile \
    gsiftp://oliver1.loni.org/work/sirish/1gbfile
 gsiftp://oliver1.loni.org/work/sirish/1gbfile \
    gsiftp://poseidon1.loni.org/work/sirish/1gbfile
 gsiftp://oliver1.loni.org/work/sirish/1gbfile \
    gsiftp://painter1.loni.org/work/sirish/1gbfile

On using without parallel streams:

 $ globus-url-copy -vb -f Xfertest.urls.1gb
 Source: gsiftp://bluedawg.loni.org/work/default/sirish/
 Dest:   gsiftp://ducky.loni.org/work/default/sirish/
   1gbfile
     104857600 bytes        18.87 MB/sec avg        10.83 MB/sec inst
 Source: gsiftp://ducky.loni.org/work/default/sirish/
 Dest:   gsiftp://zeke.loni.org/work/default/sirish/
   1gbfile
     104857600 bytes        30.30 MB/sec avg        30.30 MB/sec inst
 Source: gsiftp://zeke.loni.org/work/default/sirish/
 Dest:   gsiftp://neptune.loni.org/work/default/sirish/
   1gbfile
     104857600 bytes        22.73 MB/sec avg        22.73 MB/sec inst
 Source: gsiftp://neptune.loni.org/work/default/sirish/
 Dest:   gsiftp://lacumba.loni.org/work/default/sirish/
   1gbfile
     104857600 bytes         6.76 MB/sec avg         6.77 MB/sec inst

On using the parallel streams:

 $ globus-url-copy -p 4 -vb -f Xfertest.urls.1gb
 Source: gsiftp://bluedawg.loni.org/work/default/sirish/
 Dest:   gsiftp://ducky.loni.org/work/default/sirish/
   1gbfile
     104857600 bytes        27.03 MB/sec avg        27.03 MB/sec inst
 Source: gsiftp://ducky.loni.org/work/default/sirish/
 Dest:   gsiftp://zeke.loni.org/work/default/sirish/
   1gbfile
     104857600 bytes        47.62 MB/sec avg        47.62 MB/sec inst
 Source: gsiftp://zeke.loni.org/work/default/sirish/
 Dest:   gsiftp://neptune.loni.org/work/default/sirish/
   1gbfile
     104857600 bytes        40.00 MB/sec avg        40.00 MB/sec inst
 Source: gsiftp://neptune.loni.org/work/default/sirish/
 Dest:   gsiftp://lacumba.loni.org/work/default/sirish/
   1gbfile
     104857600 bytes        62.50 MB/sec avg        62.50 MB/sec inst

See the performance improvement!!

===How do I use globus online to transfer files===

Globus Online(GO) manages file transfer for you, monitoring
performance, retrying failures, auto-tuning and recovering from faults
automatically where possible, and reporting status. And there’s no
custom infrastructure or software to install, so you don’t have to be
an IT wizard to use Globus Online.

As a user you need to follow the following steps to use globus online
to transfer files from LONI clusters

# Obtain a LONI Grid Certificate

# Sign Up for a Globus Online Account

# Install GridFTP or Globus on your server.
  
  ** LONI clusters already have Globus tools installed, so you can
     skip this step for transferring files between LONI clusters or
     between LONI and other resources which have globus installed

  ** To install GridFTP on your local machine, globus online provide a
     software [https://www.globusonline.org/globus_connect Globus
     Connect] which will install gridftp and create a local endpoint
     for use with GO.

# Login to the head node where globus is installed e.g. qb1.loni.org,
  and create and store your grid credentials.

# Login to your GO account

# Click on "Start Transfer". You should see two windows side by side

# Select you endpoint for the machines that you want to transfer files
  between e.g. loni#qb on one window and your local machine on another

# Enter the password/passphrase requested # Select files/directories
  that you want to transfer and hit the transfer button (the
  triangle).

# You will receive an email when the transfer is complete.

To learn more about GO, look through the documentation on the GO
website.

▶ LONI Certificates?

The LONI Certification Authority

LONI CA Certificates for the Users/Other Certification Authorities (CAs)

LONI Certification Authority (CA) provides the CA public certificates intended for Browser and Grid related use by the users. For example:

  • Users can have their LONI certificates trusted by other Certification Authorities(CAs).
  • Users can import LONI CA public certificate into their browser to validate the server certificates of the LONI websites to ensure security.
  • Other Certification Authorities can use the LONI CA revocation list to maintain an updated version of active LONI CA issued certificates

LONI CA (Root) Certificate

Following are the different formatted versions of LONI CA (Root) Certificate:
  • Privacy Enhanced Mail (pem)
  • Distinguished Encoding Rules (der)
  • Browser Importable (crt)
  • Hashed File format

Usage Scenarios

Importing LONI CA certificates into your Browser

For Internet Explorer

  • Click on link.
  • Click Open.
  • Click "Install Certificate..."
  • Click Next twice.

For Firefox:

  • Click on the browser importable format link for the LONI CA public/root certificate.
  • Check the boxes depending on your needs.
  • Click Ok.
Importing LONI CA Revocation list into your Browser

For Internet Explorer:

  • Click on link.
  • Click Open.
  • Click "Install Certificate..."
  • Click Next twice.

For Firefox:

  • Click on the browser importable format link for the LONI CA published revocation list.
  • You will see a message that LONI CA certificate revocation list has been successfully imported. You will be asked whether to enable automatic updates. Click Yes.
  • Check the enable automatic updates option for the certificate revocation list. Add the Url of the certificate revocation list and click ok.

Getting your LONI Grid Certificate mapped onto machines with other CAs

You have to supply your Distinguished Name (DN) of the globus certificate to the CAs of the machines where you want your certificate to be mapped. Provide this DN to the CA of the machine. In case the CA of the machine does not already trust LONI CA, you will have to provide the CA with LONI CA (root) certificate in the Hashed file format and the CA signing policy file in addition to your DN.

Adding LONI CA to your list of Trusted Certification Authorities

This means you want to add LONI CA to be trusted by your Globus installation. After configuring the Globus environment on your machine, add LONI CA Root certificate file in hashed file format and LONI CA Signing Policy file to the trusted certificates directory.

On an uncustomized Globus installation, this trusted certificates directory is usually /etc/grid-security/certificates.

Setting up LONI CA Published Certificate Revocation list in your Globus Installation

After adding LONI CA to your list of trusted certificates directory or if you have already done so, place the LONI CA revocation list in hashed file format along with LONI CA Root certificate. For this:

  • Download the file at the Url for the hashed file format.
  • Rename it as named in the Url.
  • move it into globus trusted certificates directory.

On an uncustomized globus installation, this trusted certificates directory is usually /etc/grid-security/certificates.

▶ Request LONI Cert?
== Login to one of the Grid Service Nodes ==

Login to one of the #1 headnodes on a LONI Cluster.

== Generating Globus Certificate Request ==

=== If this is your first request for a LONI Grid Certificate: ===

Then run the command:

 $ grid-cert-request
 A certificate request and private key is being created.
 You will be asked to enter a PEM pass phrase.
 This pass phrase is akin to your account password,
 and is used to protect your key file.
 If you forget your pass phrase, you will need to
 obtain a new certificate.
 
 Generating a 1024 bit RSA private key
 ..................................++++++
 ............++++++
 writing new private key to '/home/sirish/.globus/userkey.pem'
 Enter PEM pass phrase:

As the text above says, you need to choose a pass phrase for
activating the proxy from your certificate. This pass phrase is
used to secure your private key, so DO NOT leave it blank. Also, DO
NOT use your account password as the pass phrase.

Note: Your signed grid certificate is valid for six months. It is the
responsibility of the user to store/manage the pass phrase, and we
trust our users to take that responsibility seriously. Once a signed
certificate is issued to a user for the first time, no new certificate
will be issued to the same user within three months. If the new pass
phrase is ever lost again, the user must wait until the current one
expires. If a user renews an expired certificate and forgets the pass
phrase, then he or she must wait until the expiration as well. For
example, if a grid certificate is signed and issued to a user for the
first time on January 1, the earliest date for the user to receive a
new certificate is April 1. If the pass phrase is lost again within
six months, then the user must wait until October 1. So please choose
a secure pass phrase which is easy for you to remember or store it at
a safe place.
 
=== If this is not your first request for a LONI Grid Certificate: ===

Sometimes the users want to renew an expired certificate or request
for a new certificate. In such case use the -force switch in the
command:

The -force switch is required on the machine where you already have an
existing .globus folder in your home directory.

 $ grid-cert-request -force

 $ $GLOBUS_LOCATION/bin/grid-cert-request -force

=== What is happening? ===

Once grid-cert-request is issued, the directory .globus will be
created in your home directory with three files: usercert_request.pem,
userkey.pem, and an empty usercert.pem.

==Sending LONI CA your certificate request==

You need to email the usercert_request.pem file to ca@loni.org to
aquire a user certificate signed by the CA.

DO NOT email ca@loni.org for anything else other than sending the
generated certificate request.

At this point a LONI administrator will take over, issue the
certificate, and normally, this is sent it back to the user by email.

==Installing your certificate==

After receiving the signed certificate, you need to copy it
to ~/.globus/usercert.pem. Then, run the command:

 $ grid-cert-info -s

[[Category:Grid Computing]]
[[Category:Globus]]
[[Category:Globus Certificates]]

To check your grid certificate information. 
▶ Manage LONI Cert?

We assume that you already have an active globus certificate with the
LONI CA. If you do not have the LONI issued globus certificate yet,
please refer to Obtaining a LONI Grid Certificate to obtain one.

Once you have a valid LONI grid certificate, using the myproxy server
setup at proxy01.hpc.lsu.edu, you can store the globus certificate
credentials to be made available for your applications.

First, we need to make sure you have myproxy tools available in your
path. If not already, login to a number 1 headnode for an already set
up globus environment.

Second, make sure that you have globus certificates directory on the
machine from which you are trying to store the credentials into
proxy01.hpc.lsu.edu.

 $ ls .globus/
 job                   usercert_request.pem
 usercert.pem          userkey.pem

Optionally, you can verify if the certificate is valid or not using

 $ grid-proxy-init 
 Your identity: /C=US/O=Louisiana Optical Network \
    Initiative/OU=loni.org/OU=sys.loni.org/CN=Sirish Tummala
 Enter GRID pass phrase for this identity:
 Creating proxy ...................................................... Done
 Your proxy is valid until: Thu Dec  4 01:33:01 2008

 $ grid-proxy-info 
 subject  : /C=US/O=Louisiana Optical Network \
    Initiative/OU=loni.org/OU=sys.loni.org/CN=Sirish Tummala/CN=1150172559
 issuer   : /C=US/O=Louisiana Optical Network \
    Initiative/OU=loni.org/OU=sys.loni.org/CN=Sirish Tummala
 identity : /C=US/O=Louisiana Optical Network \
    Initiative/OU=loni.org/OU=sys.loni.org/CN=Sirish Tummala
 type     : Proxy draft (pre-RFC) compliant impersonation proxy
 strength : 512 bits
 path     : /tmp/x509up_u26027
 timeleft : 11:59:50

Now that you have setup a valid and working certificate, its time to
store it using myproxy tools in our credential management server at
proxy01.hpc.lsu.edu

 sirish@l3f1n03$ myproxy-init -s proxy01.hpc.lsu.edu
 Your identity: /C=US/O=Louisiana Optical Network \
    Initiative/OU=loni.org/OU=sys.loni.org/CN=Sirish Tummala
 Enter GRID pass phrase for this identity:
 Creating proxy ..................................... Done
 Proxy Verify OK
 Your proxy is valid until: Wed Dec 10 13:35:50 2008
 Enter MyProxy pass phrase:
 Verifying - Enter MyProxy pass phrase:
 A proxy valid for 168 hours (7.0 days) for user sirish now \
    exists on proxy01.hpc.lsu.edu.

The first password you enter during the above command is your LONI
grid certificate password. The second password is the one that 'you
will assign' for your applications to use so that they can retrieve
the proxy from the server.

If you want to verify the retrieval of your proxy from the credential
server, first destroy the generated proxy on the machine using

 grid-proxy-destroy

Then use the following command and enter the second password that you
assigned for the applications to use to retrieve.

 $ myproxy-logon -s proxy01.hpc.lsu.edu
 Enter MyProxy pass phrase:
 A credential has been received for user sirish in /tmp/x509up_u26027.

By default, the proxy will be valid for 7 days on the credential
management server. If you would like to customize this behavior or
know about any more options that you might want to use, it can be
useful to read the immediate help available to the command.

 myproxy-init -help
 myproxy-logon -help

▶ Check LONI Cert Info?
== Certificate Information ==

You can view your grid certificate information using the command:

 $ grid-cert-info

This will display your entire certificate.

=== Distinguished Name (DN) ===

You can find your Distinguished Name (DN/Subject) only using:

 $ grid-cert-info -subject
 /C=US/O=Louisiana Optical Network \
     Initiative/OU=loni.org/OU=sys.loni.org/CN=your name

=== Issuer Hash ===

To find the issuer hash of your certificate:

 $ grid-cert-info -issuerhash
 a3bf9f3c

=== More Details ===

For more interesting details as how to use the above command to obtain
your certificate information:

 $ grid-cert-info -help

== Generating a valid proxy ==

Initializing your certificate into a working proxy can be done using
the command:

 $ grid-proxy-init
 Your identity: /C=US/O=Louisiana Optical Network \
    Initiative/OU=loni.org/OU=sys.loni.org/CN=your name
 Enter GRID pass phrase for this identity:
 Creating proxy .................................... Done
 Your proxy is valid until: Tue Mar  3 09:15:28 2009

Resources

Last modified: November 11 2014 16:48:02.