Deprecated: please see new documentation site.


Contents

What do I need?

  • Access to LONI Account.
  • Active LONI Grid Certificate associated with your account.
  • Active LONI Allocation for submitting and running compute jobs. (optional during testing and development of workflows)

Where are the Globus services on LONI?

Deprecated: please see new documentation site.


Cluster Grid computing service nodes
Queenbee qb1.loni.org
Eric eric1.loni.org
Louie louie1.loni.org
Oliver oliver1.loni.org
Painter painter1.loni.org
Poseidon poseidon1.loni.org
Painter painter1.loni.org
Bluedawg bluedawg.loni.org

Grid Computing Service Node is usually the first interactive node of each LONI cluster. The services available on each of these nodes are:

  • Pre-WS Gram (Globus Gatekeeper)
  • WS Gram (Globus Web services container) available only on Queen Bee
  • Gridftp (Globus Gridftp server)
  • GSI-OpenSSH (GsiSSH server)
  • Globus client utilities
  • Myproxy client utilities
  • GSI-OpenSSH client utilities

Do I need to setup Globus Environment on LONI?

Your Globus environment on LONI machines is already setup by default. In case you choose to make changes to the Globus environment already setup, use the Link of Managing your Globus Softenv

Why Do I need a Globus Certificate?

Every user and service on the Grid is identified via a certificate, which contains information vital to identifying and authenticating the user or service.

A GSI certificate includes four primary pieces of information:

  • A subject name, which identifies the person or object that the certificate represents.
  • The public key belonging to the subject.
  • The identity of a Certificate Authority (CA) that has signed the certificate to certify that the public key and the identity both belong to the subject.
  • The digital signature of the named CA.

GSI certificates are encoded in the X.509 certificate format, a standard data format for certificates established by the Internet Engineering Task Force (IETF).

For more information on this, checkout GT 4.0 Security Key Concepts

How do I get a LONI Grid Certificate?

Use the link Requesting a LONI Grid Certificate

What is the information in a Grid Certificate?

Deprecated: please see new documentation site.


Certificate Information

You can view your grid certificate information using the command:

grid-cert-info

The above command will display your entire certificate.

Distinguished Name (DN)

You can find your Distinguished Name (DN/Subject) only using:

[sirish@l2f1n03] [~]
<> grid-cert-info -subject
/C=US/O=Louisiana Optical Network Initiative/OU=loni.org/OU=sys.loni.org/CN=Sirish Tummala

Issuer Hash

To find the issuer hash of your certificate:

[sirish@l2f1n03] [~]
<> grid-cert-info -issuerhash
a3bf9f3c

More Details

For more interesting details as how to use the above command to obtain your certificate information:

grid-cert-info -help

Generating a valid proxy

Initializing your certificate into a working proxy can be done using the command:

[sirish@l2f1n03] [~]
<> grid-proxy-init
Your identity: /C=US/O=Louisiana Optical Network Initiative/OU=loni.org/OU=sys.loni.org/CN=Sirish Tummala
Enter GRID pass phrase for this identity:
Creating proxy .................................... Done
Your proxy is valid until: Tue Mar  3 09:15:28 2009

Reference Documentation

GT4: Security: Pre-WS Authentication & Authorization User's Guide

How do I submit a Job using Globus?

Lets first visit the job submission command 'globus-job-run' and know how it is run:

[sirish@l2f1n03] [~]
<> globus-job-run -help

Read over the different options it presents for our target usage later.

Run a simple command to be executed on the target grid computing service node:

[sirish@l2f1n03] [~]
<> globus-job-run louie1.loni.org /bin/date
Mon Mar  2 21:32:56 CST 2009

Now, lets create a script using our favorite editor with the following contents:

[sirish@l2f1n03] [~]
<> cat random-generator.sh 
#! /bin/sh
NLINES=$1
NDIGITS=$2
awk "BEGIN {
  for (i = 0 ; i < $NLINES; i++ ) {
        print int(rand() * 10^$NDIGITS )
          }
        }"

first lets change the script so that it can be used as an executable using the command:

[sirish@l2f1n03] [~]
<> chmod u+x random-generator.sh 

now lets test the script using:

[sirish@l2f1n03] [~]
<> ./random-generator.sh 4 7
5138706
1757255
3086336
5345316

Now, we are ready to submit this as our executable to be run remotely on a target grid computing service node.

[sirish@l2f1n03] [~]
<> globus-job-run louie1.loni.org -s random-generator.sh 4 7
2377875
2910657
8458138
1522082

the -s switch is used to stage in the executable. Note the arguments at the end of the command.

Now, lets try staging in the input file for an executable like 'cat' be used to print out the contents.

[sirish@l2f1n03] [~]
<> globus-job-run louie1.loni.org -stdin -s /etc/hosts -l /bin/cat
127.0.0.1       loopback        localhost       # loopback (lo0) name/address

############################################################
#
# Begin LONI Infrastructure
#

Here we are staging in the input file '/etc/hosts' from the submit machine to the target machine where the local executable '/bin/cat' is used to execute upon.

Now, if we want to get the output written into a file and brought back to us, we can add some more options to the above command.

[sirish@l2f1n03] [~]
<> globus-job-run louie1.loni.org -stdout -s hellohosts -stdin -s /etc/hosts -l /bin/cat 
[sirish@l2f1n03] [~]
<> ls hellohosts 
hellohosts
[sirish@l2f1n03] [~]
<> rm hellohosts 
[sirish@l2f1n03] [~]
<> globus-job-run louie1.loni.org -stdout -l hellohosts -stdin -s /etc/hosts -l /bin/cat 

using the '-stdout' we can obtain the output in the form of a written file. We can further specify whether you want to leave the output file at the submission site or at the target compute site.

GT 4.0 Pre WS GRAM: User Guide

Can I submit Parallel Jobs too?

globus-job-run qb1.loni.org/jobmanager-pbs -np 8 -m 10 -p loni_cybertools -q checkpt -stdout \
   -s helloparallel /bin/hostname

In the above command, we are running a serial executable in parallel over Queenbee. We are requesting for 8 processors for a wall clock time of 10 minutes using the allocation of 'loni_cybertools' to be run in the 'checkpt' queue. We are specifying options to dump the output in the file to be named as 'helloparallel'.

sirish@l1f1n03$ cat helloworld.c 
#include <mpi.h>
#include <stdio.h>
int main(int argc, char *argv[])
{
        int id, nprocs;
        MPI_Init(&argc, &argv);
        MPI_Comm_rank(MPI_COMM_WORLD, &id);
        MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
        printf("Hello world from %d out of %d\n", id, nprocs);
        MPI_Finalize();
}

On compiling the parallel executable:

sirish@l1f1n03$ mpcc helloworld.c -o helloworld

Submitting the parallel job from the command line:

sirish@l1f1n03$ globus-job-run ducky.loni.org/jobmanager-loadleveler \
   -env "GBLL_NETWORK_MPI=sn_all,not_shared,US" \
   -x "&(jobType=mpi)(count=8)(maxWallTime=10)(queue=checkpt)(stdout=/work/default/sirish/hello.out)\
   (stderr=/work/default/sirish/hello.err)(hostCount=1)" \
   -l /work/default/sirish/hello

On submitting the parallel job asynchronously:

sirish@l1f1n03$ globus-job-submit ducky.loni.org/jobmanager-loadleveler \
   -env "GBLL_NETWORK_MPI=sn_all,not_shared,US" \
   -x "&(jobType=mpi)(count=8)(maxWallTime=10)(queue=checkpt)(stdout=/work/default/sirish/hello.out)\
   (stderr=/work/default/sirish/hello.err) (hostCount=1)" \
   -l /work/default/sirish/hello https://l2f1g03.sys.loni.org:50004/66178/1272437929/

On checking for the job status later:

sirish@l1f1n03$ globus-job-status https://l2f1g03.sys.loni.org:50004/66178/1272437929/
DONE

For MPIg jobs, please refer to Running a Job Using Multiple Clusters with Globus and mpiG.

How do I move files using Globus?

Globus provides globus-url-copy which is a basic command to do file transfer from source to destinations by urls.

[sirish@portal1 gridftp-tutorial]$ globus-url-copy \
   -vb gsiftp://portal1.loni.org/home/sirish/portal/gridftp-tutorial/100mb.file \
   gsiftp://qb1.loni.org/home/sirish/100mb.file
Source: gsiftp://portal1.loni.org/home/sirish/portal/gridftp-tutorial/
Dest:   gsiftp://qb1.loni.org/home/sirish/
  100mb.file
    104857600 bytes        52.63 MB/sec avg        52.63 MB/sec inst

In the above command, we are transferring a file from one machine to another while using the switch -vb that will show us more output. For complete list of available options:

 globus-url-copy -help

Lets say we test the transfers using files of different sizes, say 100mb and 1gb each.

Creating each of the files:

100mb file:

sirish@l1f1n03$  /opt/freeware/bin/dd count=100 bs=1024k if=/dev/zero of=/work/nfs101/sirish/100mbfile
100+0 records in
100+0 records out

1gb file:

sirish@l1f1n03$  /opt/freeware/bin/dd count=100 bs=1024k if=/dev/zero of=/work/nfs101/sirish/1gbfile
100+0 records in
100+0 records out
sirish@l1f1n03$ ls -lh *
-rw-r--r--    1 sirish   sys          100M 2010-04-20 09:40 100mbfile
-rw-r--r--    1 sirish   sys          100M 2010-04-20 09:40 1gbfile

Transferring the files on command line:

sirish@l1f1n03$ globus-url-copy -vb file:///work/default/sirish/1gbfile gsiftp://ducky.loni.org/work/default/sirish/1gbfile
Source: file:///work/default/sirish/
Dest:   gsiftp://ducky.loni.org/work/default/sirish/
  1gbfile
    101711872 bytes        13.22 MB/sec avg        16.00 MB/sec inst

Improving performance using parallel streams:

sirish@l1f1n03$ globus-url-copy -p 4 -vb file:///work/default/sirish/1gbfile \
    gsiftp://ducky.loni.org/work/default/sirish/1gbfile
Source: file:///work/default/sirish/
Dest:   gsiftp://ducky.loni.org/work/default/sirish/
  1gbfile
    104857600 bytes        24.39 MB/sec avg        24.39 MB/sec inst

You can even batch transfers together. Using a file containing the list of source and destination files. For example:

On aix machines:

sirish@l1f1n03$ cat Xfertest.urls.1gb.aix 
gsiftp://bluedawg.loni.org/work/default/sirish/1gbfile gsiftp://ducky.loni.org/work/default/sirish/1gbfile
gsiftp://ducky.loni.org/work/default/sirish/1gbfile gsiftp://zeke.loni.org/work/default/sirish/1gbfile
gsiftp://zeke.loni.org/work/default/sirish/1gbfile gsiftp://neptune.loni.org/work/default/sirish/1gbfile
gsiftp://neptune.loni.org/work/default/sirish/1gbfile gsiftp://lacumba.loni.org/work/default/sirish/1gbfile

On linux machines:

sirish@l1f1n03$ cat Xfertest.urls.1gb.linux 
gsiftp://qb1.loni.org/work/sirish/1gbfile gsiftp://eric1.loni.org/work/sirish/1gbfile
gsiftp://eric1.loni.org/work/sirish/1gbfile gsiftp://louie1.loni.org/work/sirish/1gbfile
gsiftp://louie1.loni.org/work/sirish/1gbfile gsiftp://oliver1.loni.org/work/sirish/1gbfile
gsiftp://oliver1.loni.org/work/sirish/1gbfile gsiftp://poseidon1.loni.org/work/sirish/1gbfile
gsiftp://oliver1.loni.org/work/sirish/1gbfile gsiftp://painter1.loni.org/work/sirish/1gbfile

On using without parallel streams:

sirish@l1f1n03$ globus-url-copy -vb -f Xfertest.urls.1gb
Source: gsiftp://bluedawg.loni.org/work/default/sirish/
Dest:   gsiftp://ducky.loni.org/work/default/sirish/
  1gbfile
    104857600 bytes        18.87 MB/sec avg        10.83 MB/sec inst
Source: gsiftp://ducky.loni.org/work/default/sirish/
Dest:   gsiftp://zeke.loni.org/work/default/sirish/
  1gbfile
    104857600 bytes        30.30 MB/sec avg        30.30 MB/sec inst
Source: gsiftp://zeke.loni.org/work/default/sirish/
Dest:   gsiftp://neptune.loni.org/work/default/sirish/
  1gbfile
    104857600 bytes        22.73 MB/sec avg        22.73 MB/sec inst
Source: gsiftp://neptune.loni.org/work/default/sirish/
Dest:   gsiftp://lacumba.loni.org/work/default/sirish/
  1gbfile
    104857600 bytes         6.76 MB/sec avg         6.77 MB/sec inst

On using the parallel streams:

sirish@l1f1n03$ globus-url-copy -p 4 -vb -f Xfertest.urls.1gb
Source: gsiftp://bluedawg.loni.org/work/default/sirish/
Dest:   gsiftp://ducky.loni.org/work/default/sirish/
  1gbfile
    104857600 bytes        27.03 MB/sec avg        27.03 MB/sec inst
Source: gsiftp://ducky.loni.org/work/default/sirish/
Dest:   gsiftp://zeke.loni.org/work/default/sirish/
  1gbfile
    104857600 bytes        47.62 MB/sec avg        47.62 MB/sec inst
Source: gsiftp://zeke.loni.org/work/default/sirish/
Dest:   gsiftp://neptune.loni.org/work/default/sirish/
  1gbfile
    104857600 bytes        40.00 MB/sec avg        40.00 MB/sec inst
Source: gsiftp://neptune.loni.org/work/default/sirish/
Dest:   gsiftp://lacumba.loni.org/work/default/sirish/
  1gbfile
    104857600 bytes        62.50 MB/sec avg        62.50 MB/sec inst

See the performance improvement!!

How do I use globus online to transfer files

Globus Online(GO) manages file transfer for you, monitoring performance, retrying failures, auto-tuning and recovering from faults automatically where possible, and reporting status. And there’s no custom infrastructure or software to install, so you don’t have to be an IT wizard to use Globus Online.

As a user you need to follow the following steps to use globus online to transfer files from LONI clusters

  1. Obtain a LONI Grid Certificate
  2. Sign Up for a Globus Online Account
  3. Install GridFTP or Globus on your server.
    • LONI clusters already have Globus tools installed, so you can skip this step for transferring files between LONI clusters or between LONI and other resources which have globus installed
    • To install GridFTP on your local machine, globus online provide a software Globus Connect which will install gridftp and create a local endpoint for use with GO.
  1. Login to the head node where globus is installed e.g. qb1.loni.org and create and store your grid credentials.
  2. Login to your GO account
  3. Click on "Start Transfer". You should see two windows side by side
  4. Select you endpoint for the machines that you want to transfer files between e.g. loni#qb on one window and your local machine on another
  5. Enter the password/passphrase requested
  6. Select files/directories that you want to transfer and hit the transfer button (the triangle).
  7. You will receive an email when the transfer is complete.

To learn more about GO, look through the documentation on the GO website.

If you have issues connecting to LONI clusters, you can either contact us at sys-help@loni.org or contact the GO admins in case there is an issue from their side.


Gt4.0 GridFtp: User's Guide


Users may direct questions to sys-help@loni.org.

Powered by MediaWiki