Note: This page has been marked as "Obselete" by the site administrator.

Currently, Globus and 64 bit Mpich-G2 are available on Bluedawg, Ducky, Zeke and Neptune. As Neptune is currently dedicated to US Army Corps of Engineers, Bluedawg, Ducky and Zeke accessible for general use.


First, users should set up their environment via entries in their ~/.profile or ~/.bashrc similar to the following example:

   export GLOBUS_LOCATION=/usr/local/globus/globus-4.0.4                                                                       
   export PATH=$GLOBUS_LOCATION/bin:$GLOBUS_LOCATION/sbin:$PATH                                                                
   . $GLOBUS_LOCATION/etc/globus-user-env.sh          
   export MPIHOME=/usr/local/packages/mpich-g2-64
   export PATH=$MPIHONE/bin:$PATH
   export LD_LIBRARY_PATH=$MPIHOME/bin:$LD_LIBRARY_PATH

Then following the following steps:

  1. apply for a LONI CA user cert (if you have already done this, skip this step):
    1. locate GLOBUS_LOCATION (the path to Globus package) on any of the LONI machines, run $GLOBUS_LOCATION/bin/grid-cert-request, this will generate three files under .globus in your home: usercert_request.pem, usercert.pem (empty), userkey.pem.
    2. email usercert_request.pem to ca@loni.org,
    3. after receiving the signed certificate from ca@loni.org, copy it to usercert.pem under .globus,
  2. compile your code with compilers (mpicc, mpif90, etc.) from 64 bit mpich-g2 which is under /usr/local/packages/mpich-g2-64.
  3. start Globus proxy by "grid-proxy-init".
  4. Jobs can be submitted through Globus RSL files. Currently, users need to first peek at the queues on both machines to see available procs, then adapt procs request in rsl file, then run "globusrun -f RSLfilename" to launch your job.

Keep in mind that a process running on one machine only has access to the local disk on that machine, you will need to adapt your I/O method when running on two machines simultaneously because data will be written to disks geographically separated to each other.


A sample RSL file requesting 32 procs on Zeke and 32 procs on Ducky:

+
( & (queue= checkpt)
(resourceManagerContact="l2f1n01.sys.loni.org/jobmanager-loadleveler")
(job_type = multiple)
(project = loni_allocation_name)
(count= 32)
(host_count=4)
(minMemory = 50)
(maxWallTime = 3000)
(directory = /work/ou/flower/run)
(environment=
(GLOBUS_DUROC_SUBJOB_INDEX 0) 
(GBLL_NETWORK_MPI sn_single,not_shared,US,HIGH)
(LD_LIBRARY_PATH
/usr/local/globus/globus.4.0.4/lib/:/usr/local/packages/mpich-g2-64/lib)
(PATH
/usr/local/globus/globus-4.0.4/bin/:/usr/local/packages/mpich-g2-64/bin:.)
)
(executable = /work/ou/flower/run/hydro)
(stderr=/work/ou/flower/run/std.err)
(stdout=/work/ou/flower/run/std.out)
)
(
&
(resourceManagerContact="l3f1n01.sys.loni.org/jobmanager-loadleveler")
(project = loni_allocation_name)
(queue= checkpt)
(job_type = multiple)
(count= 32)
(host_count=4)
(minMemory = 50)
(maxWallTime = 3000)
(directory = /work/ou/flower/run)
(environment=
(GLOBUS_DUROC_SUBJOB_INDEX 1)
(GBLL_NETWORK_MPI sn_single,not_shared,US,HIGH)
(LD_LIBRARY_PATH
/usr/local/globus-4.0.4/lib/:/usr/local/packages/mpich-g2-64/lib)
(PATH /usr/local/globus-4.0.4/bin/:/usr/local/packages/mpich-g2-64/bin:.) )
(executable = /work/ou/flower/run/hydro)
(stderr=/work/ou/flower/run/std.err)
(stdout=/work/ou/flower/run/std.out)
)


Here, project is your LONI allocation name, queue is the queue name on LONI machines, count is number of processors needed by your job, host_count is number of nodes, minMemory is the minimum memory in Megabytes required by your job, maxWalltime is max wall clock time in minutes, you can define environment variables such as LD_LIBRARY_PATH in the same way shown in above.

For further information on the integration of GT 4.x and LoadLeveler, see the IBM publication LoadLeveler GT 4.0 Users Guide.

Powered by MediaWiki