Mudanças entre as edições de "Minicluster:Mpich com torque"

Edição das 10h53min de 13 de maio de 2011

O "Outro" Mpiexec

A funcionalidade Torque fornece um novo binário para o mpiexec instalado com o MPICH2 (com o mesmo nome). Este "outro" mpiexec - com a funcionalidade Torque - é produzido por Ohio Supercomputer Center e funciona somente junto com o Torque. Usuários não poderão usá-lo sem um qsub script.

Assim, mantenha o original seguro para rodar sem o Torque e instale o novo nos nós escravos.

Preparação

Mova o mpiexec original (ao menos nos nós escravos) tal que não esteja no path do root ou usuários. Ache-o com

root# which mpiexec
/usr/lib64/mpich2/bin/mpiexec

e então faça uma cópia de segurança.

root# cd /usr/lib64/mpich2/bin
root# mv mpiexec mpiexec2

Instalação

Baixe o último arquivo mpiexec de OSC mpiexec page em downloads. Baixe o arquivo (ou última versão)

wget http://www.osc.edu/~pw/mpiexec/mpiexec-0.83.tgz
wget http://www.osc.edu/~djohnson/mpiexec/mpiexec-0.84.tgz

Descompacte com

tar xvzf mpiexec*.tar.gz

and then cd into the new directory. Mpiexec follows the standard source installation paradigm. Run

./configure --help

to see a list of options. Important options include

• --prefix= - specify where you want to have the binaries installed. They need to be accessible by all of the worker nodes. An NFS mount would be a good choice.
• --with-pbs= - necessary to get the Torque functionality! Specify the location of the Torque installation. If you followed my Torque tutorial, it's located at /var/spool/pbs
• --with-default-comm=mpich2-pmi - used to indicate which version of MPI

Next, run ./configure with all the options necessary. My command looked like this:

./configure --prefix=/shared --with-pbs=/var/spool/pbs/ --with-default-comm=mpich2-pmi

Pbs_iff

To do this next part, Torque will need to already be installed. Mpiexec requires that a file named pbs_iff be on each one of the worker nodes. Normally, this file is only located on the head node and isn't installed as part of the pbs_mom installation, so it needs to be copied out from the head node to each of the other nodes.

There's an easy way to do this by scripting. The first requirement is to have a file with each of the worker nodes listed in it. Assuming Torque is running, this can be generated with

pbsnodes | grep -v = | grep -v '^$' >> machines

• grep -v = excludes all lines that have an equal sign in them
• grep -v '^$' contains a regular expression to delete all empty lines

The "machines" file of all the worker node names can then be used in a quick script to copy pbs_iff to each of the worker nodes. Find the original file with

updatedb && locate pbs_iff

(If you receive an error, apt-get install locate and then try again.) Then, replacing my locations below for the location you found it on your cluster, run

for x in `cat machines`; do rsync /usr/local/sbin/pbs_iff $x:/usr/local/sbin/; done

Next, pbs_iff needs to have its permissions changed to setuid root. (This means the binary runs with root privileges, even when run by a different user.) Again, to do this across all the worker nodes at once, use a script and make sure the location is correct for your setup:

for x in `cat machines`; do ssh $x chmod 4755 /usr/local/sbin/pbs_iff; done

Without these steps, users trying to run mpiexec will see errors like these:

pbs_iff: file not setuid root, likely misconfigured
pbs_iff: cannot connect to gyrfalcon:15001 - fatal error, errno=13 (Permission denied)
   cannot bind to reserved port in client_to_svr

mpiexec: Error: get_hosts: pbs_connect: Unauthorized Request .

Testing

Trying to run a program with mpiexec outside of a Torque job, it will give an error:

mpiexec: Error: PBS_JOBID not set in environment.  Code must be run from a
  PBS script, perhaps interactively using "qsub -I".

At least it's a helpful error! Therefore, in order to test it, mpiexec will need to be called from within a script. Continue on to the Torque and Maui: Submitting an MPI Job page to test.

References

• http://debianclusters.org/index.php/MPICH_with_Torque_Functionality
• Mpiexec - MPI parallel job launcher for PBS
• Mpiexec Readme