Open MPI Master-Servant Pattern: C Examples & Tutorial

Building on the Getting started... post from last week I've knocked up a quick example showing one way to get your MPI processes to communicate with one another.

Basic Master-Servant Pattern

The fundamental concept in MPI parallel processing is distinguishing between a master process and servant (worker) processes. Here's a simple example that demonstrates this pattern:

master_servant.c:

#include <stdio.h>
#include <mpi.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    int numprocs, rank, namelen;
    char processor_name[MPI_MAX_PROCESSOR_NAME];

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Get_processor_name(processor_name, &namelen);

    if ( rank == 0 ) {
        printf( "[%02d/%02d %s]: I am the master\n", rank, numprocs, processor_name );
        // Tell the servants to do something
    } else {
        printf( "[%02d/%02d %s]: I am a servant\n", rank, numprocs, processor_name );
        // Wait for something to do
    }

    MPI_Finalize();
    return 0;
}

Building and Running

Build this with the MPI compiler wrapper:

mpicc master_servant.c -o master_servant

When you run it, you should get output like this:

[00/08 mpinode01]: I am the master
[01/08 mpinode01]: I am a servant
[02/08 mpinode01]: I am a servant
[03/08 mpinode01]: I am a servant
[04/08 mpinode02]: I am a servant
[05/08 mpinode02]: I am a servant
[06/08 mpinode02]: I am a servant
[07/08 mpinode02]: I am a servant

This demonstrates that based on the rank returned by MPI_Comm_rank we can decide which instance of the program is going to act as the "master" and which instance(s) are going to act as "servants" - pretty neat!

Inter-Process Communication

Next example, we build on this by getting the program instances to communicate with one another. This example is borrowed from the example found here.

master_servant2.c:

#include <stdio.h>
#include <string.h>
#include <mpi.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    char idstr[32], buff[128];
    int numprocs, rank, namelen, i;
    char processor_name[MPI_MAX_PROCESSOR_NAME];
    MPI_Status stat;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Get_processor_name(processor_name, &namelen);

    // Based on example from https://wiki.inf.ed.ac.uk/pub/ANC/ComputationalResources/slides.pdf
    if (rank == 0) {
        // This is the rank-0 copy of the process
        printf("We have %d processors\n", numprocs);

        // Send each process a "Hello ... " string
        for(i = 1; i < numprocs; i++) {
            sprintf(buff, "Hello %d... ", i);
            MPI_Send(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD);
        }

        // Go into a blocking-receive for each servant process
        for(i = 1; i < numprocs; i++) {
            MPI_Recv(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD, &stat);
            printf("%s\n", buff);
        }
    } else {
        // Go into a blocking-receive waiting
        MPI_Recv(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &stat);

        // Append our identity onto the received string
        sprintf(idstr, "Processor %d ", rank);
        strcat(buff, idstr);
        strcat(buff, "reporting!");

        // Send the string back to the rank-0 process
        MPI_Send(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
    }

    MPI_Finalize();
    return 0;
}

Building and Testing

Build this example with the MPI compiler wrapper:

mpicc master_servant2.c -o master_servant2

When you run it, you should get the following output:

We have 8 processors
Hello 1... Processor 1 reporting!
Hello 2... Processor 2 reporting!
Hello 3... Processor 3 reporting!
Hello 4... Processor 4 reporting!
Hello 5... Processor 5 reporting!
Hello 6... Processor 6 reporting!
Hello 7... Processor 7 reporting!
Hello 8... Processor 8 reporting!

Practical Applications

Now you can use this master/servant technique to partition work across instances of your MPI-capable program. The master process coordinates the work distribution while servant processes handle the actual computation tasks.

Troubleshooting

If you get an error like this:

[hostname][0,1,0][btl_tcp_endpoint.c:572:mca_btl_tcp_endpoint_complete_connect] connect() failed with errno=113

Try shutting down iptables on the MPI nodes. It was a quick-fix for me, i am sure there is a 'proper' way to configure it though.

Security Note: Keep in mind it's probably not a good idea to switch off iptables on a machine if it's connected to the open internet. The machines I have used in this guide are all on an internal network.

Building on the Getting started... post from last week I've knocked up a quick example showing one way to get your MPI processes to communicate with one another.

Basic Master-Servant Pattern

The fundamental concept in MPI parallel processing is distinguishing between a master process and servant (worker) processes. Here's a simple example that demonstrates this pattern:

master_servant.c:

#include <stdio.h>
#include <mpi.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    int numprocs, rank, namelen;
    char processor_name[MPI_MAX_PROCESSOR_NAME];

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Get_processor_name(processor_name, &namelen);

    if ( rank == 0 ) {
        printf( "[%02d/%02d %s]: I am the master\n", rank, numprocs, processor_name );
        // Tell the servants to do something
    } else {
        printf( "[%02d/%02d %s]: I am a servant\n", rank, numprocs, processor_name );
        // Wait for something to do
    }

    MPI_Finalize();
    return 0;
}

Building and Running

Build this with the MPI compiler wrapper:

mpicc master_servant.c -o master_servant

When you run it, you should get output like this:

[00/08 mpinode01]: I am the master
[01/08 mpinode01]: I am a servant
[02/08 mpinode01]: I am a servant
[03/08 mpinode01]: I am a servant
[04/08 mpinode02]: I am a servant
[05/08 mpinode02]: I am a servant
[06/08 mpinode02]: I am a servant
[07/08 mpinode02]: I am a servant

Inter-Process Communication

Next example, we build on this by getting the program instances to communicate with one another. This example is borrowed from the example found here.

master_servant2.c:

#include <stdio.h>
#include <string.h>
#include <mpi.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    char idstr[32], buff[128];
    int numprocs, rank, namelen, i;
    char processor_name[MPI_MAX_PROCESSOR_NAME];
    MPI_Status stat;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Get_processor_name(processor_name, &namelen);

    // Based on example from https://wiki.inf.ed.ac.uk/pub/ANC/ComputationalResources/slides.pdf
    if (rank == 0) {
        // This is the rank-0 copy of the process
        printf("We have %d processors\n", numprocs);

        // Send each process a "Hello ... " string
        for(i = 1; i < numprocs; i++) {
            sprintf(buff, "Hello %d... ", i);
            MPI_Send(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD);
        }

        // Go into a blocking-receive for each servant process
        for(i = 1; i < numprocs; i++) {
            MPI_Recv(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD, &stat);
            printf("%s\n", buff);
        }
    } else {
        // Go into a blocking-receive waiting
        MPI_Recv(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD, &stat);

        // Append our identity onto the received string
        sprintf(idstr, "Processor %d ", rank);
        strcat(buff, idstr);
        strcat(buff, "reporting!");

        // Send the string back to the rank-0 process
        MPI_Send(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
    }

    MPI_Finalize();
    return 0;
}

Building and Testing

Build this example with the MPI compiler wrapper:

mpicc master_servant2.c -o master_servant2

When you run it, you should get the following output:

We have 8 processors
Hello 1... Processor 1 reporting!
Hello 2... Processor 2 reporting!
Hello 3... Processor 3 reporting!
Hello 4... Processor 4 reporting!
Hello 5... Processor 5 reporting!
Hello 6... Processor 6 reporting!
Hello 7... Processor 7 reporting!
Hello 8... Processor 8 reporting!

Practical Applications

Troubleshooting

If you get an error like this:

[hostname][0,1,0][btl_tcp_endpoint.c:572:mca_btl_tcp_endpoint_complete_connect] connect() failed with errno=113

Try shutting down iptables on the MPI nodes. It was a quick-fix for me, i am sure there is a 'proper' way to configure it though.

Michael Cutler

An Open MPI Master & Servant Example

Basic Master-Servant Pattern

Building and Running

Inter-Process Communication

Building and Testing

Practical Applications

Troubleshooting

Related Posts

Open MPI Master & Servant Example - BogoMips

Getting Started with Open MPI on Fedora

Fixing the Heartbleed vulnerability on CentOS

An Open MPI Master & Servant Example

Basic Master-Servant Pattern

Building and Running

Inter-Process Communication

Building and Testing

Practical Applications

Troubleshooting

Related Posts

Open MPI Master & Servant Example - BogoMips

Getting Started with Open MPI on Fedora

Fixing the Heartbleed vulnerability on CentOS