Add another compute node

For the moment we have a single compute node. In this section, we will add another one and run IOR on several nodes.

Add another role in the composition

We will rename the role node into node1 and create a new role node2 with the exact same configuration:

roles = {

  node1 = { pkgs, ... }:
  {
    # add needed package
    environment.systemPackages = with pkgs; [ openmpi ior ];
  
    # Disable the firewall
    networking.firewall.enable = false;
  
    # Mount the NFS
    fileSystems."/data" = {
      device = "server:/";
      fsType = "nfs";
    };
  };

  node2 = { pkgs, ... }:
  {
    # add needed package
    environment.systemPackages = with pkgs; [ openmpi ior ];
  
    # Disable the firewall
    networking.firewall.enable = false;
  
    # Mount the NFS
    fileSystems."/data" = {
      device = "server:/";
      fsType = "nfs";
    };
  };

  server = { pkgs, ... }:
  { # ... };
}

Building

nxc build -f g5k-nfs-store

Deploying

Reserving the resources

export $(oarsub -l nodes=3,walltime=1:0:0 "$(nxc helper g5k_script) 1h" | grep OAR_JOB_ID)

Getting the machine file

oarstat -u -J | jq --raw-output 'to_entries | .[0].value.assigned_network_address | .[]' > machines

Deploying

nxc start -m machines

Connect to the nodes

nxc connect

After building and starting the environment, we now have 3 nodes: node1, node2 and the server.

From the NFS server (/data on the nodes), we can try to run IOR by specifying the hosts.

The nodes already know each others (you can look at /etc/hosts to verify). So we will create the MPI hostfile in myhosts:

cd /data
printf "node1 slots=8\nnode2 slots=8" > myhosts

The /data/myhosts file should look like:

node1 slots=8
node2 slots=8

Now, from any node (node1 or node2), we can run start the benchmark (without the high performance network of Grid'5000) with:

cd /data
mpirun --mca pml ^ucx --mca mtl ^psm2,ofi --mca btl ^ofi,openib --allow-run-as-root --hostfile myhosts -np 16 ior