Generalization of the composition

As you can see from the previous section, the scaling of the number of computing node is a bit cumbersome.

Fortunately, NixOS Compose provides the notion of role to tackle this issue.

A role is a configuration. In our case, we actually have only two roles: the NFS server and the compute nodes. The configuration of the compute nodes is the same no matter how many compute nodes. Thus having to define the configuration for node1 and node2 is redundant.

roles = {

  node = { pkgs, ... }:
  {
    # add needed package
    environment.systemPackages = with pkgs; [ openmpi ior glusterfs ];

    # Disable the firewall
    networking.firewall.enable = false;

    # Mount the PFS
    fileSystems."/data" = {
      device = "server:/gv0";
      fsType = "glusterfs";
    };
  };

  server = { pkgs, ... }:
  { # ... };
}

Building

nxc build -f g5k-nfs-store

Deploying

Reserving the resources

export $(oarsub --project lab-2025-compas-nxc -l nodes=3,walltime=1:0:0 "$(nxc helper g5k_script) 1h" | grep OAR_JOB_ID)

Starting the nodes

The nxc start command can take an additional yaml file as input describing the number of machines per role, as well as their hostnames.

The following yaml file will create 3 machines: server (one instance per default), two node instances (node1 and node2).

# nodes.yaml
node: 2

You can deploy the composition by passing this yaml file to the nxc start command:

nxc start -m OAR.$OAR_JOB_ID.stdout -W nodes.yaml

Connect to the nodes

nxc connect

Remount the volume (to run once on any of the nodes):

cat /etc/hosts | grep node | cut -f2 -d" " | xargs -t -I{} systemctl --host root@{} restart data.mount

Release the nodes

oardel $OAR_JOB_ID