Adding scripts to the environment

The command to start the benchmark is quite obscure and cumbersome to type. We would like to create a script to wrap it. However, creating a full Nix package for such a small script is not worth it. Fortunately, Nix provides ways to create reproducible bash (or others) script easily.

Let us create a Nix file called myscripts.nix. It will be a function that takes pkgs as input and return the set of our scripts.

{ pkgs, ... }:
let
  # We define some constants here
  nfsMountPoint = "/data";
  nbProcs = 16;
  iorConfig = "/etc/ior_script";
in {
  # This function takes the number of compute nodes,
  # creates a hostfile for MPI and runs the benchmark
  start_ior =
    pkgs.writeScriptBin "start_ior" ''
      cd ${nfsMountPoint}

      NB_NODES=$(cat /etc/hosts | grep node | wc -l)
      NB_SLOTS_PER_NODE=$((${builtins.toString nbProcs} / $NB_NODES))

      cat /etc/hosts | grep node | awk -v nb_slots="$NB_SLOTS_PER_NODE" '{ print $2 " slots=" nb_slots;}' > my_hosts

      mpirun --mca pml ^ucx --mca mtl ^psm2,ofi --mca btl ^ofi,openib --allow-run-as-root -np ${builtins.toString nbProcs} --hostfile my_hosts ior -f ${iorConfig}
    '';
   remount_glusterfs =
    pkgs.writeScriptBin "remount_glusterfs" ''
        cat /etc/hosts | grep node | cut -f2 -d" " | xargs -t -I{} systemctl --host root@{} restart data.mount
    '';

}

We can now import these scripts in the composition:

# ...
  node = { pkgs, ... }:
  let
    scripts = import ./myscripts.nix { inherit pkgs; };
  in
  {
    networking.firewall.enable = false;

    environment.systemPackages = with pkgs; [ openmpi ior glusterfs scripts.start_ior scripts.remount_glusterfs ];

    environment.etc = {
      ior_script = {
        text = builtins.readFile ./script.ior;
      };
    };

    # Mount the PFS
    fileSystems."/data" = {
      device = "server:/gv0";
      fsType = "glusterfs";
    };
  };
# ...

Building

nxc build -f g5k-nfs-store

Deploying

Reserving the resources

export $(oarsub --project lab-2025-compas-nxc -l nodes=2,walltime=1:0:0 "$(nxc helper g5k_script) 1h" | grep OAR_JOB_ID)

Starting the nodes

nxc start -m OAR.$OAR_JOB_ID.stdout -W

Connecting

nxc connect

Running the script

Remount the volumes on the nodes (to run once on a single node):

remount_glusterfs

We can now simply run start_ior from the node to run the benchmark.

start_ior

Release the nodes

oardel $OAR_JOB_ID