Skip to content

Appendix D: Running COPPER on High Performance Computing Clusters

This appendix addresses the use of the Digital Research Alliance of Canada clusters, particularly Fir to run the COPPER model.

Required Packages:

  • Load your environment with a Python version that is python>=3.10
  • Download the coders2iamc package from the pypi organization website here. This wheel is relevant for interactions between COPPER and the CODERS database

Creating the Environment Using virtualenv:

  1. Activate Python: module load python/version

  2. Create your virtual environment using "virtualenv": virtualenv --no-download ENV_NAME

  3. Activate the environment: source ENV_NAME/bin/activate

  4. Install requirements individually:

    4.1. Search for package you wish to install: avail_wheels "PACKAGE_NAME"

    4.2 - Choose the library wheel from where to load the module when installing with pip

    4.3 - Run the pip install command pasting the path to the library wheel of choice: pip install PACKAGE --no-index, or pip install PACKAGE/VERSION --no-index if multiple versions are available and you have a preference.

    4.4 - One can also just run the command: pip install --no-index -r requirements.txt to install from a requirements file

Activating Packages Which are not Installed in the environment

module load python/3.10 arrow/14.0.1 StdEnv/2023

In this example, these are the dependencies where pyarrow and toml are located in Fir, for example

Example Job Submission Files

The following script is an example of running a single COPPER scenario as a job on Compute Canada. The script is divided into two sections. The first section contains the environment setup, and the second section contains the scenario execution.

#!/bin/bash
#SBATCH --time=22:59:59
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=57000M
#SBATCH --account=YOUR_PROJECT_ACCOUNT
#SBATCH --mail-user=YOUR_CEDAR_ACCOUNT_EMAIL
#SBATCH --mail-type=ALL

module load python/3.10 StdEnv/2023 gcc/12.3 arrow/15.0.1
echo ""
virtualenv --no-download $SLURM_TMPDIR/env
source $SLURM_TMPDIR/env/bin/activate
echo ""

echo "made env, now installing packages"
echo ""
pip install --no-index --upgrade pip
pip install --no-index -r /home/YOUR_USER/projects/PROJECT_ACCOUNT/YOUR_USER_FOLDER_ON_THE_PROJECT/path_to_environment_requirements/requirements.txt
echo "installed packages, now running python script"
echo ""
export PATH=$PATH:/project/rrg-mcpher16/SHARED2/software/cplex/cplex/bin/x86-64_linux/

echo "COPPER simulation starting"
srun python /home/YOUR_USER/projects/PROJECT_ACCOUNT/YOUR_USER_FOLDER_ON_THE_PROJECT/path_to_copper_folder/COPPER.py
wait
echo "COPPER simulation completed"

The script below demonstrates how you can run a series of COPPER runs using job arrays. Like before, the Python environment is created alongside the job. 

#!/bin/bash

#SBATCH --account=def-mcpher16
#SBATCH --cpus-per-task=16
#SBATCH --mem=125G
#SBATCH --time=22:59:59
#SBATCH --output slurm_test.out
#SBATCH --array=1-10

module load python/3.10 StdEnv/2023 gcc/12.3 arrow/15.0.1
echo ""
virtualenv --no-download $SLURM_TMPDIR/env
source $SLURM_TMPDIR/env/bin/activate
echo ""

echo "made env, now installing packages"
echo ""
pip install --no-index --upgrade pip
pip install --no-index -r /home/YOUR_USER/projects/PROJECT_ACCOUNT/YOUR_USER_FOLDER_ON_THE_PROJECT/path_to_environment_requirements/requirements.txt
echo "installed packages, now running python script"
echo ""
export PATH=$PATH:/project/rrg-mcpher16/SHARED2/software/cplex/cplex/bin/x86-64_linux/

# Define a function to generate different input flags for each task
generate_flags() {
  local i=$SLURM_ARRAY_TASK_ID
  case $i in
    1) flags="-sc BAU_CEDAR_Scenario -t false -so cplex";;
    2) flags="-sc BAU_CEDAR_Scenario -t false -so cbc";;
    3) flags="-sc BAU_CEDAR_Scenario -t false -se 4000";;
    4) flags="-sc BAU_CEDAR_Scenario -t false -so cplex -rd true";;
    *) flags="";; # or raise an error
  esac
}

# Call the function to generate input flags for each task
generate_flags

echo ""
echo "Job Array ID / Job ID: $SLURM_ARRAY_JOB_ID / $SLURM_JOB_ID"
echo "This is job $SLURM_ARRAY_TASK_ID out of $SLURM_ARRAY_TASK_COUNT jobs."
echo ""

# Run your Python script with the generated input flags
srun python COPPER.py ${flags}

Note that these example job submission files create the environment to run the COPPER script alongside with the job itself. This is the recommended way in which to run your jobs that need a Python environment, as the environment will be created inside the $SLURM_TMPDIR temporary directory, hence being job-specific.

Running Jobs

Here are the steps for running COPPER using Compute Canada computers:

  1. Navigate to your project folder on your fir.alliancecan.ca SSH connection (something like: /home/your_username/projects/your_project_name/your_username)
  2. Clone COPPER from the main repository: git clone https://gitlab.com/sesit/copper.git
  3. Navigate to the "copper" folder in your terminal: cd copper
  4. Upload your scenario folders to the "scenarios" folder and DELETE all other scenarios, EXCEPT the "common" folder
  5. Play around with the parameters in your shell script file (those in the header of the file, following #SBATCH)
  6. Submit the job from within the "copper" directory (this way the output log is saved to this folder): sbach my_runs.sh

    Make sure to set the save_lp flag from the scenario you wish to run (in the config.toml file) to FALSE before submitting your job

COPPER benchmark on the FIR cluster: after running 1600 full-length (12 run days) simulations using 8 CPUs each on the FIR cluster, the total CPU runtime observed averaged to 2:51:32 (h:mm:ss), and the average per-CPU runtime was 1:13:30 (h:mm:ss)

References

Python Environments FAQ

Running Jobs on DRA Computers

SLURM Workload Manager

Storage on Dgital Research Alliance Clusters

SLURM Commands