Setup

Four repositories, three environments. Everything is cloned under dependencies/ of M3-linkages; the linkage never modifies the model repos beyond writing into the scenario folders it owns.

1 · Repositories

git clone -b cims-macro-linkage-v3 https://gitlab.com/sesit/M3-linkages.git
cd M3-linkages
git clone -b cims-macro-linkage-v3 https://github.com/uvic-sesit/macroabm-ca.git dependencies/macroabm-ca
git clone -b cims-macro-linkage-v3 https://gitlab.com/sesit/macrocosm/labourabm/model.git dependencies/labourabm
git clone -b EPM_working https://gitlab.com/sesit/cims-epm.git dependencies/cims-epm
Repo Branch Role
M3-linkages cims-macro-linkage-v3 orchestration, adapter, linkage scripts, scenario configs
macroabm-ca cims-macro-linkage-v3 MacroABM-CA (exports the labour-force index the LabourABM stage needs)
labourabm cims-macro-linkage-v3 LabourABM (absolute demand mode, labour-force entries)
cims-epm EPM_working CIMS — the version the workflow runs; its SESIT scenario runner (scenarios/scenario_runner.py) is the contract the CIMS stage drives

COPPER is not a dependency: the workflow consumes staged COPPER outputs (copper/outputs/*_output_summary_IDEA.csv) and never launches it.

2 · Environments

  • Orchestration (m3linkages_env, conda): runs linkage_cims_copper_macroabm.py and every scripts/*.py. Needs toml, pandas, numpy, tables, and plotly (for the analysis report).
  • CIMS-EPM (uv): cd dependencies/cims-epm && uv sync — creates the model's own virtual environment (Python 3.12, polars/pandas pinned by its pyproject.toml). The CIMS stage invokes uv run python ./scenarios/ scenario_runner.py from that repo root, so uv must be on the PATH of the shell the orchestrator runs in.
  • MacroABM-CA and LabourABM (uv): both are run through uv run by their runners; uv syncs them on first use.

3 · CIMS model inputs

dependencies/cims-epm/data/model_inputs/ (model/, policies/, defaults/) must be present — the EPM_working checkout carries them. The inputs are in CIMS's long format (Year/Value columns); the wide-to-long converter in cims-epm/scripts/ exists for older input sets.

A linkage scenario's CIMS run reads <scenario>/cims/inputs/:

<scenario>/cims/inputs/
├── config.toml        # CIMS run config (regions, years, sectors, policies)
└── policies/          # scenario-specific policy folders, incl. the
                       # copper_electricity_prices / _multipliers feed of a
                       # converged CIMS-COPPER iteration

Policies named in the config resolve first from that policies/ folder, then from the repo's common data/model_inputs/policies/ tiers — a scenario folder copied from a converged CIMS-COPPER run is self-contained.

Wide vs long policy files — a silent trap the stage guards against. Older CIMS vintages wrote policy CSVs in a wide layout (a navigation row plus one column per year); CIMS-EPM reads the long layout (Year/Value columns) and applies wide files without error and without effect (measured in a test run: 2025 personal-transport electricity 3.0 PJ with the reference policy set in wide form vs 33.5 PJ with the same policies in long form). The CIMS stage therefore runs the repo's idempotent converter (scripts/convert_model_inputs_wide_to_long.py --root <scenario>/cims/inputs) on the scenario's policies/ before every run, so a folder copied from an older run — including its copper_electricity_prices feed — is normalised automatically.

4 · Scenario layout

scenarios/<name>/
├── config.toml                  # the linkage scenario config
├── cims/inputs/                 # CIMS inputs (config + policies)         [staged]
├── cims/outputs/                # CIMS outputs                            [CIMS stage, or staged]
├── copper/outputs/              # COPPER outputs                          [staged]
├── iterations/cims-copper/      # iteration price files                   [staged, optional]
├── data/                        # adapter-derived linkage inputs          [generated]
├── macroabm/                    # MacroABM-CA outputs                     [generated]
├── labourabm/                   # LabourABM outputs                       [generated]
└── IDEA/                        # consolidated IDEA-compatible artefacts  [generated]

The [Models.CIMS] block in config.toml turns the CIMS stage on:

[General]
    models_to_run = ["CIMS", "CIMSCOPPER", "MacroABM", "LabourABM"]

[Models.CIMS]
    repo_path = "dependencies/cims-epm"
    runner    = "uv"
    inputs    = "cims/inputs"
    outputs   = "cims/outputs"
    strict_policies = true        # abort on an unresolved policy name
    # [Models.CIMS.Config]        # optional overrides merged into cims/inputs/config.toml
    #     [Models.CIMS.Config.General]
    #     year_list = [2000, 2005, ..., 2050]

Omit the block (and "CIMS" from models_to_run) to run against staged CIMS outputs instead. The stage skips the run when outputs already exist; python linkage_cims_copper_macroabm.py --refresh forces it. A full 2000–2050 CIMS run takes on the order of an hour per scenario.

[Models.CIMSCOPPER] names the files the adapter reads, and [Models.EnergyData] the adapter-derived CSVs and mapping tables the macro stage consumes (the CER pipeline's original [Models.CER] name is still accepted). The CIMS stage writes the adapter's inputs as <scenario>_results_general.csv, results_tech.csv and <scenario>_results_general_IDEA.csv (the scenario name is the folder name).

5 · Mapping tables (in scripts/data/)

  • cims_copper_sector_map.csv — CIMS sector → macro industry codes and fuel-name assignments (household-routed sectors → L).
  • cims_copper_allocation.csv — size shares of the industries pooled under CIMS Light Industrial.
  • cims_copper_cost_reference_{ref,nz}.csv, cims_copper_transition_composition.csv — the transition-capital cost and composition tables.
  • cer_generation_technology_map.csv — COPPER technology strings → reference technologies (1:1 rows appended for the CIMS-COPPER workflow).
  • cer_investment_tax_credit.csv — scenario-keyed ITC table; the pair-level prep appends each scenario's rows under its own label.
  • policies.csv — the policy assumptions database (copy of the maintained SharePoint original); source of the policy_*.csv overlay tables via the policy_*.py builders.

See the methodology for how each is used.

6 · MacroABM-CA data — on the shared drive, in no repository

The macro stage needs two inputs that no repo carries. Both live in the SESIT SharePoint folder Projects - Shared/SESIT - MacroABM data (sync it via OneDrive or copy it locally); the CIMS-COPPER run folders for the staged pairs (cims-copper-raw.zip) are in the same place.

Item Config key What it is
raw_data/ [Models.MacroABM] raw_data_path MacroABM-CA raw-data root: ICIO tables, canadian_inputs/ (StatCan provincial series), the PUMF drops under can_2022/pumf/, exchange rates, ECB/IMF/OECD series. Point the key at the root itself (the folder holding icio/, canadian_inputs/, ihs_markit_goods_criticality/); the runner walks up one level if given icio/ or canadian_inputs/. With the pinned pickle present it is never read, but the key is mandatory.
io2022_10prov_2022_v2_pumf_built-2026-08-18.pkl [Models.MacroABM.Config] pkl_path The pinned provincial DataWrapper pickle: 10-province 2022 IO base, v2 PUMF households (SFS-2023 + CIS-2022 + SHS-2023), LFS-2022 t0 unemployment. 20 MB, built once on 2026-08-18. On the build machine it sits in dependencies/macroabm-ca/dev/pkl_files/, but macroabm-ca git-ignores all of dev/, so a fresh clone never has it — take it from the shared folder.

Both keys in the staged configs are absolute paths from the machine the pair was staged on (C:/gitlab/integrated/M3-linkages/... and a C:\Users\...\SESIT - MacroABM data\raw_data OneDrive path). Edit them for your machine before the first run. pkl_path may be relative — it is resolved against the M3-linkages root, then the working directory; raw_data_path is resolved against the working directory, so an absolute path is safer.

A missing pickle aborts the macro stage (FileNotFoundError: Provincial data pickle not found from run_cer_linkage.py) — this is what a fresh clone hits. Do not work around it by deleting the pkl_path key: with the key unset the stage silently builds a pickle from raw_data_path with the generic CER recipe (French-proxy households, no LFS calibration), i.e. a different population from the staged runs. And the population sampler draws randomly, so even a rebuild with the right recipe is not bit-identical to the pinned file. Results only reproduce the staged runs with the pinned pickle itself.

If the pinned file is ever lost, the recipe (from the macroabm-ca root, with MACROABM_RAW_DATA pointing at the shared raw_data/) is:

uv run python dev/io2022/household_prototype/prepare_household_canadianization.py --real
uv run python dev/io2022/household_prototype/prepare_household_consumption.py
uv run python dev/io2022/build_2022_datawrapper_10prov.py --canadianized-households --lfs-unemployment --build-only --pickle dev/pkl_files/<new-name>.pkl

and every config that pins the old name must be re-pointed and the pair rerun.

Nothing else under macroabm-ca dev/ is needed to run the workflow. dev/raw_data_10prov/ only feeds scripts/build_transition_allocation_2022.py, whose output (cims_copper_allocation.csv) is committed; the dev/io2022/household_prototype/prototype_household*.csv files are intermediates of the pickle build; the other dev/pkl_files/*.pkl variants are earlier bases that no staged config references. LabourABM's inputs (data/labour_abm_inputs/) and CIMS-EPM's model inputs are tracked in their repos.