CIMS–macroABM Quick-start¶

This walk-through runs the iterative two-way CIMS↔macroABM linkage locally. It assumes you have completed the Setup Guide (repos cloned, m3linkages_env / cims_env created, uv installed for macroABM, and scenarios/MacroABM Reference/config.toml pointed at your paths).

Heads-up on runtime: a full provincial macroABM run (2014→2050, 10 provinces) and a CIMS run each take a while. Use --max_iterations to cap the outer loop while you are getting set up.

1 · Activate the orchestration environment¶

conda activate m3linkages_env

2 · Run a single iteration first¶

Confirm the plumbing works end-to-end with one pass (no convergence yet):

python linkage_macroabm.py -sc "MacroABM Reference" -mi 1

This will:

Run CIMS for the base scenario → MacroABM Reference_1/cims/outputs/.
Run the provincial macroABM, which extracts the CIMS results, simulates with firms.link() at each CIMS milestone year, writes the production feedback policy, and emits gdp_growth.json.
Run the convergence check (which reports "no previous iteration" on the first pass).

3 · Run the full iterative linkage¶

python linkage_macroabm.py -sc "MacroABM Reference"

The loop repeats CIMS → macroABM until national GDP growth per CIMS milestone period changes by less than the configured tolerance (default 10%) between iterations, or until --max_iterations is hit. Set Models.MacroABM.Convergence.mode = "provincial" to restore the legacy all-provinces-must-pass rule.

Useful flags:

Flag	Meaning
`-sc "<name>"`	Scenario folder under `scenarios/` (required)
`-mi N` / `--max_iterations N`	Stop after at most `N` outer iterations
`-o "<path>"`	Write iteration outputs under an absolute path
`-sk` / `--skip`	Reuse an existing CIMS/macroABM stage if its outputs are already present
`-si N` / `--start_iteration N`	Resume the loop from iteration `N` (see below)
`-rfm Y` / `--rerun-from-milestone Y`	Resume macroABM from milestone year `Y` (warm_start; see below)

3b · Restarting / resuming a linkage¶

Each iteration is written to its own self-contained <scenario>_<N>/ folder, so you can restart the loop from any point without redoing completed work.

Run the next CIMS iteration on the macroABM feedback you already produced. For example, after iteration 1 has finished (its macroabm_production/ feedback exists), run a second iteration — CIMS will automatically ingest iteration 1's macroabm_production service-request policy:

python linkage_macroabm.py -sc "MacroABM Smoke" -si 2 -mi 2

-si 2 starts the outer loop at iteration 2. The CIMS stage copies MacroABM Smoke_1/iterations/cims-macroabm/new_cims_inputs/macroabm_production/ into MacroABM Smoke_2/cims/inputs/policies/ and appends macroabm_production to the CIMS policy list, so CIMS runs on macroABM's growth signal. Resuming requires the previous iteration's feedback to exist (the orchestrator checks and exits with a clear message if it is missing).

Add -sk to also skip any stage whose outputs already exist (e.g. to re-run only the convergence check, or to avoid re-solving a CIMS run you have already completed).

Just the CIMS stage? To confirm CIMS alone runs on the feedback without touching macroABM, call the CIMS runner directly:

console python scripts/cims_macro_runner.py -lp "scenarios/MacroABM Smoke" -i 2

Resume a macroABM run that stopped mid-simulation (e.g. disk full, crash partway through 2040). With warm_start = true, load the latest milestone checkpoint and continue to sim_end_year:

python linkage_macroabm.py -sc "MacroABM Reference" -si 1 -sk -rfm 2035

-sk skips CIMS if iteration 1 outputs already exist. -rfm 2035 tells macroABM to load checkpoint_2035.pkl (or replay from the history checkpoint to 2035 if that file was deleted under rolling retention). The flag applies only to the first macroABM run in that session; later outer-loop iterations use linkage_state.json as usual.

You can also pass the flag directly to the macroABM runner:

python scripts/macroabm_runner.py -lp "scenarios/MacroABM Reference" -i 1 --rerun-from-milestone 2035

3c · Warm-restart (faster iterations)¶

When warm_start = true in the scenario config.toml (enabled by default in MacroABM Reference and MacroABM Smoke), both CIMS and macroABM avoid rerunning from the base year every iteration.

CIMS

Iteration 1 (cold): solves history (<= history_boundary_year, default 2020), saves <scenario>_cims_checkpoint.pkl under the scenario folder, then solves future years.
Iteration 2+ (warm): loads the checkpoint, injects this iteration's macroabm_production feedback, and re-solves only the required post-boundary milestone years.

macroABM

Iteration 1 (cold): simulates through history_boundary_year, saves <scenario>_macroabm_history_checkpoint.pkl, then simulates through sim_end_year. Per-milestone checkpoints are saved under <scenario>_macroabm_checkpoints/ (with checkpoint_retention = "rolling", only the latest milestone file is kept).
Iteration 2+ (warm): loads the checkpoint at the start of the earliest non-converged period and simulates forward with updated CIMS inputs. If an intermediate milestone file was deleted, the driver replays from the history checkpoint.

Each checkpoint_<year>.pkl is a full simulation snapshot through that year, not a delta. You cannot delete checkpoint_2025.pkl and restart from 2025 using only checkpoint_2035.pkl. With checkpoint_retention = "rolling" (default), superseded milestone files are removed automatically; keep the history checkpoint and the latest milestone file. To reclaim disk space from an in-progress run that used the old "all" behaviour, delete every checkpoint_*.pkl except the newest one (leave <scenario>_macroabm_history_checkpoint.pkl untouched).

The checkpoints are reused across iterations — you do not need to rebuild them unless you change the base scenario or the history boundary. To force a fresh history solve, delete the checkpoint files (e.g. scenarios/MacroABM Smoke/MacroABM Smoke_cims_checkpoint.pkl, MacroABM Smoke_macroabm_history_checkpoint.pkl, and the MacroABM Smoke_macroabm_checkpoints/ folder) and re-run iteration 1.

Set warm_start = false to revert to full solves each iteration. See Configuration Reference for details.

3d · (Optional) Run the LabourABM stage¶

To translate the converged macroABM production into occupation-level labour-market outcomes, add "LabourABM" to models_to_run and a [Models.LabourABM] section to the scenario config (see the Setup Guide). Then run the linkage exactly as before:

python linkage_macroabm.py -sc "MacroABM Reference"

The CIMS↔macroABM loop runs to convergence (or --max_iterations) as usual, and then the LabourABM stage runs once. The orchestrator:

Ensures the converged iteration's full simulation_results.h5 exists (exporting it first if the loop stopped at --max_iterations).
Runs scripts/labourabm_runner.py, which drives LabourABM's MacroABM→LabourABM bridge — format the macroABM production HDF5s → build the wide labour-demand scenario → run the labour ABM — and collects the results into <scenario>_<final iteration>/labourabm/.

Because LabourABM computes a per-industry shock as (policy − baseline) / production, it needs a baseline macroABM run in addition to this (policy) run. Point at one with baseline_scenario or baseline_h5; with neither, the policy file is reused as the baseline, giving a zero labour-demand shock (useful only to confirm the plumbing).

Runtime. The labour ABM adds a few minutes on top of the CIMS↔macroABM loop. runner = "uv" downloads its dependencies (including torch) on the first run only. Set heatmap = false under [Models.LabourABM] to skip the heavier occupation-transition-matrix outputs.

You can also run the stage on its own once a converged iteration exists (handy while iterating on labour-side settings):

python scripts/labourabm_runner.py -lp "scenarios/MacroABM Reference" -i <final iteration>

4 · Find your results¶

Each iteration writes a self-contained folder under the scenario directory:

scenarios/MacroABM Reference/
├─ MacroABM Reference_cims_checkpoint.pkl           # (warm_start) shared CIMS history checkpoint
├─ MacroABM Reference_macroabm_history_checkpoint.pkl # (warm_start) shared macroABM history checkpoint
├─ MacroABM Reference_macroabm_checkpoints/         # (warm_start) per-milestone macroABM checkpoints
├─ MacroABM Reference_linkage_state.json            # per-period convergence state for warm restarts
└─ MacroABM Reference_1/
   ├─ cims/
   │   ├─ inputs/                       # CIMS scenario config + policies used this iteration
   │   └─ outputs/                      # standard CIMS result CSVs
   ├─ macroabm/
   │   ├─ cims_data/                    # extracted requested_quantities_* / investment_* matrices
   │   ├─ production_annual_01_CAN_*.csv# production by sector & province (reference copy)
   │   ├─ data_provincial_model.pkl     # built once, reused across iterations
   │   ├─ gdp_growth_by_period.json     # per-period provincial GDP growth (drives convergence)
   │   ├─ gdp_growth.json               # start-to-end totals (reference copy)
   │   ├─ simulation_shallow.h5         # summary time series (every iteration, default)
   │   └─ simulation_results.h5         # full macroABM HDF5 (converged iteration only, default)
   ├─ iterations/cims-macroabm/
   │   └─ new_cims_inputs/
   │       └─ macroabm_production/      # service-request policy fed into the NEXT CIMS run
   │           ├─ macroabm_production_AB.csv
   │           ├─ macroabm_production_BC.csv
   │           └─ ...                   # one per CIMS region (Atlantic provinces -> AT)
   └─ labourabm/                        # (LabourABM stage; only on the final iteration)
       ├─ results.h5                    # full labour-market results (region x occupation time series)
       ├─ macromodel_integration_iamc_results.csv  # IAMC employment/unemployment summary
       ├─ national_expansion_shock_scenario.csv    # wide labour-demand scenario driving the ABM
       ├─ baseline_employment.csv       # node-level baseline employment
       └─ plot_*.png                    # diagnostic plots

The labourabm/ folder only appears when "LabourABM" is in models_to_run (see 3d).

macroabm/gdp_growth_by_period.json — the convergence metric for that iteration.
macroabm/simulation_shallow.h5 — compact per-province summary time series (GDP, unemployment, inflation, …) on every iteration when save_shallow_h5 = true (default).
macroabm/simulation_results.h5 — full macroABM HDF5 output, written only on the converged iteration when save_h5_on_convergence = true (default).
iterations/cims-macroabm/new_cims_inputs/macroabm_production/ — the growth-scaled CIMS service-request CSVs (same 24-column schema as cims-models-fork/csv), automatically picked up by CIMS on the next iteration.
labourabm/ — LabourABM stage outputs, written only on the final iteration when "LabourABM" is in models_to_run: results.h5 (full labour-market time series by region × occupation), the IAMC employment/unemployment summary, the wide labour-demand scenario, baseline employment, and diagnostic plots.

5 · What "converged" means¶

After each macroABM run the linkage records each province's GDP growth for every CIMS milestone period (e.g. 2020-2025, 2025-2030) in gdp_growth_by_period.json, and (by default) a national entry computed from summed provincial GDP levels in simulation_shallow.h5. The next iteration is compared against the previous one; when every period's national GDP growth moves by less than gdp_growth_tolerance (default 0.10), the loop stops.

Set Models.MacroABM.Convergence.mode = "provincial" to restore the legacy rule requiring every province's growth figure to pass for every period.

Periods that still fail are recorded in <scenario>_linkage_state.json; the next iteration warm-restarts both models from the earliest failing period instead of from the base year. Lower the tolerance for a stricter convergence, or set --max_iterations for a hard cap.

Troubleshooting¶

Empty investment matrices → CIMS isn't logging new_stock/total_stock. Check cims-models-fork/results/results_tech.txt contains those lines (the runner adds them automatically; see the setup note).
A CIMS sector/fuel is being skipped → it isn't in cims_macro_sector_map.csv; add or correct the row. Editing the CSV never crashes the run.
A province isn't fed back to CIMS → check cims_macro_region_map.csv and the feedback_enabled flag in the sector map.