Efficiency and Pareto reporting

The efficiency report joins current benchmark identities to launch reports, Slurm accounting, trainer logs, saved artifacts, and optional checkpoint profiling. It performs no training and submits no jobs.

python scripts/efficiency_report.py \
  --output-dir "${PGVL_STORAGE_ROOT}/PGVL-Gym-results/analysis/efficiency/current"

By default the command discovers all six run matrices, launch-report-shaped CSVs under benchmarks/, trainer logs under logs/slurm/, and live sacct records. It writes:

• efficiency.csv, including every matrix row and explicit missing, partial, stale, or complete state;
• pareto.csv, containing only rows on at least one reported Pareto front;
• slurm_accounting.csv, a normalized snapshot that can be reused later;
• job_log_bindings.csv, exact config-to-job log paths and content hashes;
• manifest.json, defining the measurement boundaries and hashing every input matrix and launch report.

Pass the saved accounting snapshot back through --accounting-csv to reproduce the report without querying Slurm. --no-sacct creates an inventory with accounting fields left blank. Missing measurements are never imputed.

What is measured

accounted_gpu_hours is the sum of Slurm elapsed time multiplied by allocated GPUs for every numeric job ID associated with the exact result directory in a discovered launch report or bound to the exact config path by the compute wrapper's log header. It is therefore consumed campaign cost, including recorded retries, rather than an estimated ideal runtime. CPU hours use the same definition. Peak host memory comes from MaxRSS; peak GPU memory and maximum sampled GPU utilization come from Slurm TRES when the cluster records them.

Trainable parameters are parsed only when trainer logs contain one stable count. Training-free adapters report zero only when the log explicitly states that the optimizer is disabled. Checkpoint sizes are direct file sizes for the configured folds.

The report marks two higher-performance/lower-GPU-hour Pareto fronts:

• broad: benchmark, cohort, and shot count match;
• encoder-controlled: feature signature, resolution signature, and encoder provenance also match.

Only complete results that match the current configuration fingerprint and have finite performance and accounted GPU time are eligible. Slide-level balanced accuracy is the default performance objective; select another saved objective with --performance-column.

Optional inference profile

Slurm accounting captures whole-job cost, not isolated test throughput. To add a controlled checkpoint profile, evaluate a completed configuration once:

python eval.py \
  --method pathpt \
  --config benchmarks/tcga_nsclc/configs/pathpt/nsclc_4shot.yaml \
  --ckpt_dir /path/to/results \
  --device cuda:0 \
  --profile \
  --output /path/to/results/efficiency_profile.json

The profile records total/trainable parameters, test samples, wall time, slides per second, and peak allocated/reserved CUDA memory per fold. Its scope is cached-feature test inference including loader time. It excludes offline WSI feature extraction, model construction, and checkpoint loading. Compare throughput only on the same hardware, software environment, batch policy, and test population.