Statistical analysis

PGVL-Gym can derive uncertainty, calibration, class-collapse, and paired method comparisons from the saved out-of-fold prediction CSVs. This is a post-processing step: it does not load a model, select a checkpoint, change a split, or rerun training.

python scripts/statistical_report.py \
  --matrix benchmarks/tcga_nsclc/run_matrix.csv \
  --output-dir "${PGVL_STORAGE_ROOT}/PGVL-Gym-results/analysis/statistics/nsclc"

The command accepts repeated --matrix, --cohort, --method, and --experiment options, plus repeated --shots. It defaults to 1,000 bootstrap replicates and a 95% confidence interval. Use --overwrite to replace a previously generated report intentionally.

Identity and resampling contract

Every included metrics.json must match the method and current resolved YAML fingerprint. By default, every configured fold must be complete and every fold<K>_predictions.csv must exist. Duplicate test slides, patients appearing in more than one held-out fold, conflicting case labels, malformed probability columns, and non-normalized probabilities are rejected. Stale and incomplete runs appear in exclusions.csv; --allow-partial is an explicit exploratory override.

Intervals use a stratified patient bootstrap. All slides for one patient are averaged before resampling, and resampling preserves the observed number of patients per class. This avoids treating multiple slides from one patient as independent samples.

Outputs

File	Contents
run_summary.csv	Patient-level point estimates and percentile confidence intervals
fold_metrics.csv	Slide- and patient-level diagnostics for every included fold
reliability.csv	Equal-width calibration bins with counts, confidence, accuracy, and gap
figures/reliability_*.png / .pdf	Patient-level reliability curve and confidence-frequency panel for each included run
paired_comparisons.csv	Optional patient-paired candidate-minus-reference differences
exclusions.csv	Runs or comparisons rejected by an identity/data contract
manifest.json	Input paths and hashes, seed, interval settings, and included run identities

The extended metrics are balanced accuracy, raw accuracy, macro-F1, AUROC, macro AUPRC, Brier score, NLL, ECE, per-class recall, mean normalized predictive entropy, class coverage, maximum predicted-class fraction, exact class collapse, and a 95% near-collapse flag.

Reliability diagrams

Figures are generated by default; use --no-plots for CSV/JSON output only. Each run has a separate PNG and vector PDF so large campaigns do not produce unreadable overlays. The upper panel plots mean predicted confidence against observed accuracy, with a perfect-calibration diagonal and vertical calibration gaps. The lower panel shows the fraction of patients in each confidence bin. The displayed ECE is computed from exactly the same bins as reliability.csv. --bins controls both the table and figures (default: 10 equal-width bins).

These are top-label calibration assessments, not fitted calibrators or class-specific probability curves. Patient probabilities are the mean of that patient's slide probabilities; their maximum defines confidence. Curves pool the included held-out folds and do not fit temperatures or alter predictions. Empty bins remain gaps, not zero-accuracy observations. No per-bin confidence intervals are implied; bootstrap intervals in run_summary.csv describe the reported overall metrics. Small few-shot cohorts can have sparse, noisy bins.

Titles identify cohort, experiment, shots and completed/expected fold count. An explicitly allowed incomplete run is marked PARTIAL; rejected/stale runs produce no figures. manifest.json maps every current figure to its run ID, relative path, format and SHA-256, plus plotting-source hash and Matplotlib version. After --overwrite, use that manifest as the current figure index: unrelated or older unreferenced files are not deleted.

For paired comparisons, select an experiment name:

python scripts/statistical_report.py \
  --matrix benchmarks/tcga_nsclc/run_matrix.csv \
  --reference focus \
  --output-dir /work/hdd/bhwm/dchanda/PGVL-Gym-results/analysis/statistics/nsclc-focus

Pairing is deliberately strict. Reference and candidate must share benchmark, cohort, shot count, feature signature, resolution signature, encoder provenance, patients, and labels. Positive deltas mean the candidate metric is numerically larger; probability_candidate_better correctly reverses the direction for Brier, NLL, and ECE.

Confidence intervals quantify sampling uncertainty for a fixed trained run. They do not replace variation across folds, training seeds, prompt generators, or checkpoints.