Benchmark protocols

The benchmark generator provides two protocol families with identical configuration semantics.

TCGA

The TCGA protocol covers:

• NSCLC: lung adenocarcinoma vs. lung squamous cell carcinoma;
• BRCA: invasive ductal vs. invasive lobular carcinoma;
• RCC: clear-cell, papillary, and chromophobe renal cell carcinoma.

It generates 306 configs across the cohort-specific protocol matrices: 16 BRCA and 34 NSCLC experiment variants at 4/8/16/32/64 shots, plus 14 RCC variants at 4/8/16/32 shots. Seven NSCLC variants are explicitly labeled BioClinicalBERT WSI-FiVE encoder extensions and six more are paired-tower ablations; none are native-paper duplicates. The additional ConVLM row is an explicitly partial QuiltNet-B-16 feature-bag reconstruction, not an upstream reproduction. DyKo adds NSCLC/RCC TITAN knowledge conditions, MGPATH adds BRCA/NSCLC PLIP-only conditions, and HIPSS adds an NSCLC hierarchy condition. See the repository's benchmarks/tcga_brca/README.md for cohort construction, feature coverage, and readiness details. The complete 23-source NSCLC feature inventory, including registered zero-coverage magnifications, is documented in benchmarks/tcga_nsclc/README.md.

Additional tasks

The additional-task protocols cover CAMELYON16, UBC-OCEAN, and a separate combined CAMELYON16+17 FOCUS/MUSE matrix with the same fold, shot, encoder, prompt, and feature-provenance machinery. They generate 216 configs: 28 CAMELYON16 variants at 4/8/16/32/64 shots, 22 UBC-OCEAN variants at 4/8/16 shots, and two combined-cohort variants at 4/8/16/32/64 shots. HIPSS and the other explicitly compatible encoder-extension families remain separately named conditions. Together, the six current protocol matrices generate 855 valid, launch-ready configs. See benchmarks/camelyon16/README.md, benchmarks/camelyon_combined/README.md, and benchmarks/ubc_ocean/README.md for task-specific metadata expectations.

Fair-comparison controls

• one seed and the same outer-fold range for every experiment;
• 4, 8, 16, 32, and 64 labeled patients per class where cohort capacity permits (RCC stops at 32; UBC-OCEAN stops at 16; combined CAMELYON 32/64 are disclosed extensions beyond the released 4/8/16 comparison);
• nested few-shot subsets where metadata permits;
• patient/case-disjoint train, validation, and test partitions;
• identical split files across methods;
• method variants named separately when the encoder, resolution pair, or learned projection changes;
• accuracy reported independently from any continuous optimization objective.

Readiness versus validity

config_valid means the configuration is internally consistent and satisfies the adapter's declared contract. ready means every file needed to launch that specific run currently exists. This distinction allows future RCC or additional-resolution features to be registered before extraction finishes, without presenting those runs as executable.

Feature production is asynchronous bookkeeping, not protocol generation. scripts/run_benchmark.py refreshes per-source coverage and each row's missing_feature_files before planning, using the paths already frozen in the manifest. It changes only feature-derived coverage/readiness cells; prompt, encoder, auxiliary, metadata, split, and config validity remain independent gates. The refresh also expands portable ${PGVL_*} paths before checking the filesystem. A full tcga_benchmark.py all regeneration is therefore unnecessary when a backfill merely adds the previously expected feature files.

The inventory and prepare commands validate the registry schema without importing model or Torch runtime code. Runtime encoder capability and native width checks still run for configs, validate, and all. This keeps feature bookkeeping usable on login nodes while preserving strict launch validation.

Generated artifacts

Each protocol directory contains:

Artifact	Purpose
protocol.yaml	source-of-truth registry
run_matrix.csv	one row and launch command per run
config_audit.csv	normalized configuration comparison
feature_coverage.csv	per-source missing/available counts
validation_report.json	static validation result
smoke_report_<cohort>.json	isolated model-build/forward result
configs/	generated run YAML files
splits/	deterministic few-shot fold files