CP-HRIC · Experiment 002

APL or another AML?

One question at a second hospital: is it acute promyelocytic leukaemia (APL, PML::RARA) or any other AML?

CP-HRIC-MIL: MatchedCP-HRIC-S1: In progress
The green cp-hric-mil robot at a fluorescence microscope beside the orange System One robot pipetting samples

The experiment

Data, paper and protocol.

Data
Johns Hopkins deepAPL (figshare 14294675), Johns Hopkins Hospital. 106 patients, 14,241 single-cell images. Licence CC0. Images cleaned before analysis (v2 (duplicates and non-cell images removed)).APL 34Other AML 72
Papers
MILLIE, Manescu et al. 2023, Scientific Reports: AUROC 0.96 ± 0.02 by random patient-level 3-fold cross-validation.Reported as the mean over its 3 folds ± the spread across folds.DeepAPL, Sidhom et al. 2021, npj Precision Oncology: AUROC 0.86 by a fixed split: 82 training patients, 24 held-out patients (score matched; the model was not re-run).Reported as one AUROC on the 24 held-out patients, no spread; its model was not re-run.
Protocol
Two published ways to test, shown side by side because they are not comparable: MILLIE's random 3-fold cross-validation over all 106 patients, and DeepAPL's fixed split, where the 24 held-out patients are never used for training or tuning. Chance is 0.50.

At a glance

Our multi-task models, on the paper’s own test.

Each is one model trained on more data to answer several questions, scored on the paper’s own test and against the paper’s own number.

  • 0.926CP-HRIC-MIL · multi-task model · MILLIE's test · 3-fold CV · 106 patients · ± 0.031 · 10 seeds · −0.034 vs MILLIE · 10 models combined: 0.967Matched
  • 0.869CP-HRIC-MIL · multi-task model · DeepAPL's test · 24 held-out · ± 0.034 · 10 seeds · +0.009 vs DeepAPL · 10 models combined: 0.875Matched within margin of error
  • 0.885CP-HRIC-S1 · multi-task model · MILLIE's test · 3-fold CV · 106 patients · ± 0.025 · 10 seeds · −0.075 vs MILLIE · 10 models combined: 0.944In progress
  • 0.864CP-HRIC-S1 · multi-task model · DeepAPL's test · 24 held-out · ± 0.038 · 10 seeds · +0.004 vs DeepAPL · 10 models combined: 0.917Matched within margin of error
MILLIE's test · 3-fold CV · 106 patients · AUROC several questions System One never trained here / reference N models combined MILLIE (0.96)
  1. DinoBloom-G + MILLIE-style headthis data only · 10 seedsshortcut-only floor 0.661 ± 0.031 for this protocol0.967+0.007
  2. CP-HRIC-S1 · multi-task modelseveral questions · 10 seeds · 10 models combinedshortcut-only floor 0.634 ± 0.031 for this protocol0.885−0.075
  3. CP-HRIC-MIL · multi-task modelseveral questions · 10 seeds · 10 models combinedshortcut-only floor 0.634 ± 0.031 for this protocol0.926−0.034

Bar = mean across seeds · whisker = ± spread across seeds · diamond = the seeds’ models combined into one · axis starts at 0.5. Deltas vs MILLIE.

DeepAPL's test · 24 held-out · AUROC several questions System One never trained here / reference N models combined DeepAPL (0.86)
  1. DinoBloom-G + MILLIE-style headthis data only · 10 seedsshortcut-only floor 0.642 for this protocol0.889+0.029
  2. CP-HRIC-S1 · multi-task modelseveral questions · 10 seeds · 10 models combinedshortcut-only floor 0.608 for this protocol0.864+0.004
  3. CP-HRIC-MIL · multi-task modelseveral questions · 10 seeds · 10 models combinedshortcut-only floor 0.608 for this protocol0.869+0.009
  4. CP-HRIC-S1 · multi-task model, decider-vision featuresseveral questions · 10 seeds · 10 models combinedshortcut-only floor 0.608 for this protocol0.851−0.009

Bar = mean across seeds · whisker = ± spread across seeds · diamond = the seeds’ models combined into one · axis starts at 0.5. Deltas vs DeepAPL (its published score; the model was not re-run).

Models

Every result, on every test.

Our models and the reference on every test in this experiment. The headline is the mean across seeds ± the spread across seeds, with the range; where the seeds’ models were also combined into one, that score follows. “Not run” rows stay visible until they are filled.

ModelMILLIE's test · 3-fold CV · 106 patientsAUROC · MILLIE 0.96 ± 0.02DeepAPL's test · 24 held-outAUROC · DeepAPL 0.86 (score matched)
Trained on this experiment's data only
DinoBloom-G + MILLIE-style headpublished foundation encoder (DinoBloom-G), frozen, with a MILLIE-style head trained on Johns Hopkins patients only0.967 ± 0.01410 seeds · range 0.935–0.981+0.007 vs MILLIEshortcut-only floor 0.661 ± 0.031 for this protocolMatched0.889 ± 0.00910 seeds · range 0.875–0.903+0.029 vs DeepAPLshortcut-only floor 0.642 for this protocolMatched
Trained on more data (Munich and Johns Hopkins patients), several questions at once
CP-HRIC-S1 · multi-task modelone System One model for the genetic subtype and APL; Munich patients + the Johns Hopkins training patients0.885 ± 0.02510 seeds · range 0.840–0.918−0.075 vs MILLIE10 models combined: 0.944 (−0.016)shortcut-only floor 0.634 ± 0.031 for this protocolIn progress0.864 ± 0.03810 seeds · range 0.806–0.917+0.004 vs DeepAPL10 models combined: 0.917 (+0.057)shortcut-only floor 0.608 for this protocolMatched within margin of error
CP-HRIC-MIL · multi-task modelone model with a head per question (declared recipe f1-l34-lr1e-5); Munich patients + the Johns Hopkins training patients0.926 ± 0.03110 seeds · range 0.863–0.957−0.034 vs MILLIE10 models combined: 0.967 (+0.007)shortcut-only floor 0.634 ± 0.031 for this protocolMatched0.869 ± 0.03410 seeds · range 0.826–0.917+0.009 vs DeepAPL10 models combined: 0.875 (+0.015)shortcut-only floor 0.608 for this protocolMatched within margin of error
CP-HRIC-S1 · multi-task model, decider-vision featuresthe CP-HRIC-S1 multi-task model on a vision-language model's cell embeddingsnot run0.851 ± 0.05010 seeds · range 0.785–0.958−0.009 vs DeepAPL10 models combined: 0.861 (+0.001)shortcut-only floor 0.608 for this protocolMatched
  • A model trained only on Johns Hopkins patients (our cell encoder with an attention-based patient model) scores 0.620 AUROC on the 24 held-out patients (mean of 10 seeds), close to the 0.608 that a shortcut-only model reaches on the same test from each patient's image sizes and background colour alone, so Johns Hopkins results may partly reflect image shortcuts rather than cell morphology until our shortcut controls run.

Beats = our mean is above the paper’s number by more than run-to-run noise (twice that model’s measured run-to-run spread on that test, divided by √seeds; judged for the two joint models only) · Matched within margin of error = above the paper, but by less than run-to-run noise · Matched = our mean ± the spread across training runs overlaps the paper’s reported range (or reaches its number when it gives no range) · Approaching = within 0.05 · In progress = further away.

Scores to three decimals; a delta is our figure minus the paper’s published figure on the paper’s own patients. Reference rows carry no delta and no status.

History

As we add data.

The declared model at each step, in the order the training data grew. The axis is the step and how much data the model learned from, not a date. Each bar is the mean over seeds where there are several.

AUROC (24 held-out)DeepAPL (0.86)
  1. Step 3 · 2 hospitalsMunich + Johns Hopkins, two questions (CP-HRIC-MIL · multi-task model, mean of 10 seeds)more data, several questions0.869
  2. Step 3 · 2 hospitalsMunich + Johns Hopkins, two questions (CP-HRIC-S1 · multi-task model, mean of 10 seeds)more data, several questions0.864

Hospitals = whose patients the patient-level model learned from: Munich only (never Johns Hopkins), Johns Hopkins only, or both. Every bar is scored on the same 24 held-out Johns Hopkins patients.

Data and reporting notes

  • Johns Hopkins data were cleaned before scoring: duplicated images, slide overviews and nested non-cell images were removed (prep v2). They had hit APL patients harder and inflated scores by about 0.04; every Johns Hopkins score here was re-run on the cleaned data.
  • How MILLIE reported vs how we report: MILLIE reports AUROC as the mean over its 3 cross-validation folds, 0.96 ± 0.02 (the spread across folds). Ours is the mean over seeds of that same 3-fold mean, ± the spread across seeds.
  • How DeepAPL reported vs how we report: DeepAPL reports one AUROC, 0.86, on its 24 held-out patients, with no spread. Ours is the mean over seeds of the AUROC on those same 24 patients. DeepAPL's model was not re-run; we compare with its published score.
  • MILLIE's often-quoted "0.94" has no stated split, so it is not used as a target.
  • The MILLIE paper counts 33 APL patients; the released metadata has 34. We keep all 106 patients; why the paper counts one fewer is not stated.
  • DinoBloom's pretraining data possibly includes Johns Hopkins data, because its documentation is ambiguous: the paper's list of 13 pretraining datasets (arXiv 2404.05022) omits it, but its GitHub README lists it in a general table of public datasets, and one pretraining set's source is unpublished.
  • Combined-model scores (several seeds' models combined into one) are not single-model scores; they are shown after the seed mean, never instead of it.
  • Read every Johns Hopkins score against its shortcut-only floor, shown beside it, not against 0.5: a model that sees no cell content (only each patient's image sizes and background colour) already reaches that AUROC on the same cells and the same test.

Read the original

  • MILLIEManescu et al. 2023, Scientific ReportsAUROC 0.96 ± 0.02 by random patient-level 3-fold cross-validationRead the original paper
  • DeepAPLSidhom et al. 2021, npj Precision OncologyAUROC 0.86 by a fixed split: 82 training patients, 24 held-out patientsRead the original paper