CP-HRIC · Experiment 001

Which genetic subtype?

From one patient's blood-smear cells, which of five answers is it: PML::RARA (APL), NPM1-mutated AML, CBFB::MYH11, RUNX1::RUNX1T1, or a healthy control?

CP-HRIC-MIL: BeatsCP-HRIC-S1: Matched
The green cp-hric-mil robot at a microscope beside the orange System One robot taking a sample from a cryo box

The experiment

Data, paper and protocol.

Data
TCIA AML-Cytomorphology (MLL Munich), Munich Leukemia Laboratory. 189 patients, 81,214 single-cell images. Licence CC BY 4.0.RUNX1::RUNX1T1 32control 60NPM1 36PML::RARA 24CBFB::MYH11 37
Paper
SCEMILA, Hehr et al. 2023, PLOS Digital Health: macro-F1 0.81 by the paper's own five patient-level folds.Reported as one pooled macro-F1 over the five folds, no spread.
Protocol
Five patient-level folds from the SCEMILA paper, shared by every run: each of the 189 patients is tested exactly once. Chance is 0.20.

At a glance

Our multi-task models, on the paper’s own test.

Each is one model trained on more data to answer several questions, scored on the paper’s own test and against the paper’s own number.

  • 0.841CP-HRIC-MIL · multi-task model · Munich · 189 patients · SCEMILA's folds · ± 0.019 · 10 seeds · +0.031 vs SCEMILABeats
  • 0.809CP-HRIC-S1 · multi-task model · Munich · 189 patients · SCEMILA's folds · ± 0.016 · 10 seeds · −0.001 vs SCEMILAMatched
Munich · 189 patients · SCEMILA's folds · macro-F1 several questions System One never trained here / reference N models combined SCEMILA (0.81)
  1. CP-HRIC-S1 · multi-task modelseveral questions · 10 seeds0.809−0.001
  2. CP-HRIC-MIL · multi-task modelseveral questions · 10 seeds0.841+0.031
  3. CP-HRIC-S1 · multi-task model, decider-vision featuresseveral questions · 10 seeds0.762−0.048
  4. DinoBloom-B + an attention readerreference · these patients' images were in its pretraining · 10 seeded runs (an earlier unseeded run scored 0.849) · 10 seeds0.834

Bar = mean across seeds · whisker = ± spread across seeds · diamond = the seeds’ models combined into one · axis starts at 0.5. Deltas vs SCEMILA.

Models

Every result, on every test.

Our models and the reference on every test in this experiment. The headline is the mean across seeds ± the spread across seeds, with the range; where the seeds’ models were also combined into one, that score follows. “Not run” rows stay visible until they are filled.

ModelMunich · 189 patients · SCEMILA's foldsmacro-F1 · SCEMILA 0.81
Trained on more data (Munich and Johns Hopkins patients), several questions at once
CP-HRIC-S1 · multi-task modelone System One model answering the genetic subtype AND APL vs other AML; Munich patients + 82 Johns Hopkins patients0.809 ± 0.01610 seeds · range 0.785–0.833−0.001 vs SCEMILAMatched
CP-HRIC-MIL · multi-task modelone model with a head per question, same training data (declared recipe f1-l34-lr1e-5)0.841 ± 0.01910 seeds · range 0.823–0.884+0.031 vs SCEMILABeats
CP-HRIC-S1 · multi-task model, decider-vision featuresthe CP-HRIC-S1 multi-task model on a vision-language model's cell embeddings instead of our encoder0.762 ± 0.04210 seeds · range 0.697–0.841−0.048 vs SCEMILAApproaching
Reference: a foundation model that has seen these patients
DinoBloom-B + an attention readerpublished foundation encoder, frozen, with an attention reader trained on SCEMILA's folds; its pretraining images include these test patients' cells0.834 ± 0.01310 seeds · range 0.809–0.854reference · these patients' images were in its pretraining · 10 seeded runs (an earlier unseeded run scored 0.849)

Beats = our mean is above the paper’s number by more than run-to-run noise (twice that model’s measured run-to-run spread on that test, divided by √seeds; judged for the two joint models only) · Matched within margin of error = above the paper, but by less than run-to-run noise · Matched = our mean ± the spread across training runs overlaps the paper’s reported range (or reaches its number when it gives no range) · Approaching = within 0.05 · In progress = further away.

Scores to three decimals; a delta is our figure minus the paper’s published figure on the paper’s own patients. Reference rows carry no delta and no status.

Per answer

How each answer is recognised.

F1 for each of the five answers, next to the paper’s own per-answer numbers. Multi-seed models show the mean over seeds.

F1 per answer CP-HRIC-S1 · multi-task model (mean over seeds) CP-HRIC-MIL · multi-task model (mean over seeds) SCEMILA
  1. RUNX1::RUNX1T1
    1. 0.734
    2. 0.813
  2. control
    1. 0.988
    2. 0.992
  3. NPM1
    1. 0.693
    2. 0.710
  4. PML::RARA
    1. 0.829
    2. 0.860
  5. CBFB::MYH11
    1. 0.804
    2. 0.808

Dashed mark = the paper’s own F1 for that answer · axis starts at 0.5 · multi-seed models show the mean over seeds.

ModelRUNX1::RUNX1T1controlNPM1PML::RARACBFB::MYH11
Published (SCEMILA)0.7501.0000.7500.8600.690
CP-HRIC-S1 · multi-task model (mean over seeds)0.734Approaching0.988Approaching0.693In progress0.829Approaching0.804Matched
CP-HRIC-MIL · multi-task model (mean over seeds)0.813Matched0.992Approaching0.710Approaching0.860Matched0.808Matched

Beats = our mean is above the paper’s number by more than run-to-run noise (twice that model’s measured run-to-run spread on that test, divided by √seeds; judged for the two joint models only) · Matched within margin of error = above the paper, but by less than run-to-run noise · Matched = our mean ± the spread across training runs overlaps the paper’s reported range (or reaches its number when it gives no range) · Approaching = within 0.05 · In progress = further away.

History

As we add data.

The declared model at each step, in the order the training data grew. The axis is the step and how much data the model learned from, not a date. Each bar is the mean over seeds where there are several.

macro-F1SCEMILA (0.81)
  1. Step 8 · 6 datasets+ Johns Hopkins patients, two questions (CP-HRIC-MIL · multi-task model, mean of 10 seeds)more data, several questions0.841
  2. Step 8 · 6 datasets+ Johns Hopkins patients, two questions (CP-HRIC-S1 · multi-task model, mean of 10 seeds)more data, several questions0.809

Datasets = the public datasets behind the model: those that trained our cell encoder, plus Johns Hopkins once the patient-level model also learned from its training patients.

Data and reporting notes

  • How SCEMILA reported vs how we report: SCEMILA reports one pooled macro-F1 (0.81) over its five patient-level folds, with no spread. Ours is the same pooled macro-F1 on the same folds; for a model trained with several seeds, the headline is the mean of that pooled score across seeds ± the spread across seeds.
  • Combined-model scores (several seeds' models combined into one by averaging each patient's probabilities) are not single-model scores; they are shown after the seed mean, never instead of it.
  • DinoBloom's pretraining included these test patients' images (TCIA AML Hehr, its 80% training split; confirmed), so it is shown as a reference only, with no delta.
  • DinoBloom's size makes little difference on this test: the same attention reader on each size of the published encoder, 10 seeded runs each, scores S (22M parameters) 0.842 ± 0.014, B (86M parameters) 0.834 ± 0.013, L (303M parameters) 0.830 ± 0.008, G (1.1B parameters) 0.835 ± 0.010 macro-F1. All four are references for the same pretraining reason; the site shows DinoBloom-B, the size the project declared.
  • Our cell encoder trained on every readable labelled image in its five public datasets. One image is left out because the file is damaged and cannot be decoded (MYB_05527.jpg, in the Bone Marrow Cytomorphology dataset); it is not counted in training or in the image totals shown here.
  • The encoder has so far learned 17 of the 21 cell classes in the BMC bone-marrow dataset.

Read the original