A reference dataset only sub-specialists can build
Ground truth for a regulatory submission is not an annotation task that scales by adding people. The client needed consistency across more than a thousand studies, which meant every reader had to clear the same certification, experience and annual reading-volume bar — and the workflow had to produce an auditable record of how each final label was reached.
Three things had to hold at once: independent assessment, full adjudication of disagreements, and a final classification informed by the supplied biopsy and radiology results.
ABR-certified, MQSA-qualified breast radiologists
Qualification requirements
Medcase recruited four specialized breast radiologists as truthers, each with five or more years of experience, and one senior breast radiologist as adjudicator with ten or more years. Every recruited reader was ABR-certified, MQSA-qualified, and interpreting 1,000 or more screening studies annually. A separate in-house radiologist performed quality assurance.
A second round with new readers
For the second round the client asked for a fresh pair of truthers. Medcase supplied them from the network while retaining the same adjudicator — preserving adjudication consistency across both rounds while giving the client the reader independence it wanted.
Onboarding and pilot validation
Readers received project materials and the annotation protocol prepared by Medcase and the client, followed by a live onboarding session covering project background, workflow requirements and a demonstration of the annotation platform.
Each round then ran a 30-study truther pilot. The client reviewed the output, gave feedback and confirmed validation before production began. The adjudicator completed a separate 30-study pilot in the first round; none was required in the second. Pilot cases sat outside production and were excluded from the production total.
Annotation and adjudication methodology
Two independent BI-RADS reads per study
Two truthers independently reviewed 100% of production studies in each round, assigning BI-RADS scores and identifying and characterizing breast lesions. Neither saw the other's work.
Adjudication of protocol-defined discrepancies
The same adjudicator independently reviewed every production study across both rounds, assigned BI-RADS scores and resolved discrepancies according to the protocol.
Final benign/malignant classification
In a separate task, the adjudicator reviewed the biopsy and radiology results supplied for each case and assigned the final benign or malignant label — keeping the outcome label distinct from the imaging assessment that preceded it.
Quality assurance on a 10% stratified sample
An in-house radiologist performed QA on 120 production studies, a stratified random sample of roughly 10% per round. The same sampled cases were checked across both truthers' outputs and the adjudicator's outputs after annotation and adjudication were complete.
QA assessed output completeness, protocol adherence, internal consistency, image handling and documentation integrity. All sampled cases passed, with no returns for correction. QA was procedural and operational; it did not involve clinical reinterpretation or modification of the adjudicated ground truth.
Study volume and timeline
| Work package | Round 1 | Round 2 |
|---|---|---|
| Production studies | 1,000 | 195 |
| Independent truthing reads | 2,000 | 390 |
| Truther pilot studies | 30 | 30 |
| Adjudicator pilot studies | 30 | Not required |
| Studies sampled for QA | 100 | 20 |
Round 1 ran February to March 2026. Round 2 ran June to July 2026 — annotation and adjudication completed 23 June, with the additional classification task assigned 17 July and completed 19 July.
Results: 1,195 studies, 2,390 independent reads
- A ground-truth dataset of 1,195 breast ultrasound production studies supporting AI development, validation and FDA 510(k) submission activities.
- 2,390 independent truthing reads — two radiologist assessments on every production study.
- Adjudicated BI-RADS scores and lesion annotations across all production studies.
- 1,195 final benign/malignant labels informed by the supplied biopsy and radiology results.
- Procedural QA on 120 production studies with no case returned for correction.