SCCT Level III–equivalent readers, verified before production
Primary readers and adjudicators
Seven primary readers: US board-certified cardiologists and fellowship-trained cardiothoracic radiologists, each with five or more years of experience and qualifications equivalent to SCCT Level III. Two adjudicators: US board-certified radiologists with equivalent qualifications and ten or more years of experience.
Every primary reader's and adjudicator's credentials were verified against the client's SCCT Level III criteria before they started.
Pre-label annotators
Seven licensed radiologists based in Europe, Latin America and India, each with experience reviewing at least 100 cardiac CT studies. Their role is described below. Final annotations remained the responsibility of the US board-certified primary readers.
Onboarding and training the
client signed off
Clinicians received the project annotation protocol and training materials, followed by a joint live onboarding session delivered by Medcase and the client, covering clinical background, the annotation protocol and a demonstration of the annotation platform.
Every primary reader, adjudicator and pre-label annotator then completed five training studies. The client reviewed each one, gave personalized written feedback to each clinician, and assigned additional studies or corrections where needed before production began. Adjudicators and pre-label annotators had separate training sessions for their roles. Training studies sat outside production and were excluded from the production total.
Two independent reads,
adjudicated range by range
What each reader marked
For every coronary vessel, two primary readers independently marked the proximal start and distal end of the evaluable vessel, segmented the lumen, segmented calcified and non-calcified plaque, marked lesion ranges and the point of maximum narrowing, and flagged non-evaluable ranges following protocol rules for total occlusions, calcium blooming and bifurcations.
An automated consistency check
Before adjudication, an automated check tested each annotation for a continuous lumen and for plaque marked only inside a lesion range. Non-compliant annotations went back to the reader.
Stage 1: ranges annotated by one reader
Ranges marked by only one of the two readers were reviewed and then accepted or rejected on validity and contour accuracy.
Stage 2: significant disagreements
Ranges where the two readers' segmentations differed significantly were adjudicated slice by slice, with the adjudicator selecting one reader's annotation or rejecting both. Where the readers did not differ significantly, one of the two annotations was selected at random.
A pre-label stage that protected the submission date
Partway through production, full manual contouring was putting the timeline at risk. Daily output from the primary readers was falling, and adding readers at SCCT Level III is not something that can be done quickly.
Medcase added a pre-label stage as a separate project in the annotation platform. Within about two weeks of the decision, seven licensed radiologists outside the US were recruited, trained and onboarded through the same validation as the primary readers: a dedicated training session, five training studies, and client review with personalized written feedback.
Pre-label annotators produced the initial lumen and plaque contours for each vessel, including the proximal start and distal end of the segmentation and exclusions for severe artifacts. Each vessel received two independent pre-labels, one for each of its two primary readers, so the two reads stayed independent. Primary readers then reviewed and corrected the contours and added the lesion ranges and maximum-narrowing markers. The client reviewed each new pre-label annotator's first five production studies.
120 of the 150 studies went through the pre-label stage. Reader time per reading task fell from roughly 30 minutes to roughly 15, and all reading tasks were completed by 28 October. The qualification bar did not move: the final annotations remained the responsibility of the US board-certified primary readers.
Quality assurance
All quality review was performed by the client, which reviewed every training study and sampled production studies, provided written per-reader feedback, and returned identified issues to readers for rework. The automated consistency check acted as a further gate before adjudication.
Project volume and timeline
Work package | Count |
|---|---|
Production studies | 150 |
Coronary vessels annotated (2 per study) | 300 |
Independent reading tasks (2 readers per vessel) | 600 |
Studies routed through the pre-label stage | 120 |
Adjudicated vessel series | 286 |
Training studies per clinician, reviewed by the client | 5 |
Phase | Dates (2025) |
|---|---|
Project kickoff | 30 June |
Reader onboarding and training | 7–28 August |
Production reading | Late August – 28 October |
Pre-label stage | 8–23 October |
Adjudication | Late September – 10 November |
Additional readers were onboarded during production to increase capacity. Production reading to final delivery took about eleven weeks.
Results: 150 studies, 600 independent reads
- An adjudicated ground-truth dataset of 150 coronary CTA studies — 300 coronary vessels — supporting validation of an AI algorithm for coronary plaque quantification and FDA submission activities.
- 600 independent reading tasks, two expert segmentations on every vessel, completed with approximately 375 hours of primary-reader time on production studies.
- Adjudicated lumen and plaque segmentations, lesion ranges and non-evaluable ranges across all production studies.
- A pre-label workflow that roughly halved reader time per reading task without changing the qualification requirements for the primary readers.
- The client's coronary analysis software has since received FDA clearance, which the client attributes substantially to these lumen and plaque annotations.