The evaluation task

Physicians compared two AI-generated responses to the same health query, side by side, and judged which was better personalized for the user. Each evaluation required a preference rating and a written clinical justification following fixed formatting guidelines.

Three hundred evaluations were split across five physicians, between 50 and 75 each, against a time budget of ten minutes per evaluation.

At volume, quality fails quietly

Three hundred evaluations is not a large dataset. It is, however, large enough that reviewing every row by hand is slow — and small defects hide well. Empty cells, formatting that breaks the client's analysis, comments repeated across rows, justifications that contradict the rating they sit beside. None of these announce themselves. They surface when the client opens the file.

Two further problems sat upstream. Some source rows were incomplete or fell outside clinical scope, and those needed catching early rather than leaving physicians to guess. And each clinician needed specific, actionable feedback on their own output without that feedback becoming a bottleneck. 

Quality assurance built into the workflow

A live progress and quality tracker

Medcase built a tracker on top of the rating sheet, linked by formulas, showing each clinician's completion rate, open quality flags and current status in real time. Quality was visible while the work was happening rather than assessed after it finished.

Automated checks on every row

Using tooling approved by the client, the tracker checked all 300 rows for empty required fields, formatting errors, typos, repeated text, and inconsistencies between a rating and the comment justifying it. Every issue was logged with its row, field, severity and the specific fix required.

This is the difference between sampling and checking. A 10% sample tells you roughly how good the dataset is; checking every row tells you which rows to fix.

Individual feedback and time auditing

Personalized feedback, reviewed before sending

Automation built into the sheet drafted a progress and quality email for each clinician listing only their own open items. The project manager reviewed every draft before it went out, with the Medcase team mailbox copied — a human checkpoint that standard mail-merge tooling cannot provide.

Hours audited against budget

Each clinician's logged hours and average handling time were compared against the ten-minute benchmark throughout, so drift from budget was visible in week one rather than at invoice.

Flagging data issues back upstream

Where the problem was in the source data rather than the clinical work, Medcase sent it back. One AI response was missing from the source set; three queries fell outside clinical scope. Both were raised during the project and fixed at source, rather than producing evaluations that would have had to be discarded later.

Results

• 300 of 300 evaluations completed — full completion across five physicians.
  
  
• 46.7 hours logged against 50 estimated, roughly 7% under budget, at an average 9.3 minutes per evaluation against the 10-minute benchmark.
  
  
• Every blocking quality issue resolved before handover. The only outstanding flag at delivery was optional copyediting on 11 comments.
  
  
• Client-side data issues identified during the project rather than after delivery.
  
  
• Analysis-ready data delivered on time, in the format the client's team had specified.

Adopted by the client

The client's project lead asked Medcase to present the QA methodology in a dedicated section of their next Monthly Business Review, so their own team could bring the approach in-house. The same method has since been applied to further evaluation programmes for the client.

For clinical AI evaluation the value is not only the expert judgment. It is whether that judgment arrives clean, complete and usable — which is a question about the delivery system, not the clinicians.