What clinical safety assurance does a healthcare AI product need?

Start with what the AI product is intended to do, for whom, and in which clinical pathway. An AI system that drafts a consultation note raises different questions from one that recommends treatment or prioritises patients. The word “AI” does not by itself determine whether a product is a medical device, whether DTAC applies, or which organisation holds a particular safety duty.

For software or AI that may be a medical device, use MHRA's software and AI guidance and seek a specialist classification opinion where needed. MHRA's intended-purpose guidance explains why the claimed purpose must be precise.

A practical assurance pathway

1. Describe the clinical decision and the handoff

Map where an AI output enters care: who sees it, what they are expected to do, and what happens when the output is wrong, absent or delayed. State the patient groups, settings and uses that were not assessed.

2. Analyse AI-specific failure modes

Consider plausible incorrect or invented output, omission of important information, automation bias, performance differences across patient groups, changes in input data, model or prompt changes, and failure to detect degradation. Translate each into a patient-harm scenario rather than leaving it as a generic “AI risk”.

3. Make human oversight operational

“A clinician checks the output” is only a control if the workflow gives them the source information, time, authority and clear action to disagree. Specify what is checked, before which action, and how disagreement or correction is recorded.

4. Link controls to evidence

For each material hazard, record the control owner, implementation and test or monitoring evidence. The manufacturer manages its product risks under applicable DCB0129 requirements. A deploying organisation assesses its local use under applicable DCB0160 requirements. Check the actual product and organisation roles.

5. Plan for change after deployment

Name the signals that will trigger review: incidents, user corrections, drift, a new model, altered prompts, new integrations or use in another population. Agree who may pause or withdraw the system when its safety assumptions stop holding.

An AI scribe illustrates the point

A scribe may produce fluent notes that omit a negation, attribute a statement to the wrong person or place sensitive material in an unsuitable workflow. The supplier must explain product behaviour and safeguards; the deploying service must decide how notes are checked, filed, corrected and monitored in its setting. NHS England has specific ambient scribing guidance, and its DTAC guidance notes that other standards may apply to ambient voice technologies. Our AI scribe readiness page covers deployment in more detail, and data protection needs separate attention under ICO AI guidance.

For a clinical risk review of an AI-enabled product or deployment, see Assurance AI's scope and indicative fees. We do not certify AI products or replace specialist medical-device, data-protection or model-validation work.