Healthcare AI assurance
AI in clinical settings needs clinical safety thinking
An AI-enabled product introduces failure modes that conventional software assurance was not designed to catch. We assess them the way clinical risk has always been assessed: by working through what happens to a patient when the system is wrong.
Failure modes
What we look for in AI-enabled healthcare products
Intended use drift
The product is used for patients, settings or decisions it was never designed or evidenced for.
Automation bias
Clinicians defer to the model's output even when their own judgement should override it.
Silent degradation
Performance falls as populations, pathways or upstream data change, with nothing to detect it.
Unclear accountability
No named clinician owns the decision to deploy, monitor, escalate or withdraw the model.
Opaque outputs
Users cannot tell how confident the model is, or when it is operating outside its competence.
Inequity of performance
Accuracy varies across patient groups and no one has looked.
How we assess
Clinical risk analysis for AI-enabled products
- 1
Define intended use precisely
Population, setting, decision supported, and the boundaries beyond which the product must not be used.
- 2
Map the clinical workflow
Where the output enters the pathway, who acts on it, and what happens if it is wrong in each direction.
- 3
Analyse hazards and controls
Structured hazard identification with proportionate technical, procedural and training controls.
- 4
Design human oversight
Make oversight real: what the clinician sees, when they are expected to disagree, and how that is captured.
- 5
Plan ongoing monitoring
Performance monitoring, drift detection, incident routes and the trigger points for withdrawal.
A note on honesty
AI assurance is a developing field. Expectations from NHS organisations, regulators and professional bodies are still maturing and vary between settings.
We will not claim your product is compliant with a standard that does not yet exist, and we will tell you where guidance is genuinely unsettled rather than manufacture certainty.
Where your product may be a regulated medical device, take specialist regulatory advice.
Questions