Evidence standards framework for digital health technologies
The evidence standards framework (ESF) is a set of evidence standards for a wide range of digital health technologies (DHTs).
DHTs are developing at an increasing pace. We're working with stakeholders, system partners and thought leaders to develop standards that ensure these new technologies are clinically effective and offer value to the health and care system.
What is the evidence standards framework?
The ESF has been produced to promote more consistency in the evaluation of digital health technologies (DHTs) across the NHS.
The intended users are:
commissioners and evaluators in the health and care system making purchasing decisions; they can use the ESF to help them to decide whether to commission a DHT in their organisation
companies developing DHTs for use in the health and care system.
We've also produced a user guide that contains information on the purpose of the ESF along with how it can be used and by who. The guide contains some frequently asked questions relating to the ESF and explains many commonly used terms.
In this short video, Mark Salmon, Programme Director at NICE, describes why NICE has created the Evidence Standard Framework for Digital Health Technologies, how it can benefit developers of digital technology and why developers should be interested in evidence generation.
Who is the ESF for?
The ESF is for people in the health and care system who are responsible for:
identifying and evaluating new DHTs
conducting formal product reviews
authorising DHT product coverage, funding, or reimbursement
conducting ongoing clinical and economic product evaluations in real-world settings.
We refer to these people as 'evaluators' in the ESF.
The ESF is also for companies that develop or distribute DHT for use in the health and care system.
We refer to these as 'companies' in the ESF.
What does the standard help me to do?
For companies that develop or distribute, and for evaluators and decision makers in the health and care system, the ESF:
provides a classification system that helps to assign specific technologies to one of the tiers of standards within the framework
provides a set of standards that can be consistently used by developers
makes it easier to understand what good levels of evidence for digital health technologies look like.
For companies that develop or distribute DHTs, the ESF also:
helps to understand how to demonstrate the effectiveness and value of a DHT when engaging with evaluators and decision makers in the health and care system.
For evaluators and decision makers in the health and care system the ESF also:
helps to make more informed and consistent decisions when evaluating, commissioning or purchasing DHTs.
What the standard does not do
As a company, meeting the standard does not mean your DHT has been assessed or endorsed by NICE. The following web pages give advice on how we identify and assess medical technologies that include digital technologies:
NHS Innovation Service, a new service used to connect technology innovators to several public sector organisations within the UK health care system.
Find out more about how to get a medical technology evaluated.
Meeting the standard also does not mean your DHT has been given regulatory approval as a device or as a service.
The ESF is aligned to regulatory processes wherever possible. You can also approach the following organisations for guidance:
Medicines & Healthcare products Regulatory Agency (MHRA): responsible for regulating medical devices. They will be able to offer advice on submissions to their organisation.
Care Quality Commission (CQC): responsible for regulating services. They will be able to offer guidance on registration with their organisation.
How can I use the framework?
View the ESF to view all the standards.
Read the user guide to understand more about the ESF and how to use it.
View the ESF classification examples spreadsheet (Excel) to see the ESF tier that may apply to a range of technologies.
Use the budget impact tool (Excel) to estimate the costs and impacts to current practice that arise from using the DHT.
Who currently uses the framework?
The ESF is used locally and nationally. Several organisations have provided more information about how they use the framework.
Health Technology Wales (HTW) uses the ESF to assist with technology appraisal topic selection. It guides HTW’s view on whether a technology is mature enough to go forward for appraisal or not.
Decision making in this rapidly growing area is something everyone is grappling with. Up until now, we’ve had little support in this area. Health Technology Wales truly welcomes this update to the ESF from NICE.
Dr David JarromSenior health services researcher at Health Technology Wales
The Accelerated Access Collaborative within NHS England has incorporated the NICE evidence standards framework (ESF) for digital health technologies into their Artificial Intelligence (AI) in Health and Care Award.
Having an ESF that is regularly refreshed will help to ensure that the evidence we generate today will meet the system requirements of tomorrow.
Emma HughesSenior manager for MedTech and AI at the Accelerated Access Collaborative
We would like to hear from users who are using the ESF at [email protected]
How was this framework developed?
2019
We released the first version of the ESF in March 2019. It consisted of a set of standards that supported innovation and ensured an appropriate level of rigour and assurance for the health and care system.
2021
At the end of 2019, we surveyed people’s experiences of using the framework. Based on this, we published an updated framework in April 2021.
The feedback we received, and the changes made to the framework, are summarised in this feedback report (Word).
2022
We updated the framework to include artificial intelligence (AI) and data-driven technologies with adaptive algorithms. We aligned it with regulatory requirements and made it easier to use.
Following an open consultation, we published an updated framework in August 2022.
Read the consultation framework (Word) and our response (Word).
AI collaboration
The project has worked with an academic collaboration partner made up of academics from Imperial University, Birmingham University and the Turing Institute to identify standards for AI and data driven technology.
The Academic collaboration conducted a three-stage research programme:
a scoping literature review
a survey and interviews
a Delphi Consensus Process.
Details of how this research programme was run and the findings from each stage can be found in this final report (Word).
The authors of this work are:
Xiaoxuan Liu, Trystan MacDonald, Alastair K Denniston, Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK.
Viknesh Sounderajah, Hutan Ashrafian, Ara Darzi, Institute of Global Health Innovation, Imperial College London, London, UK.
Carolyn Ashurst, Adrian Weller, Chris Holmes, The Alan Turing Institute, London, UK.
This content is published with agreement from NHS AI lab.
Where can I find out more?
If you have any further questions, please contact us at [email protected]