Evidence generation plan
Closed for comments This consultation ended on at Request commenting lead permission
3 Approach to evidence generation
3.1 Evidence gaps and ongoing studies
Table 1 summarises the evidence gaps and ongoing studies that might address them. Information about evidence status is derived from the external assessment group's report; evidence not meeting the scope and inclusion criteria is not included. The table shows the evidence available to the committee when the guidance was published.
Clinical effectiveness | Resource and service impact | Engagement and acceptability | Uptake in different subgroups | |
|---|---|---|---|---|
Limited evidence | Limited evidence | Limited evidence | No evidence | |
D REACH HF | Limited evidence | Limited evidence | No evidence | No evidence Ongoing study |
Digital Heart Manual | No evidence | Limited evidence | Limited evidence | No evidence |
Gro Health HeartBuddy | Limited evidence Ongoing study | Limited evidence | Limited evidence | No evidence |
KiActiv | Limited evidence Ongoing study | Limited evidence | Limited evidence Ongoing study | No evidence |
myHeart | Limited evidence Ongoing study | Limited evidence | Limited evidence | No evidence |
3.2 Data sources
There are several data collections that have different strengths and weaknesses that could potentially support evidence generation. NICE's real-world evidence framework provides detailed guidance on assessing the suitability of a real-world data source to answer a specific research question.
The National Audit of Cardiac Rehabilitation (NACR) is the data source that is most likely to be able to collect the real-world data necessary to address the essential evidence gaps. The audit collects data to support the monitoring and improvement of cardiovascular prevention and rehabilitation services. Currently, data is collected at the start and end of a cardiac rehabilitation program. There are future plans to link patient-level data to other datasets such as the Hospital Episode Statistics and Office for National Statistics for collection of longer-term outcomes. Additional data collection is planned (September 2025) around the mode of delivery of cardiac rehabilitation as part of the audit.
Other useful sources of data are the National Institute for Cardiovascular Outcomes Research and the National Cardiac Audit Programme.
The quality and coverage of real-world data collections are of key importance when used in generating evidence. Active monitoring and follow up through a central coordinating point is an effective and viable approach of ensuring good-quality data with broad coverage.
Ongoing studies
There are 2 highly relevant, ongoing studies that may address some of the clinical effectiveness, resource and service impact evidence gaps. Both are due to end in 2025.
3.3 Evidence collection plan
The suggested approach to addressing the evidence gaps for the technologies is a real-world historical control study with propensity score methods. The study would compare outcomes before and after implementation of the technologies. Quantitative data for the historical control arm is likely to exist in the NACR dataset, which includes patient-level data such as components of cardiovascular risk profiles, exercise capacity, health-related quality of life, psychological wellbeing and nutrition. The dataset also details service-user characteristics such as age, sex, ethnicity, geographical location and employment status, which will enable subgroup analyses. These baseline cohort differences may affect clinical outcomes and should be corrected for in future analyses. An additional survey of specific digital technologies used to deliver cardiac rehab is planned for September 2025.
Qualitative data could be generated through appropriate methods such as surveys, focus groups or interviews, as highlighted in NICE's real world evidence framework. This could include reported outcomes (acceptability, usability and preferences) from people using the technologies.
Despite consistent eligibility criteria, non-random assignment to interventions can lead to confounding bias, complicating interpretation of the intervention effect. To minimise bias and identify a suitable control group, appropriate statistical approaches that balance confounding factors across comparison groups should be used, for example, propensity score matching. The comparator group of primary interest is cardiac rehabilitation face-to-face sessions, or a hybrid programme of in-person group-based and home-based programmes (including paper manuals, live online classes, home visits or telehealth) without digital cardiac rehabilitation technologies. NICE's real-world evidence framework provides further detailed guidance on the planning, conduct and reporting of real-world evidence studies assessing comparative effects.
3.4 Data to be collected
Study criteria
At recruitment, eligibility criteria for the suitability of using the digital technologies and inclusion in the real-world study should be reported. Detailed descriptions of the technologies should include their training needs, digital-safety assurance and the specific versions.
Service-user characteristics and clinical outcomes
Information about individual characteristics at baseline, for example, sex, age, ethnicity, first language, medicines, diagnosis, comorbidities, socioeconomic status, and location, with other important covariates chosen with input from clinical specialists. Characteristics should include those needed for adjustment to address confounding, and for subgroup analysis.
Measures recorded at baseline and follow up (at least 12 months, ideally 18 months), of:
exercise capacity (for example shuttle walk test)
cardiovascular risk profile (including blood pressure, weight, height, cholesterol)
psychological wellbeing (Patient Health Questionnaire 9 score, Generalised Anxiety Disorder Questionnaire 7 or Hospital Anxiety and Depression Scale)
health-related quality of life (EQ-5D or Dartmouth Coop questionnaire)
nutrition status (Mediterranean diet score tool)
medication adherence.
Time from post-discharge referral to start of core cardiac rehabilitation programme
Adverse events
Resource and system use
Time from post-discharge referral to start of core cardiac rehabilitation programme.
Number and cost of face-to-face cardiac rehabilitation sessions (and details about profession and banding of staff leading or supporting the sessions).
Referrals to other specialist services.
Number of appointments in primary, secondary and community care.
Costs of digital technologies for supporting cardiac rehabilitation, including:
licence fees
healthcare professional staff time and training costs to support the service
integration with digital NHS systems
implementation costs.
Other technology costs.
Engagement and acceptability
Usability and acceptability of the technologies.
Intervention adherence, uptake, completion and attrition rates (including reasons for not using the technology).
Data collection should follow a predefined protocol and quality assurance processes should be put in place to ensure the integrity and consistency of data collection. See NICE's real-world evidence framework, which provides guidance on the planning, conduct, and reporting of real-world evidence studies.
3.5 Evidence generation period
This will be 3 years to allow for setting up, implementing the test, data collection, analysis and reporting.
3.6 Following best practice in study methodology
Following best practice in conducting studies is paramount to ensuring the reliability and validity of the research findings. Adherence to rigorous guidelines and established standards is crucial for generating credible evidence that can ultimately improve patient care. The NICE real-world evidence framework details some key considerations.
In the context of an early value assessment, a key factor to consider as part of the informed consent process is making sure that patients (and their carers, as appropriate) understand that data will be collected to address the evidence gaps identified in section 2. Where applicable this should take account of NICEs guidance about shared decision making.
How are you taking part in this consultation?
You will not be able to change how you comment later.
You must be signed in to answer questions