Digital front door technologies to gather service user information for NHS Talking Therapies for anxiety and depression assessments: early value assessment

3.1 Evidence gaps and ongoing studies

The external assessment group identified 4 ongoing or unpublished studies, 3 for Limbic Access and 1 for Wysa Digital Referral Assistant (DRA), that may address some of the evidence gaps.

3.2 Data sources

The NHS Talking Therapies: for anxiety and depression and Mental Health Services Data Set (MHSDS) are real-world datasets that could also be used to collect information about the impact that conditions have on mental health. Most of the data needed to address the evidence gaps is already collected within the Talking Therapies services, for example the:

New studies will be needed to collect data on measures that are more specific to using the technologies, such as the:

NICE's real-world evidence framework provides detailed guidance on assessing the suitability of a real-world data source to answer a specific research question. The quality and coverage of real-world data collections are of key importance when used in generating evidence. Active monitoring and follow up through a central coordinating point is an effective and viable approach of ensuring good-quality data with broad coverage.

3.3 Evidence collection plan

A suggested approach to addressing the evidence gaps is a mixed-methods longitudinal parallel cohort study. This approach would follow an intervention arm and a control arm, and compare their outcomes. This design would allow assessment of the clinical impact of the technologies and the resource use associated with their implementation. Qualitative data could be generated through appropriate methods such as surveys, focus groups or interviews, as highlighted in NICE's real-world evidence framework. This could include reported outcomes (acceptability, usability and preferences) from people using the service.

If technology-derived problem descriptors are being used to support clinical diagnoses, evidence around their accuracy should be generated for future assessment. Ideally, a cross-sectional diagnostic accuracy study would compare agreement between clinical assessor alone and clinical assessor aided by the technology-derived problem descriptors. It would use an internationally recognised diagnostic interview such as the Mini-International Neuropsychiatric Interview as the 'gold standard'. It would be possible to report accuracy (including sensitivity, specificity, and negative predictive values and positive predictive values).

Relevant data may already exist within published studies, which are outside of the current scope but may be useful for future assessments.

The studies should enrol a representative population, that is, people who would be offered a pre-assessment, including people who have self-referred and people referred through any other method. The pre-assessment may include web- or paper-based forms, or telephone pre-assessments. The studies should compare people using digital front door technologies for pre-assessments with a similar group having standard care. Eligibility for inclusion and the point of starting follow up should be clearly defined and consistent across comparison groups to avoid selection bias.

Data should be collected in all groups from the point at which a person would become eligible for standard care (referral). The data from both the intervention and comparison groups should be collected at appropriate time intervals. Data from a comparable population, but with no access to digital technologies for self-management, should form the comparison group. Ideally, the studies should be run across multiple centres, with the aim of recruiting centres that represent the variety of referral pathways in the NHS.

Despite consistent eligibility criteria, non-random assignment to interventions can lead to confounding bias, complicating interpretation of the treatment effect. So, approaches should be used that balance confounding factors across comparison groups, for example, using propensity score methods. To achieve this robustly, data collection will need to include prognostic factors related both to the intervention delivered and patient outcomes. These should be defined with input from clinical specialists. Incomplete records and demographically imbalanced groups can lead to bias if unaccounted for.

Data collection should follow a predefined protocol. Quality assurance processes should be put in place to ensure the integrity and consistency of data collection. See NICE's real-world evidence framework, which provides guidance on planning, doing and reporting real-world evidence studies. This document also provides best practice principles for robust design of real-world evidence when assessing comparative treatment effects using a prospective cohort study design.

Alternative methodological approaches may be applicable, for example, a stepped wedge clinical trial could be a pragmatic and comprehensive approach.

3.4 Data to be collected

3.5 Evidence generation period

This will be 3 years to allow for setting up, implementation, data collection, analysis and reporting.