Measuring the effectiveness of care innovations is a tricky business, but Tim Burdsey believes it can be done. He shares some insights from a recent review of the evidence base
A lot has been written in recent years about opportunities for new technologies to enable so-called “channel-shift” in the provision of health and care services. But what does the available evidence tell us about which technology solutions are most effective, and about where providers and commissioners should prioritise local investment?
To provide services that meet local needs and expectations and that address anticipated demand, we need to think differently about how we deliver care. New technologies are an important part of the answer; however, service providers and commissioners often don’t know which technology solutions are best placed to realise particular kinds of benefit.
Let’s talk about TECS
In December 2016, NHS England’s New Care Models programme approached the HIN to review the evidence for technology-enabled care services (TECS) to provide colleagues with information to make effective decisions about TECS to support the development of new models of care. The programme was established in 2015 in response to the Five Year Forward View. It aims to build sustainable health and care models which respond to the “triple gap” – health and wellbeing, quality and care, and cost. The programme is built on four guiding principles of clinical engagement, patient involvement, local ownership and national support.
“The evidence just isn’t there,” people warned us. “There’s plenty of great stuff, but it’s not in peer-reviewed journals”
From the start, we were cautioned that the review would be a tricky undertaking. “The evidence just isn’t there,” people warned us. “There’s plenty of great stuff, but it’s not in peer-reviewed journals.” To some extent, this has proven true. However, we wanted to find some useful insights that would assist planning and decision-making, and so decided to adopt a pragmatic approach to the evidence review and press on as best we could.
The first challenge we faced is that the concept of TECS is very broad. It covers everything from assistive equipment in the patient’s home (“telecare”), to remote diagnostic equipment (“telehealth”), to the provision of psychological therapies “at a distance” (“telecoaching”), to digital health apps on smartphones or tablets. To make matters more complicated, we found that sometimes, some of the concepts above are used interchangeably, or a term that relates to a specific type of technology is used as an umbrella term to refer to the field of TECS as a whole. This required us to exercise due rigour when devising the search strategy that would inform our interrogation of the evidence.
The second challenge we faced was the sheer size of the evidence base. An initial search yielded more than 10,000 primary studies, which meant that it was not going to be possible for us to analyse everything within the scope of the project. We therefore decided to focus on systematic reviews and other meta-analyses listed in the Cochrane and DARE databases, of which there were 411 – which made our analytic endeavour manageable.
So, what did the evidence reveal?
1. There is limited evidence of the effectiveness of TECS, and information on cost-effectiveness is particularly scant
The studies covered in a single systematic review can be diverse, which can make it difficult for the authors to draw firm, cohesive conclusions. On the occasions when we drilled down into the detail of primary studies, we found issues with poor study design, lack of relevance, or simply findings that were equivocal. We hope this will be addressed as new, more rigorous studies are developed and published in this area.
2. The evidence base hasn’t yet caught up with the pace of technology innovation
Many new and emerging technologies are felt to have the potential to revolutionise healthcare delivery: virtual/augmented reality (VR/AR), robotics, artificial intelligence (AI), “Big Data”-based analytics, the Internet of Things (IoT), to name but a few. While compelling, these are at the preliminary stages of spread and adoption – indeed some are at an even earlier stage in the innovation pathway than that. As a result, such technologies are conspicuously absent from the evidence base. It will be for future reviews to explore their impacts, and to make recommendations about whether they are worthy of consideration and investment at that stage.
While stakeholders were generally aware of, and excited by, some of these developments, their attention was focused on understanding the potential impact of more “mainstream”, currently available technologies. SMS text messaging, for example, is so ubiquitous that most would consider it unremarkable – and its applications well-understood. However, there is a sense that we haven’t fully appreciated its potential to support health-related behaviour change, and so stakeholders were keen that we examined the evidence for its effectiveness.
Stakeholders indicated five areas of technology delivery that they are interested in. The evidence of effectiveness for each of these areas is as follows:
- SMS text messaging: Helpful in supporting adherence to medication; enabling smoking cessation (at least in the short-term) and substance misuse interventions; and encouraging glycaemic control in diabetes management. However, must be tailored for the individual, and is most effective when it is delivered in the context of a proven behaviour-change framework.
- Telemonitoring: Effective for people with diabetes or heart failure. Telemonitoring is frequently undertaken in conjunction with educational interventions and in the context of enhanced relationships with medical professionals. It is unclear precisely which of these aspects is responsible for the impacts observed.
- Video consultation: Felt to be at least as effective as face-to-face interventions. Effective in treating mental illness through consultation, short-term support, and counselling.
- Web-based interventions: Effective in reducing anxiety symptoms. Again, personalisation and interactivity are important, as is the need to combine online interventions within other support measures.
- Mobile digital health apps: we found only one study covering smartphone apps – a situation that will surely improve in future. Apps can increase adherence to diet monitoring, and enhance compliance with treatment instructions among patients undergoing routine cardiac procedures.
Findings have informed the development of a benefits realisation model, produced by mHabitat and the York Health Economics Consortium. This will provide practical assistance to those seeking to understand the benefits of a particular TECS intervention.
For more information and to view or download the final report from the project, and to provide feedback on the recommendations, please visit https://healthinnovationnetwork.com/projects/tecs/.