The methodology of the friends and family test is flawed and NHS trusts need support to improve patients’ experience, not more meaningless and misleading data, argues Rachel Reeves

From April 2013, the government requires all NHS inpatients and accident and emergency attendees to be asked, within 48 hours of discharge, “How likely are you to recommend our ward/A&E department to friends and family if they needed similar care or treatment?” 

‘Governments’ preferences for high profile quick fixes are behind the lack of progress in patient-centred care’

Already, NHS trusts are concentrating their energies on complying with the friends and family test and appointing private survey contractors. Unfortunately, the data generated in this way will be so unreliable they will do more harm than good. They will reduce the impact of more valid measures of patient experience and divert resources away from where they are most needed.

This policy will be implemented across the NHS without scientific evidence to support it. The methodology imitates the “net promoter” measure of customer loyalty enthusiastically advanced by business guru Fred Reichheld.

Puzzling patients

However, there are no published scientific trials demonstrating its validity, nor is there evidence for its efficacy in raising the quality of services, and the test has many criticsA pilot of friends and family is being conducted in NHS Midlands and East, but there are no plans to wait and see if it improves standards of care before it is launched nationally.

‘Lack of rigour in survey methods provides many opportunities for questioning and manipulating results’

Man looking confused

Many NHS patients are likely to be puzzled by the wording of the question, which assumes consumer choice. The NHS context is rather different from selecting a car rental firm or a restaurant. 

Where patient choice of provider exists for planned inpatient admissions, it is primarily driven by their GP’s recommendations, availability, proximity and urgency, not by friends and family members’ recommendations. For A&E attenders and emergency inpatient admissions, choice is almost always limited to the nearest hospital.    

The implementation guidance says “the friends and family test is… a quick, consistent, standardised patient experience indicator”. Is it? In practice, the guidance allows NHS trusts the options of conducting the surveys by post, text message, online, telephone, booths at hospital exits or a combination of approaches, while expecting only a 15 per cent response rate.

It will be impossible to tell whether differences among trusts are real, or the result of methodological difference or biased responder groups. There is no such thing as statistical adjustment that could remove biases from data with so many variations and permutations. 

A further concern is the requirement that 100 per cent of patients should be invited to take part. It is a common but mistaken belief that larger samples improve validity, regardless of response rates.

Unreliable data

Much more reliable, valid and representative results could be obtained at a fraction of the cost by taking a smaller, properly selected sample of patients and giving each one time and encouragement to respond. 

On the other hand, if the 100 per cent requirement is supposed to give patients and the public the impression of having a voice, they might be disappointed that their voices have no real chance of being heard above the din.

For a measure to have any impact, it must be trusted and robust. If it is to form the basis of financial rewards, it must conform to the highest possible standards. If trusts or wards achieve poor scores on frisends and family they will have strong grounds for arguing that data collection biases were the cause. 

The lack of rigour in survey methods provides many opportunities for questioning and manipulating results and those temptations will increase if financial rewards are attached to them.

‘NHS trusts and commissioners already have more data than they can handle and they don’t need more’

A modified version of Commissioning for Quality and Innovation (CQUIN), using the reliable data generated by the annual Care Quality Commission national inpatient surveys, would be a much better place to start looking for a measure of patient experience by which to drive improvements through financial incentives.  

Information overload

At first glance, “real time” patient feedback is very appealing, with the potential to identify problems quickly and tackle them without delay. However, the idea assumes poor NHS care pops up in isolated pockets and that remedies can be devised and implemented with immediate effect. 

The evidence from the annual CQC surveys tells a different story, showing that poor standards of care usually persist in NHS trusts for many years. Those with experience of acting on patient feedback know that engaging staff and designing and implementing effective changes takes months or years, not days or weeks. 

Arguably, successive governments’ preferences for high profile quick fixes are behind the lack of progress in achieving patient-centred NHS care.  

The lack of improvement in NHS patients’ experiences is not down to a lack of data. NHS trusts and commissioners already have more data than they can handle and they don’t need more meaningless or misleading figures. 

What they do need is support and encouragement to make full use of the valid patient experience data they already have, and scientifically grounded information on what they can do to improve patients’ experiences.

Dr Rachel Reeves is principal research fellow at the School of Health and Social Care, University of Greenwich