The friends and family test works best if we forget about scores

In August 2013, I wrote on hsj.co.uk that the friends and family test had real potential as a service improvement tool, but that changes were needed to manage the tensions inherent in a system that was at once a local quality improvement intervention and national data collection.

• Friends and family test ‘unreliable’ comparison tool, in-depth study concludes

Nine months on, we await the imminent publication of NHS England’s review of the controversial programme. But with the first year completed and over 2 million responses collected, it’s worth reflecting on the practical experience so far.

In a forthcoming paper, due to be published in the Journal of Health Services Research and Policy, colleagues and I will show that test scores are strongly related to patient demographics and to the way data is collected.

This raises a fundamental question about the use of the test for measuring performance.  Because the test doesn’t have a standardised methodology, trusts are free to choose how they collect data.  This decision will (partly) determine their results because data collected via contrasting methods are simply not comparable.

Beyond the theory

Our analysis was a statistical one: we looked at data from individual respondents to understand differences within a number of trusts. But we wanted to know whether these differences were apparent in the real world, too.

‘Staff notice when they see fresh feedback from their own patients describing their experiences in their own words’

We collated published data from the first year of the test, and we found striking results, especially in accident and emergency settings, where collecting large numbers of responses proved the most challenging. Some of these examples, which are just two of many, have already been published by HSJ.

To be clear, there’s no suggestion whatsoever that these trusts have failed to follow official guidance. What’s apparent is that methodological variation in the collection of the data isn’t just a theoretical problem: trusts are exercising the flexibility that the test affords them, and the results obtained are influenced by this.   

When the friends and family test was first announced, two years ago next week, its stated aim was to “improve patient care… identify the best performing hospitals in England [and] allow the public to compare healthcare services”.

Fixated on scores

Our findings suggest that the “simple, headline metric” used for test does not provide a reliable basis for comparing services or identifying those performing best. But can it be used to improve patient care?

From talking to trusts, the answer is yes: used appropriately, the test can be a valuable improvement tool. But it’s not scores that count. Indeed, neither patients nor staff understand them and they can undermine morale.

Worse, there are cases where frontline staff fixate on scores and response rates so much they miss the real detail – the comments patients write alongside their ratings. 

Collecting comments from patients is not a mandatory requirement of the test but most trusts are doing it at least to some degree. The majority of trusts I have spoken to agree that the comments are the most useful information collected. 

The best results seem to be produced by collecting and returning comments to wards on a weekly basis, then using these as part of regular discussions about quality. Staff notice when they see fresh feedback from their own patients describing their experiences in their own words.  Where comments are reported in this way, it tends to be effective in creating a sense of patient experience having “moved up the agenda”.

Frontline focus

A couple of things stand out about the test comments. First, the vast majority are praise. One fairly typical trust I spoke with recently told me that over 95 per cent of the comments they receive are positive.

For staff accustomed to bad news about the NHS this can be revelatory – not to mention a significant morale booster. Second, where comments are more critical they can be really specific: things that make sense on one ward only while inviting improvement. The changes arising from these can have a particular focus: changing disabled bathroom mirror heights for example, or improving signs to help patients navigate hospitals. Small things, maybe, but things that can make an immediate difference for patients.

‘The test does not work as a performance measure or a comparative tool’

So what next for friends and family test? Our evidence shows that it does not work as a performance measure or a comparative tool. However, a range of existing national surveys already accomplishes these aims. Our experience is that test can be useful for improving services and sparking discussion locally. 

Even in its first year the test has already evolved. The way results are reported on NHS Choices has been softened and how the test is described has changed. 

To fulfil its potential its strengths must be further accentuated. It must be freed from trying to do the things it can’t. In short, we must separate the data collection from the intervention: forget about comparisons and scores, and let the frontline focus on comments and improving services. 

Chris Graham is the director of research and policy at Picker Institute Europe

Declaration of interest: Mr Graham is employed by the Picker Institute. The Picker Institute is an independent not for profit organisation but works with NHS organisations to collect and use data from the friends and family test and other patient experience measures, including the NHS national patient survey programme.