Research commissioned by the Department of Health has raised doubts about use of the “friends and family” test as a single indicator of patient experience.

Providers across the NHS have been told by government to ask patients whether they would be happy for their friends and family to be treated by their service in a test intended to be used to compare trusts’ performance.

The measure, used by some major firms in the independent sector, also known as the net promoter score.

Research published by the Picker Institute Europe and Care Quality Commission involved testing six questions - one of which was the friends and family test - which could potentially be used for measuring “overarching” experience, with 85 patients.

It aimed to find out which questions were understood correctly and answered consistently, that could be used in a wide range of services and settings. 

Three slightly different versions of the net promoter score were tested. Researchers found various problems including that “interviewees objected to the term ‘recommend’ in all questions in all of the testing rounds, and, most unusually, some were so annoyed by the idea of ‘recommending’ a hospital or mental health service that they refused to answer despite the interviewer’s presence”.

Another was that, “interviewees often did not understand that they were being asked about the likelihood of recommending a specific health service and provider, instead, they understood the question to be about recommending ‘the NHS’ (care that is free at the point of delivery) or recommending that other people should get treatment if they had a similar condition”.

Another problem was that “some interviewees gave high scores despite describing very poor experiences to the interviewer, while others gave ‘passive’ scores despite describing very good experiences”.

Researchers found one question which worked well, in which patients were asked to score their overall experience of care from “0 (I had a very poor experience) to 10 (I had a very good experience)”.

Picker chief executive Penny Woods said in a statement: “This work underlines the importance of using robust, tested questions for patient surveys.  Asking the right questions can give us invaluable insight into people’s experiences of healthcare, but asking the wrong questions risks collecting data that is misleading or invalid.”

Responding to the research, Jocelyn Cornwell, leader of the King’s Fund work on patient experience, said: “This new research by the Picker Institute casts fresh doubt on the usefulness of the net promoter score. Although the NPS is intuitively appealing, the research highlights a number of problems with it, chiefly that patients don’t understand it consistently. “

A DH spokeswoman said: “The friends and family test will definitely happen in all hospitals from April 2013. We want the public to be able to compare hospitals. To enable comparability we expect to have to ask hospitals to ask the same question and use similar methods.

“We will work with the NHS over the coming months to develop the most appropriate question and method, building on the learning already available in many parts of the NHS. It is important that the question we use is meaningful for patients, can discriminate reliably between good and poor services and drive improvements.

“This report provides valuable feedback on the overarching questions that were tested, and we are now working with key stakeholders to develop a standardised question that can be used by the NHS.”