Analysed: the friends and family test
HSJ Local Briefing is our in-depth analysis of key issues facing the NHS or major health economies. This week we examine the friends and family test being piloted by NHS Midlands and East.
The friends and family test will be introduced nationally next April, having been piloted by NHS Midlands and East. It will see all hospitals ranked on the basis of whether patients would recommend them to a friend and family member.
Ministers believe adoption of the test will empower patients and lead to data which can be used to improve services. However, it is claimed some trusts are manipulating their scores and there are wider questions over the credibility of data.
Buy-in from the top of government and upper echelons of the NHS means the test will remain in place but an exercise in expectation management is necessary. It won’t be a silver bullet: expectations of what is deliverable must remain realistic. Greater standardisation of the process will lessen trusts’ ability to manipulate data but, if the test takes off as the prime minister envisages, the high stakes involved will only intensify the temptation to do so.
The concept behind friends and family
The friends and family test is, in theory, an incredibly simple concept which ministers hope will empower patients and improve services.
It is based on the net promoter score system used by commercial companies to rate customer satisfaction and asks patients if they would recommend a hospital service to a friend or relative.
Patients’ scores are amalgamated, generating an overall score for the service.
How the test works
The NPS customer satisfaction survey used by corporations the world over including Apple, Virgin and British Airways.
Customers are asked if they would recommend a service or product to others using an 11-point scale (0-10) and in most cases asked to provide their reasons for the score in a “free text” box.
Customers are then characterised as:
- Promoters (score 9-10) are loyal and will recommend you to others
- Passives (score 7-8) are largely satisfied but are not immune to competitors’ offers
- Detractors (0-6) are unhappy and are likely to bad mouth you
The overall score is the percentage of promoters minus detractors. Scores range from -100 (everyone is a detractor) to +100 (everyone is a promoter). Any positive score is regarded as good and any score of +50 is seen as excellent.
For more on the NPS, read NPS, Will it work in health? by Toby Knightley-Day, managing director of Fr3dom Health and author of the research used to support the Department of Health guidance for the test published last month.
Mr Knightley-Day emphasised to HSJ the importance of trusts understanding that the test must be seen as more than just a score. They must concentrate on the cultural and behavioural changes needed to drive the score, rather than focusing on the score itself, he said.
The friends and family test is built on this principle.
Under the national guidance which will become the requirement from April 2013 (although has not been introduced for the pilot), the process will be as follows:
Patients are asked: “How likely are you to recommend our ward/A&E department to friends and family if they needed similar care or treatment?”
They answer with one of six statements beginning with “extremely likely” and ending with “extremely unlikely”. Crucially, they can provide reasons for their answer in a free text box.
|Question scale||Score||Point scale|
|Extremely likely||Promoter||10 or 9|
|Likely||Passive||8 or 7|
|Likely nor unlikely||Detractor||0-6|
The statement will generate the NPS-style score but answers from the text box hold the key to the crucial answer of why they have scored as they have and what trusts can do about it.
The government’s enthusiasm for friends and family
The Department of Health wrote to trusts in October “strongly encouraging” all organisations to begin collecting data and reporting it from the “autumn” so they are ready for the national rollout next year.
From April 2013 it will be a contractual requirement on all providers of NHS-funded acute inpatient services and A&E departments.
However, trusts in the Midlands and East of England strategic health authority cluster have already been piloting the test.
Prime minister David Cameron said in May the test would allow the public to compare services and clearly identify the best performers in the eyes of patients.
It is also hoped the test will drive cultural and behavioral changes as staff become explicitly aware of how their actions impact upon patient experience, according to government guidance.
Managers will be able to pinpoint poorly performing wards and use patient feedback (the test offers patients an opportunity to explain the reasoning behind their awarding of their score) to assess why patients gave the ratings they did.
Ambitious supporters talk enthusiastically of building a “TripAdvisor for health”. This health version of the prominent website for the travel industry would entail hospitals, doctors, wards, services and more scored in real time by patients.
The test was also name-checked in the first NHS mandate published on 13 November. It said hospitals with good scores “will be financially rewarded”.
It is not clear if the financial “reward” will be under the existing Commissioning for Quality and Innovation (CQUIN) payments system or a different arrangement in the NHS standard contract between trusts and commissioners.
The case against friends and family
Clinicians and managers contacted by HSJ were divided as to the benefits of the test.
Critics told HSJ the new test was vulnerable to manipulation and “game playing” and will act to distract trusts from “their real purpose”.
They say patient feedback should be dealt with by the incumbent annual national patient survey, which they say is far more statistically robust.
Critics also fear parachuting in a system a private sector system will fail because customers choosing a holiday have a far greater array of choices than NHS hospital patients.
If the test delivers, the test will turn the rhetoric around empowering patients into reality and turbocharge service improvement through a cultural and behavioral revolution from the top to the bottom of the NHS.
If it fails, it will be just another burdensome data collection exercise (a fear of which ministers are acutely aware, according to sources) generating little more than league table with which trusts will be beaten around the head with.
Lessons from the Midlands and East
NHS Midlands and East has been piloting the project since May across its 46 acute trusts and is managing the national programme on behalf of the DH and NHS Commissioning Board.
The project has been largely driven by the cluster’s director of policy and strategy Stephen Dunn.
Out of nearly 200,000 inpatients treated up to September across the strategic health authority cluster, some 36,000 completed a friends and family test – around 18 per cent.
The results are compiled into monthly NPS league tables and published on the SHA’s strategic project team’s website.
Scores are only being published at trust level for the pilot, rather than ward level.
How trusts have fared so far
- Papworth Hospital Foundation Trust topped the most recently published table (October), based on September’s data, scoring 87. The Princess Alexandra Hospital Trust in Essex came bottom, scoring 35.
- Robert Jones and Agnes Hunt Orthopaedic Hospital Foundation Trust in Shropshire has been in the top five performing trusts for all five months.
- University Hospitals Coventry and Warwickshire Trust and Cambridge University Hospitals Foundation Trust have been in the bottom five of every table compiled to October.
- However, some trusts have seen their rating fluctuate markedly. Princess Alexandra Hospital Trust secured top five finishes in three of the first four months, plummeting to third bottom in the September table.
- Hinchingbrooke Health Care Trust hung out the bunting after topping the first table published but has not notched up a top five place since.
Overall, scores have improved robustly. The cluster region has increased its combined NPS score from 61 to 68, paving the way for it to achieve the 10-point improvement over 12 months targeted by the SHA.
Apples, pears and manipulation
Encouraging as they are, the results must be treated with caution for a number of reasons.
Senior managers in the East of England raised concerns to HSJ that scores were being manipulated and that the methodology was not providing robust results.
One senior figure, who spoke anonymously, told HSJ trusts were already “manipulating” scores and “game playing”.
He added: “I think the test is fundamentally flawed. The national patient survey is not game-able but this is. The national patient survey is statistically valid; this is producing results which are statistically incredible.”
The figure said the variety of scores across trusts was far wider than those in the national patient survey and that month-by-month variations were implausible.
The concerns arise in part from methodological issues.
Firstly, trusts are using different scales. Some are using the traditional 11-point NPS scale while others are using the six-point scale advocated in the national guidance.
The SHA cluster said both scales “map” onto the same classifications but the Royal Statistical Society questioned the validity of comparing results gathered on two different scales.
Peter Lynn, professor of survey methodology at the University of Essex and a leading figure in the society, told HSJ the number of points on a scale and whether or not there an explicit “don’t know” option was offered were factors known to influence respondents’ answers.
HSJ contacted the best and worst five performing trusts in September’s league table.
Four out of the bottom five (all except University Hospitals Coventry and Warwickshire Trust) are using the six-point scale while three out of the top five trusts are using the 11-point scale.
These results do not allow for a conclusive judgement to be made about the two scales but the use of two scales is certainly a complication the pilot could have done without.
Professor Lynn, along with some trusts in the region, also raised concerns about discrepancies in data collection methods.
National guidelines published last month explicitly state face-to-face interviews should not be used as a collection method due to response bias.
They also “strongly recommend” patients complete the survey “away from the care setting”.
The national guidelines do not officially come into force until next April and some trusts are already adhering to the standards.
But other trusts, including the Royal Orthodaedic Hospital Foundation Trust and James Paget University Hospitals Foundation Trust told HSJ they were currently carrying out face-to-face interviews in the care setting.
NHS Midlands and East director of customer service strategy Andrew MacPherson said data collection issues could indeed “distort” the results to some degree.
But he was adamant the data and resulting league tables were credible and dismissed the accusations of data manipulation as “anonymous sniping”.
He added: “The test is currently being audited as part of a planned regular review of the system which is only just six months old and ‘settling down’”.
The SHA’s rationale for allowing marked differences in collection methods and scales is understandable.
Mr MacPherson told HSJ a “permissive” approach was critical to get the project off the ground as quickly as possible.
He agreed credible data was critical but in order to install the culture change required it was important trusts were able to move as quickly as possible with the tools already at their disposal.
Both 11-point scales and face-to-face interviews will be banned from April but they certainly constitute a caveat to the pilot’s results.
Some chiefs buying into the test
Despite the collection issues, a number of participating trusts are buying into the concept and – crucially – reporting behavioural and cultural changes.
One trust chief executive said the score had created a “heightened awareness” among staff about patient experience and that the test would be a “lynchpin” performance measurement.
He said staff were beginning to realise that it is the whole patient experience that they are being judged on, not just the care.
A dirty toilet or another patient acting in an anti-social way is just as likely to influence the score as the care itself, he said.
Another chief called it the “real acid test for quality of care experience”.
A long way to go
Despite the progress, supporters of the test warned there was a considerable way to go before the test fulfilled the prime minister’s vision.
Neil Bacon, founder of provider iWantGreatCare which is offering the test free for other trusts through the NHS Confederation, said the shortfall was largely an issue of scale.
Trusts in the pilot are not collecting data from every ward or publishing scores for individual wards as they will be required to do from next April.
Trusts will also have to collect the data continuously. Some pilot trusts are collecting their data just one week a month at present.
But it is envisaged trusts will eventually be collecting data ward-by-ward, minute by minute and results will update on a website in real-time.
The lessons so far
- The test is not just about achieving a score, it is about changing culture
- The culture change is already under way
- The programme needs considerable finessing
- The wider hospital experience is key, this is not just about care
- Good governance will be essential to avoid “game playing”
- Commissioners, not just providers, have a potentially rich new data set
Buy-in from the top of government and the NHS’s upper echelons have secured the test time to bed in, despite the shortcomings in its implementation so far. A charm offensive on staff and an exercise in expectation management is necessary.
Accusations of game playing and manipulation will seriously undermine credibility and must be dealt with urgently.
A future in which patients select hospitals like they would a holiday on TripAdvisor works well on paper but it quickly runs into practical difficulties.
Firstly, there simply is not the range of choice in the health sector, there are only a finite number of hospitals, and secondly, the main users – older people and families with young children – tend to be unable to travel great distances to exercise choice.
But this does not necessarily detract from the test.
NHS Confederation chief executive Mike Farrar, an enthusiastic supporter of the test, said that rather than driving choice, patients were more likely to benefit by the behaviour change by the trusts that the test would inspire.
“My sense is people will want to complete the test in order to improve their local hospital,” he said.
“What they are not looking for, in my view, is to effectively create a situation where they are able to understand that 50 miles away there is somewhere else that they could go.”
It is critical the test is viewed as part of the wider agenda to empower patients by changing the culture throughout NHS organisations and developing a more collaborative working relationship with service users.
Despite warnings for trusts not to become fixated on their score, it seems inevitable that, to some degree, many will do precisely that, especially if (and when) the media seizes upon poor performers.
The pilot suggests the test could drive profound changes – if trusts buy in and prepare themselves for some bumps and scrapes on the way.
But it won’t be a silver bullet; expectations of what is deliverable must remain realistic and work must be done to make sure there is widespread support at all levels.
Greater standardisation of the process will lessen trusts’ ability to manipulate their data but if the test takes off as the prime minister envisages, the high stakes involved will only intensify the temptation to do so.
Robust and potentially costly policing of the process will be needed but the impression of another top-down imposition must also be avoided if trust are to be kept on side.
The prime minister and the test’s supporters face a stiff challenge but successful implementation could begin to justify the rhetoric around the desire to build a patient-centric NHS.