Benchmarking out of hours healthcare

When out of hours services hit the headlines it rarely means good news for the NHS. The 2004 contract that allowed GPs to opt out of providing 24-hour care provoked widespread criticism and last year medical insurers warned complaints about out of hours services had soared since it came into effect.

Then there are the individual stories. A 41-year-old woman died of multiple organ failure after consulting eight doctors working for a London out of hours service; an 84-year-old woman in South Wales with chest pains waited two hours for a call back and had a heart attack.

No matter how good healthcare is from 8am-6.30pm on weekdays, a poor service when surgeries are shut has the potential to destroy public confidence in the NHS.

"Out of hours services have frequently taken a battering, often after catastrophic failings with individuals," admits NHS Alliance out of hours lead Rick Stern.

It does not help that it remains unclear how many out of hours providers there actually are, nor that each service is different in locality, demand and set-up.

It is estimated there are about 100 providers operating in the sector. Some primary care trusts run services in-house. Others employ a private contractor to run a service. Still more out of hours services are run as social enterprises, often the result of an evolved GP-run co-operative.

And while in some PCT patches all GP practices have signed up to a PCT-contracted service, in others some practices still go it alone.

The mix of call-handlers, nurses and GPs operating a service also varies. Some providers use NHS Direct as the first point of contact; others employ their own staff. Others provide a different mix of services, including daytime support. Combine this with software and data often adapted to local need (rather than providing data that can be compared nationally) and it is notoriously difficult to check whether services are meeting the 13 national performance requirements for out of hours.

Damning figures obtained by the Liberal Democrats last year claimed that in nearly half of PCT areas providers were not answering the phone quickly enough - and almost a quarter were struggling to send details of consultations on to the patient's own GP the next day.

And there has been little hard evidence to help services answer their critics. The most recent attempt, a National Audit Office report in 2006, was bedevilled by relying on data submitted by PCTs and providers themselves, all of which used IT systems adjusted to their own needs. After the report was published its authors had to put out a swift correction admitting that the service they had rated as the most cost-effective in rural areas was not actually the best.

Tooled up

Could a new way to measure the performance of different services across the country be the answer to a commissioner's prayers?

The Primary Care Foundation is a primary and urgent care consultancy commissioned by the Department of Health to benchmark out of hours providers' performance.

"The benchmark allows PCTs to offer evidence of how they are performing on the basis of the overall experience - the eight million people who use out of hours services every year - rather than important, but exceptional, service failures," says Mr Stern, who is a Primary Care Foundation partner.

The DH is backing the tool. "At last there is a real hope that we will soon be able to accurately compare services across all out of hours providers and drive up the quality of care for patients," says national primary care director David Colin-Thome.

What the benchmark has revealed is a startling variation in performance around the UK.

Results from almost 30 anonymised pilot sites, seen exclusively by HSJ, reveal striking differences in costs and the way patients are being treated.

The cost per call to a service ranges from less than£30 to a massive£180. The cost per head of population varies from around£7 to more than£16.

One provider treats nearly 70 per cent of calls it receives as "urgent" but in less than 50 per cent of cases fails to start a definitive clinical assessment within the 20-minute response time demanded by the national requirements for out of hours care. Another provider put just 3 per cent of calls in the urgent category. The latter "would worry me", comments Mr Stern. "It seems an incredibly low figure."

Robust information is needed to raise service quality, says Dr Colin-Thome. The benchmark will allow providers and commissioners to perform regular audits on their services. "What we want is to be continually updating so the PCT and its provider can look at their performance. It is about sustainable improvement rather than one-off checking," he says.

The number of calls handled by each doctor or nurse is another area with wide variations. Previous attempts to compare these figures have been undermined by averaging out all responses, but the foundation's benchmark singles out weekend mornings as the time of peak demand.

The results show that while one service was handling more than nine calls an hour per clinician, those at the other extreme were taking less than one. "What we see is enormous variation in quality," says Primary Care Foundation partner Henry Clay. There are expensive providers offering poor patient care and far too many referrals to hospital, he adds, while some cheaper services do much better.

"In some areas, you can have confidence that they would recognise if it was an urgent case - in others I very much doubt it."

Some differences are inevitable, says Dr Mark Reynolds, medical director of out of hours provider South East Health. "Productivity is a very difficult thing to measure."

Levels of demand and the availability of other urgent or emergency care services also vary. People in rural areas cannot always just head for the nearest accident and emergency department or 24-hour chemist, for instance.

Dr Colin-Thome agrees: "You would expect that there would be some variation. The task of NHS managers and clinicians is to iron out unwarranted variation."

Knock-on effects

The figures suggest part of that task might be to look closely at the level of referrals to hospital by provider. The benchmarking data shows this ranges from just over 5 per cent of all handled cases to 20 per cent - suggesting the extremes might be under-treating some patients or conversely being far too cautious, with knock-on effects for the rest of the NHS.

There were also significant differences in the rates of home visits, the number of patients being seen at out of hours centres run by providers and the number being referred to other services. One organisation was seeing less than 6 per cent of patients in their own home while at the other end of the scale another was seeing more than 26 per cent at home.

Mr Clay says it is not just that some areas are performing much better than others. There is widespread confusion about how and what out of hours services are doing.

"Even on some of the standard performance measures, in 30-50 per cent of cases the figures are not what the PCT or provider thought. People are not very good at reading the [applicable] standards."

For example, says Mr Clay, many services failed to understand that the 20-minute deadline for telephone assessment of urgent cases is a deadline for the definitive clinical assessment, not merely to speak to a clinician. The clock keeps ticking if a nurse handling the call passes the patient on to a doctor.

The Primary Care Foundation is convinced its findings are robust, chiefly because it adjusts for the way different services set up their IT systems and extracts data directly from those systems, rather than relying on an organisation's interpretation of its own numbers.

Mr Stern believes PCTs now charged with delivering world class commissioning desperately need this level of detailed scrutiny.

"Unless we have good quality information the judgements we are making are inevitably based on all sorts of assumptions and prejudices. We need to have the best quality information so we have a genuine comparison."

NHS Alliance chair Michael Dixon agrees. "Often we have been commissioning things without knowing what we needed. We don't know how to specify exactly what we wanted to do or check that we got what we thought we needed in the first place."

Dr Colin-Thome also cites world class commissioning as a main driver for PCTs to use the benchmark. "We are saying, 'what you need to do is to assess yourself against other services. It's up to you how you get there but, by implication, the easiest way is to buy this tool which is already up and running'." Otherwise, he warns, "you have got to demonstrate some other way of measuring your quality against others and I would argue that would be more expensive".

Birmingham East and North PCT deputy chief executive Andrew Donald plans to use the measure when his organisation's out of hours contract comes up for retendering. "We have a responsibility to the taxpayer to make sure we can demonstrate we are delivering a high quality service for a good cost," he says.

Out of hours services that took part in the pilot benchmarking are equally enthusiastic.

"I found it a really transparent process," says Liverpool-based provider Urgent Care 24 chief executive Nigel Wylie. "I knew a lot about our service, but it gave me an extra dimension which allows you to see if things are going slightly awry." He is bullish about his own company, having seen results that suggest it is one of the top performers.

And the benchmark will boost good services, he adds. "It's in the provider's interest in terms of market stability," he argues.

Commissioners will be able to test providers that say they can offer the same service at half the current contract price, rather than relying on assurances that they can do this while meeting the national quality requirements.

Good as it gets

Wirral PCT chief executive Kathy Doran was on the advisory group that helped develop the benchmark. "It gives you very specific data about how your service is measuring up against others nationally and enables you to ask questions of your provider and drive up quality."

Her organisation runs its own out of hours service. "I want to make sure Wirral out of hours is as good as it can be but getting really good comparable data is really difficult, particularly in primary and community care."

"It is not just enough for a provider to tick the box against the quality standards of the commissioner. Providers - not just out of hours but any providers - are wanting to improve themselves," says Dr Colin-Thome.

Mr Clay says the data drills down to deeper comparisons between services. On costs, for instance, "we asked about the detail of what a service includes - does the PCT provide its premises, or support for finance or human resources or clinical governance?" he says.

This makes possible comparisons between, say, different nurse-led services or those provided in rural or urban areas - and Ms Doran, for one, welcomes this clarity. While the National Audit Office report was "a good start", she admits "a number of us had concerns about the quality of the data that we were giving them".

Dr Colin-Thome accepts criticism that the DH should have been able to say with more confidence how the NHS functions in the community overnight and at weekends: "You could say we should have been up to speed on that." Instead, it was left to PCTs to assess their own local services - hindered by a lack of comparable data.

"We should also have had a central review process to compare across the country," admits the primary care czar. "I would accept that we should have been faster - but all I can say is that we have responded to the criticisms that were in the NAO report."

Funding the new benchmarking tool has been an attempt to rectify the situation, he adds. He argues reliable information on performance will help restore public confidence in out of hours, as should the inclusion of a new emphasis on patient experience.

"The public scandals have been overplayed by the media when you consider there have been no public safety issues that you could ascribe specifically to out of hours - that's a pretty good, safe service."

The foundation is working with its suppliers to develop a patient questionnaire, but Mr Stern says it will be fascinating to see the detail of people's experiences, rather than merely top-line questions. "What we are looking at is lots of indicators, not just 'are we the best thing since sliced bread?'," he says.

Access to medicines, for instance, is a significant issue for patients overnight that is rarely included in the usual patient surveys.

"We are trying to pick out indicators that are more testing, not a questionnaire in isolation from the quality of the service," adds Mr Stern.