The NHS should not use hospital-wide mortality rates as a ‘smoke alarm’ to identify poor quality hospitals, a major study has concluded.

  • NHS England commissioned study of mortality rates concludes they should not be used as “smoke alarm” for poor quality
  • Research finds “no significant association” between mortality rates and proportion of avoidable deaths
  • Recommendation that “avoidable death rate” should not be used to compare trusts’ quality challenges government plans for hospital banding
  • HSJ’s brief guide to mortality rates

The research, commissioned by NHS England, found “no significant association” between hospital wide standardised mortality ratios and the proportion of avoidable deaths in a trust.

Researchers also said that producing “avoidable death rates” via case note reviews was not a good judge of quality, calling into question the basis of government plans to “band” hospitals according to their projected rate of avoidable deaths.

The review looked at the link between the two main mortality indicators and the proportion of avoidable deaths in a hospital as determined by case note reviews.

The hospital standardised mortality ratio and the summary hospital level mortality indicator measure whether the number of deaths observed at a hospital is higher or lower than expected, based on a statistical calculation.

In 2013, 14 trusts with higher than expected mortality rates were inspected by NHS England medical director Sir Bruce Keogh because of concerns they could be providing poor quality care.

Nick Black

Nick Black said avoidable death rates were “not the best way of assessing quality of hospitals”

Following the Keogh review, which placed 11 of the 14 trusts in special measures, Sir Bruce commissioned a study to look at whether standardised mortality ratios provide an accurate indication of the number of avoidable deaths.

The study, published tonight by The BMJ, was led by Helen Hogan and Nick Black of the London School of Hygiene and Tropical Medicine and Lord Darzi of Imperial College London.

Looking at 3,400 deaths in 34 randomly selected trusts, the largest ever nationally representative review of hospital deaths in England, the researchers found an overall proportion of avoidable deaths of 3.6 per cent.

They found “no significant association between hospital-wide standardised mortality ratios and the proportion of avoidable deaths in a trust”.

Professor Black, a longstanding critic of mortality rates, told HSJ the study found that a “pretty dramatic difference” in HSMR - rising from 105 to 115 - was only associated with a “pretty small, [statistically] insignificant change in avoidable deaths of 0.3 per cent”.

However, he added that comparing trusts based on their rate of avoidable deaths as calculated by case note review was also flawed, because of the small proportion of avoidable deaths and little variation between trusts.

“Dreadful though any avoidable death in a hospital is… they make up such a small proportion of the [annual] 15 million hospital admissions, that no metric based on hospital deaths is a meaningful indication of the quality of that hospital,” Professor Black said.

He added that proposals unveiled by the health secretary Jeremy Hunt for producing “projected avoidable mortality rates” for every trust were “not the best way of assessing quality of hospitals”.

While the authors of the report concluded that hospital-wide mortality rates were not a good way of comparing trusts, they said looking at these indicators for individual, high fatality conditions could still be valuable.

Reviewing records of patients who die in trusts was also an “opportunity for identifying local quality problems and stimulating improvements”.

However, the review findings were challenged by Paul Aylin from the Dr Foster Unit at Imperial College, which has pioneered the use mortality rates.

Professor Aylin said the study had a number of limitations, such as a small sample of case notes per hospital and flaws in case note reviews in attributing the preventability of deaths.

“With such small numbers [Professor Black] is never going to find a relationship with hospital mortality, or as likely or not with any other measure, except by chance alone,” Professor Aylin said.

“He’s an epidemiologist, he should know about sample size,” he added.

A Department of Health spokeswoman told HSJ the review would form part of the evidence base for their work on avoidable deaths.

Sir Bruce Keogh said the report’s findings provided “additional insight and show we need to look at a whole range of methods to measure healthcare and quality”.

Sir Mike Richards, the chief inspector of hospitals at the Care Quality Commission, said he agreed that “looking at mortality rates in isolation will not give a full picture of quality”.

“This is why we look at a range of measures during every inspection, including talking to staff and patients, to assess if a trust is providing safe and effective care,” he added.