Academics doubt link between death and standards

In a report commissioned by NHS West Midlands, academics at Birmingham University have been highly critical of the way hospital standardised mortality ratios produced by health information provider Dr Foster Intelligence were reported and published.

“We found little or no evidence that a high standardised mortality ratio systematically reflects poor quality of care or a failing hospital,” it said.

The strategic health authority commissioned the research by Birmingham University’s department of public health and epidemiology after five of its trusts were listed as “poor performing” on mortality in Dr Foster Intelligence’s Hospital Guide 2007, published in April last year.

Ambiguous and misleading

The report described the Dr Foster mortality ratio as “a metric which is at best ambiguous and at worst potentially misleading”.

NHS West Midlands director of public health Rashmi Shukla told HSJ: “There are a lot of questions that need to be answered before we can publish the data in a way that makes it easy for the media and public to understand.

“Clearly you do need to look at mortality but it is about making sure that whatever measures we use do reliably tell us about quality of care.”

The SHA is working with its trusts to develop new performance measures and improved systems for reporting mortality rates.

There was also anger at the way in which the results are made public - on Dr Foster’s website and, last year, in a national newspaper.

Flawed leagues

The report said: “Given that league tables are fundamentally flawed devices, the publication of mortality statistics in the form of a league table raises concern… A league table is too simple a device and hence it misleads.”

When the DH published national mortality rates for four kinds of operation in July, it urged journalists not to use them to compile league tables.

The researchers found that about 30 per cent of variations in mortality rates were caused by differences in coding - though this is widely accepted to have improved across the NHS since 2003-06, the period covered by the research. The method used by Dr Foster also “apparently penalises hospitals with less provision for the dying in the community”, the report said.

Mechanical shortcomings

The authors believe they have found “mechanical shortcomings in the methodology” used by Dr Foster Intelligence, the details of which are undergoing peer review.

Mohammed A Mohammed, the lead author of the report, said Dr Foster’s system “implicates quality of care by process of elimination, which is flawed”.

“You cannot convict the 10th person because the other nine were not present at the scene of the crime,” he told HSJ.

Dr Mohammed said moves towards more outcome measurement and reporting were “a bad thing” if people could misinterpret them.

Paul Aylin, assistant director of the Dr Foster unit at Imperial College London, which carries out data analysis of hospital standardised mortality ratios but is not involved in their publication, said there were a number of possible explanations for variations in the rates. These included chance, poor coding, inadequate risk adjustment and quality of care.

Robust measurements

He said: “We believe the HSMR to be a robust measure of overall mortality, but that it should be used in conjunction with other indicators in the assessment of care quality.

“Analysis of mortality in individual diagnoses and procedures, as well as the analysis of other outcome and process indicators, is invaluable in explaining variations between trusts.”

NHS medical director Sir Bruce Keogh toldHSJ last month: “The days of discussing whether or not we should publish outcomes are over, but now we need to get some more sophisticated discussions into publishing ones that are meaningful.”

The DH, responding to the West Midlands report, said: “We welcome all contributions to the development of indicators that will help improve our knowledge of clinical quality.

“It is important that indicators are accurate, technically robust, meaningful and adjusted for risk. The more scientific debate there is about how to achieve this, the better.”

A spokeswoman for Dr Foster Intelligence said it is “committed to improving the quality of care through continued advancement in the use of information”.

“Hospital standardised mortality rates are one of 30 outcome measures Dr Foster Intelligence uses in its provision of information to organisations to help them compare standards and services on a variety of levels.”