There has been a degree of disquiet about publishing mortality rates. Supporters hoped this would lead to greater transparency, quality and patient choice - but has reality matched expectations? Daloni Carlisle reports

It takes a fair bit of delving on the NHS Choices website to find the "operation success rates" that were published in July with much trumpeting of their benefits for transparency, quality and patient choice.

The data does not come up under a search for success rates or mortality rates. Instead, you get an article from NHS medical director Sir Bruce Keogh explaining why the health service has published data on four operations: abdominal aortic aneurysm - both emergency and elective - and hip and knee replacements.

No, to find the rates you have to do a multi-step "find and compare" search that eventually brings up the rates for hospitals in a given area, alongside data on MRSA rates, the 18-week wait target and parking. Click on mortality rates and it pulls up an explanation of what they tell you.

Perhaps this is deliberate. For while officially the British Medical Association, the Royal Colleges and assorted surgical societies all welcome the publication of the data, talk to them and a deep breath and a series of "buts..." always follow the initial expression of welcome.

The feeling that distinctly emerges as a result is that this summer's publication was the first rumble of a seismic shift. However, no one wants to frighten the horses.

Broadly, the "buts" concern the following: Does publication of this data improve quality? Does it improve choice? Is the data quality good enough? Is it meaningful? Will the data be used for performance management?

Let's start with quality. Leslie Hamilton is the new president of the Society of Cardiothoracic Surgeons, the first UK surgeons to publish their mortality data under Sir Bruce's leadership. He offers this welcome: "We have led the way on this," he says proudly. "We felt there was a professional obligation to monitor our performance. But it's been a sensitive issue and there has been a degree of nervousness about it."

But... the following quote is from Mr Hamilton's inaugural editorial in the society's bulletin publication in August: "Providing the best quality care to our patients has to be our primary aim. Publishing data on post-operative mortality has set the scene, but I do not believe it has helped us actually improve the quality of care significantly. We need to move on to aspects of care where we can make changes."

His point is this: mortality in cardiac surgery is so low - at less than 1 per cent - that there is precious little surgeons can do to influence it.

"I am very keen to move away from baseline mortality rates and to move towards looking at complications. Maybe that would allow us to identify units that have more or fewer complications and then we can learn from best practice," he says.

The same point can be made for the new data published. "For most procedures, mortality rates are so low as to be almost irrelevant," says president of the Royal College of Surgeons John Black.

The data on all four operations published so far tells us merely that there are no outliers: in other words, no hospital has death rates so high as to ring alarm bells.

Reassuring, says Sir Bruce. But many senior clinicians ask: does this really tell us very much about quality?

Mark Davies, medical director of the NHS Information Centre, the body now responsible for compiling the data available on NHS Choices, is not at all sure this is the right question.

"One of the barriers to publishing this sort of data is that we do not have the perfect measure," he says. "I welcome the point where we say pragmatically 'here are some indicators but they do not give you the whole story'.

"It is starting the dialogue in a mature way. Standardised mortality rates do not give you the whole picture and, I would argue, do not give you an answer in terms of quality. But they provide the basis for discussion."

NHS Confederation chair Bryan Stoten agrees the move is "a start". He points out: "It is important we bear in mind that the NHS stands or falls by the public confidence expressed in it."

Even Sir Bruce agrees: "Mortality is a blunt instrument," he says. "I chose these measures for two reasons. The first was that when Ken Kizer took over running the Veterans Association [a US healthcare system he took charge of in 1994], he found quite significant variations in mortality. The second was that I wanted to send a clear message to the service that the discussion on whether we should measure outcomes was over and we are now in the phase of finding the right ones to measure in the right way."

Interestingly, though, the most recent evidence on public disclosure of hospital performance (from a September report commissioned by the Health Foundation from the Rand Corporation) is that it does drive up quality. Rand's original work in 2000 on the same question is cited as evidence for the publication move by the Department of Health.

Health Foundation deputy chief executive Vin McLoughlin is keen to qualify the findings.

"There are very few data reporting systems and of course we need more evidence. However, I do think the report shows that reporting of data on performance does stimulate change, particularly at the hospital level."

The impact is on the poor performers rather than the good, she says. "That's where we see the dramatic change."

At its starkest, individual surgeons shown to have high mortality rates will be obliged to stop practising voluntarily. At an individual hospital level, public disclosure of poor performance leads to changes in those organisations to address underlying problems behind the issue.

But the review found that there was no impact at all on market share. "In a sense that's not unsurprising because a lot of the material is very technical and difficult for members of the public to get a proper context for what the data is showing them," says Ms McLoughlin.

This brings us neatly to the next "but": does publication of mortality data influence patient choice? Well, no, says the Health Foundation.

Mr Hamilton, of the Society of Cardiothoracic Surgeons, agrees. "We have two cardiac units in the North East [where he works] and patients are routinely asked which one they want to go to. I have never heard anyone raise it in discussion and I have never had a patient ask about mortality rates."

Informed choice

Yet questions remain. At a recent lecture held by the National Confidential Enquiry into Patient Outcome and Death, Sir Bruce outlined his take on choice.

"By choice, I mean people can choose who they see, where they see them and hopefully when. But most importantly they get the right information to engage their health professionals in a conversation which enables them to understand the risks and benefits of their treatment so they can make a choice about it."

The next "but" is data quality. "Most clinicians have concerns that the NHS dataset is not terribly accurate or fit for purpose," says George Findlay, a clinical co-ordinator at the national enquiry.

Getting data up to scratch will require senior clinical input - and that is a long way off, says Dr Findlay.

"Most clinicians see data as the remit of the coding department and not something that they have a responsibility for. That's because they do not have to use it."

Not everyone agrees with this pessimistic outlook. "Publication of this data is a good idea but does depend on its quality," says Royal College of Surgeons president John Black. "If the data is flawed, it brings it out into the open and we get better data."

The cardiothoracic surgeons' data was a case in point. Getting it ready for publication took a massive amount of work, not just in risk stratifying, but also in making sure it was "clean".

Chair of the BMA consultant committee Jonathan Fielden says a common managerial and clinical plan in delivering high quality data is beginning to emerge, driven primarily by payment by results and service line reporting.

"It is producing data that is relatively clean and can be used clinically. It gives you data about your throughput, how long it takes and your mortality," he says. And once a trust has such data, it is bound to end up in the public domain.

So how much better, he says, for the health service to take control and publish its own data along with a useful narrative?

But can the data be published in a way that helps the public to understand that an independent treatment centre, with an intake of the least challenging patients, would expect to have a lower mortality rating than a tertiary centre taking those with substantial co-morbidities?

"In a sense, the problem of inaccurate data is a historical one," says Dr Fielden. "Now it is one of public understanding."

"One of the issues we have in the UK is health literacy," adds Dr Davies at the Information Centre.

"We need to get to a point where patients are much more enabled consumers of healthcare, are asking much more pertinent questions and are much more actively involved in the discussions around their care. Giving them information that allows them to ask sensible questions is a first step."

The final "but" regards performance management. It has already been mooted that revalidation of individual doctors will in future depend on some performance data such as mortality rates.

This would be vigorously fought by the Royal College of Surgeons.

"[To do so] is fundamentally unfair because it puts the spotlight on surgeons who do operations with significant mortality," says Mr Black at the royal college.

A classic case is cancer of the oesophagus, for which surgical treatment carries a 5-10 per cent mortality rate.

"But everybody who needed but didn't have the operation would have died within six months anyway," says Mr Black. "Nobody advises a patient to have an operation with a high risk of dying unless the underlying disease process has a higher risk of dying."

Professor of clinical epidemiology at the London School of Hygiene and Tropical Medicine and director of the Royal College of Surgeons' clinical effectiveness unit Jan van der Meulen makes an analogy with parenting.

"I think the use of standardised mortality rates in performance management is like telling parents they will get more child benefit if their child does well at school. Parents have limited impact on their child's performance and surgeons have limited impact on mortality."

It may also be antipathetic to the next developments in performance rating. There is universal agreement that the success rates published this summer point the direction of travel towards something more complex than the data published so far.

Sir Bruce expects the debate to hot up over the winter. Along with chief nursing officer Christine Beasley, he has written to every trust to ask which measures they are already collecting and to "see if any have merit for assessing quality" and will be launching a consultation on the issue this autumn.

He told HSJ: "There is no point in me as medical director producing thousands of metrics. What I want is to kick-start a serious discussion within professional circles and among professional organisations on how we use currently available data to maximise efficiency and then how we start to move into more sophisticated measures."

The publication of mortality rates is here to stay and patient-related outcomes are not so far behind. Up until now, the data is pretty non-controversial. You can be sure that will not always be the case.

How we got here

Florence Nightingale was the first female member of the Royal Statistical Society and proposed outcome measures for surgical procedures in London hospitals.

NHS medical director Sir Bruce Keogh says: "After she died, her league tables fell into disrepute. Critics said the data was dodgy, they introduced perverse incentives and gaming was going on. I invite you to consider whether the arguments have changed."

Fast-forward to 1998 and the Bristol Royal Infirmary Inquiry into the deaths of children undergoing heart surgery. Sir Bruce was then president of the Society of Cardiothoracic Surgeons and pushed for heart surgeons to publish survival rates for heart bypass surgery. It took him eight years. "Sceptics warned it would lead to cardiac surgeons turning down difficult cases," he says. "It didn't, because doctors are dedicated professionals who rise to challenges. Doctors know we owe it to our patients to look at any information that could possibly improve or inform our performance and skills."

Two years later health minister Lord Darzi published his review of the health service, promising publication of mortality data on the NHS Choices website. In July 2008, the DH moved on this, marking a world first.

The role of the press

When mortality data was published, the Department of Health referred to it as "survival rates" and has asked the media not to turn the rates into league tables. Fat chance.

Viewers of BBC online on 10 July 2008 would have seen a headline that did not refer to survival rates or even mortality rates. No, it was "Operation death stats made public".

On 11 July, The Daily Telegraph ran with death rates in some hospitals "three times too high". The story did not make clear whom this was quoting. Possibly the headline writer?

The article went on to say there were fears that a league table could lead to defensive practice - and then published a list of the "bottom" and "top" performers.

NHS Confederation chair Bryan Stoten condemns this approach. "Using this data to create mass hysteria is irresponsible and irrational," he says.

Head of communications for University Hospitals of Leicester Tiffany Manser handled the media operation for one of the "top" performers.

"It went really well and we were really pleased with the way the information was translated. We did not have much national coverage but a lot of regional interest. Our consultants were more than happy to talk about it and we were able to find a patient to talk about their experience," she says.

It might have been a different matter if the trust had been in the bottom of the league.

"We would have had to handle that differently," she acknowledges.

"We would have said the devil was in the detail and questioned the interpretation of the data.

"We also do regular polling of our patients and so we know pretty much what they think about different aspects of our services.

"We would have started to pull that information apart to give more background."

Chair of the Association of Healthcare Communicators Nick Samuels offers some comforting words to NHS press officers dreading the next round of headlines: "The press get very excited initially but over time they will calm down.

"They will get more mature about it, although those hospitals with the worst figures will always have more explaining to do."