Published: 01/04/2004, Volume II4, No. 5899 Page 18 19

On the day that CHI's reign as NHS inspectorate ends, new research shows what lessons its record holds for the new Healthcare Commission

Today the Healthcare Commission takes on responsibility as NHS regulator. It has the luxury of a fresh start - it can choose which of the Commission for Health Improvement's ideas to take up and which to discard.

But how will it know what worked and what did not? What evidence is there that the new commission can use to inform its decision-making? Faced with the task of designing a new regulatory regime for the NHS to bring about real and lasting improvement, what should the new commission do?

There is no shortage of evidence - plenty of research has been done on regulators in all kinds of industries, and there is an emerging understanding of how to regulate in a way that maximises performance improvement and minimises some of the costs and adverse consequences.

The key principles boil down to being very focused on achieving improvement, treating organisations differently according to how well they perform or respond, and having some real levers - both rewards and sanctions - and being willing to use them (see box).

The job of a regulator is a difficult and contested one, and regulated organisations are quick to protest and rather slower to praise. It is easy for regulation to become a ritualised, formal and mechanistic process, all about ticking boxes and checking checklists, writing reports and issuing guidance. Somehow, in the process we can lose sight of the original purpose: bringing about real improvements in services which will benefit patients.

There have been several studies of CHI's performance over the last three years and during that time CHI has revised its approach to clinical governance reviews.

During the last six months, we have been working with CHI to understand what impact clinical governance reviews have had on trusts, and how far the recommendations they contain have been implemented.

Manchester Centre for Healthcare Management examined the CHI reports on 30 individual trusts, the trust action plan which followed and any subsequent progress.We also collected data through a questionnaire to each trust, a matched primary care trust and its strategic health authority, and through interviews with key staff in four case study trusts.

Across the 30 trusts we looked at, CHI had identified a total of 958 recommendations (or 'key areas for action').This was around 32 per trust.However, the numbers of recommendations varied hugely, from just five at one trust, to 57 at another. The reports themselves were anything from 20-80 pages long. In response, trusts produced action plans of equally varying length and quality - from just four to 89 pages in length. The longest contained an eye-watering 247 action points.

Over three-quarters of CHI's recommendations were addressed by those action plans. The data on implementation revealed that about 22 per cent of recommendations had been fully implemented and 43 per cent had been mainly or largely implemented. At least some action had been taken on almost all of the rest. In our survey, we found that for about 57 per cent of changes, people attributed the fact that change had happened to CHI's intervention rather than to other factors or pressures.

There are four key lessons for the Healthcare Commission.

First, it has been very hard for CHI to assure consistency in the review process, in part because it decided against the use of explicit standards and instead relied heavily on the professional expertise of its review teams. The new commission will work within the framework of healthcare standards set by the Department of Health and it has a difficult balance to strike. It needs explicit standards and criteria against which it will review performance, but it should avoid those standards becoming so detailed and specific that they leave no space for the professional judgement of the reviewers.

Another reason for the Healthcare Commission to use clear standards is that when regulators set standards, most organisations comply of their own accord.Much of the regulator's influence on performance comes not from its individual inspections or reviews of organisations, but just from setting standards.

Second, CHI focused on clinical governance, arguably because that is what the legislation said it should do. But this meant that most of its recommendations were to do with management systems or processes, and few had much to do with patient care.

There was an implicit - and largely untested assumption - that things like better risk management, or improved continuing professional development would lead to better care for patients. The Healthcare Commission will look more broadly at NHS trusts' performance, and while some of its recommendations should still concern clinical governance and other management issues, most need to be clearly focused on improvements that patients will actually understand and notice.

One of the challenges for the commission is that the DoH's proposed healthcare standards within which it has to work are largely focused on process issues.

Third, CHI put huge effort into setting up its clinical governance reviews, and more or less achieved the target set for it by government of reviewing all organisations within four years. The job of following up on reviews to make sure that NHS trusts did what they had promised was given not to CHI, but to the now-abolished DoH regional offices, and latterly to strategic health authorities, neither of which seem to have done very much until they were forced to do progress reviews for the 2003 star-ratings.

The lesson for the Healthcare Commission is that it should invest more time and effort in following up its inspections and making sure that promised changes have materialised.

Formally, the commission is meant to leave this to SHAs (and to the new independent regulator for foundation trusts) but the evidence suggests that doing so is a risky strategy.

Fourth, CHI was originally tasked with reviewing each trust every four years, but we found that most of the changes resulting from CHI's recommendations took place within 12 months of the review.

This meant there was a period of sometimes frenetic pre and post-review activity and then a long pause of up to two-and-a-half years.

The Healthcare Commission needs to shorten this cycle so that it has a more continuous involvement with organisations and can intervene more frequently where this would help to bring about improvement.

CHI achieved a great deal in its short existence. It established an acceptance of external regulation in the NHS and drove forward clinical governance.

It also showed a real commitment to improvement in the way it did business and was willing to learn from experience and change its review process when it needed to.

The Healthcare Commission now has a great opportunity now to learn from CHI's successes and its failures. It is, of course, to be hoped that the commission gets rather longer in which to make its mark on the NHS.

Kieran Walshe is professor, Lawrence Benson is lecturer and Alan Boyd is research associate at Manchester Centre for Healthcare Management, Manchester University.

Further information

Learning from CHI: the impact of healthcare regulation www. chi. nhs. uk /eng/about/kieran_walshe_report. pdf

Principles of effective regulation

Improvement focus

Give relentless attention to real changes which matter to people

Responsiveness

Adjust the intervention to fit the way in which organisations respond

Proportionality and targeting

Tackle organisations and areas where there are known problems

Rigour and robustness

se valid, well grounded standards and performance measures

Flexibility and consistency

Allow organisations to be different, but be consistent in how they are assessed

Cost consciousness

Minimise the costs imposed on regulated organisations

Openness and transparency

Put as much information as possible in the public domain

Enforceability

Make sure the regulator has some serious sanctions which can be used

Accountability and independence

Give the regulator enough space to do the job properly but make them accountable

Formative evaluation and review

Measure whether it works and change things if it does not