Published: 04/03/2004, Volume II4, No. 5895 Page 4

A three-year study of the work of the Commission for Health Improvement has suggested the body's assessments have been based on little more than a 'rag bag of reflections'.

The provocative report, funded by the King's Fund and written by Patricia Day and Professor Rudolf Klein, comes on the eve of CHI's abolition and replacement with the Commission for Healthcare Audit and Inspection.

In the foreword King's Fund director of health policy Steve Dewar says the report suggests CHI has not got a proper grasp of important but nebulous aspects of good organisations such as leadership and culture.

He continues: 'In the absence of a sophisticated framework for assessment the authors question the process by which 'a rag bag of reflections' (including variable attempts to capture the patient's experience) are converted to global and quantified assessments'.

The report, entitled The NHS Improvers: a study of the Commission for Health Improvement, looked at the modelling and impact of CHI reviews in 25 acute trusts.

Many of the criticisms focus on the subjective nature of CHI's work, and a lack of consistency between reviews.

The report says: 'The achievement of consistency is the (usually elusive) quest of the Holy Grail of all inspectorates. In the case of CHI, it is a quest made more difficult by the way it decided to carry out its remit.'

It adds: 'CHI has developed an elaborate coding system for recording the evidence collected by the teams, and for cross-checking the impressions first gathered.

Even so, reports are only consistent in being inconsistent'.

Given the subjective nature of CHI reviews, authors were also critical of the way trusts were scored, and the way these scores influenced government star-ratings.

'In short, ' the report alleges, 'the problem of inconsistency in gathering evidence is compounded when it comes to interpreting and weighing that evidence in order to decide on a score.'

The report examines the relationship between CHI and the government, which it describes as one based on 'mutual exasperation' and tensions.

It states: 'The tensions between ministerial expectations and CHI's self perception persisted throughout its existence. The relationship between the Department of Health and the commission was one of mutual exasperation.

CHI had to accept - albeit resentfully - the department's role in setting its performance targets, but it guarded its independence jealously. Conversely, the department felt frustrated by what it saw as CHI's resistance to its legitimate demands, and its exclusion from detailed discussion of CHI's methodology.'

The report says ministers' assumptions that CHI would report on trusts' performance were based on 'an act of faith' that overall performance could be equated to clinical governance.

The report praises the government for giving CHAI a wider remit than its predecessor body.

But CHI performance ratings project director Matt Tee said:

'The report says very little that we have not already said about ourselves in the past.

'We were always of the opinion that we were not given a broad enough remit by only focusing on clinical governance, ' he argued.

Mr Tee added that CHI had found it difficult to codify the results of reviews on culture and leadership. But he did not accept that the reviews had been based on a 'rag bag of reflections'.

The report also makes several recommendations to CHAI. It concludes that 'the aim of the new inspectorate should be to provide information that is both consistent and comparable, if only on a limited number of critical dimensions.

'This will be all the more important if the government succeeds in implementing its goals of patient choice.'