CQC risk monitoring 'wrong more than right', study warns

The system was developed in 2013 by management consultancy McKinsey under a £1.2m contract with the CQC. It assigns a risk score to hospitals based on 150 equally weighted metrics including mortality, whistleblower reports and staff vaccination uptake rates.

The quality regulator is developing a new risk assessment system incorporating elements from the existing tool.

The CQC is moving to a more risk based regulatory regime – so providers rated good and outstanding would be inspected less often – due to a 25 per cent cut in its government grant by 2020.

However, research by Alex Griffiths, from the London School of Economics, casts doubt over the efficacy of the CQC’s intelligent monitoring tool.

Mr Griffiths, from LSE’s centre for analysis of risk and regulation, said: “The process the CQC had, and elements of it are being used in their new system, wasn’t up to the job. It cannot effectively work and the risk is that poor care can continue longer than it should do and good quality providers will be unfairly burdened.

“There is certainly a danger that the revised system will not effectively predict the risk outcomes, which is what is necessary if you are becoming ever more reliant on it as funding is reduced.”

Mr Griffiths, along with colleagues from King’s College London, compared the results from more than 100 CQC hospital inspections with the initial predictions made using intelligent monitoring.

He said: “The risk estimates were wrong more often than they were right. The tool was, in that sense, actively misleading. The danger of that is that if you’re missing poor quality care than people can go on suffering. There is also the opportunity cost – that if you’re burdening high quality providers then you’re distracting them from administering the good quality care at the same time, so there is a double cost of getting it wrong.”

The research, published in the BMJ Quality and Safety Journal, found that when the researchers attempted to predict using the tool the trusts performing poorly and those performing well, they achieved an overall accuracy rate of 48 per cent. There were 11 false negatives and 43 false positives.

The overall median risk score for trusts rated good was higher than those trusts rated requires improvement. Even where the tool was limited to whether it could identify trusts rated as inadequate, the research found it correctly identified six out of 13 trusts as inadequate, while wrongly identifying three times as many trusts that were not inadequate.

The research concluded: “The continuous risk scores generated by the CQC’s statistical surveillance tool cannot predict inspection based quality ratings of NHS hospital trusts.”

Due to the scale of trusts found to require improvement – 81 out of 103 – it added: “In that context, rather than trying and failing to target inspections, it may be preferable to continue systematically inspecting each trust or adopting a random approach to signal to managers and the public alike that every hospital matters.”

Mr Griffiths said about the CQC’s new system: “It is a more nuanced system and they are moving away from equally weighting indicators but if you are using indicators that are at a very high level and subject to known ways of gaming, even if you take a very advanced mathematical approach, the data is very unlikely to give you an effective risk prediction.”

Mr Griffiths added that he supported the CQC and trust regulation, arguing it was “beneficial”, but he added: “The approach they have tried has not worked by any reasonable measure and there is a strong impetus from this to do something better if they are going to avoid a lot of poor quality care, which is an awful lot more likely to arise in the current NHS funding climate.”

A CQC spokesman said: “Our intelligent monitoring tool served a clear purpose, which was to help prioritise our comprehensive inspection programme of every NHS trust in this country. It was a planning tool, helping with our initial scheduling – it was not a formal judgment of risk or a tool to predict ratings.”

He said the CQC was now using the information from its inspection regime to develop its new tool.

“This will consider more intelligence in real time and will place greater emphasis on qualitative data, such as the views of individual healthcare professionals, patients and local partners. Ultimately, this will allow us to be even more responsive to risk and to carry out more focused and targeted inspections of NHS services,” he said.