North by North West: The software upgrade that sparked huge IT failure

A full post-mortem on the extraordinary IT outage across four hospitals in Greater Manchester is still being carried out, but some early findings were identified in a board report last week.

On 18 May, multiple systems went down across the hospitals in Oldham, Bury, Rochdale and North Manchester, forcing nearly all services to move to paper-based processes for a period of almost three weeks.

The hospitals had all experienced repeated tech issues over the last decade, due to their ageing IT infrastructure, but nothing as serious as this. The report described the extent of the outage as “highly unusual”.

It is miraculous perhaps - and a tribute to staff who had to cope with the switch to paper notes, and treat patients without access to their full history - that no serious safety incidents have so far been reported in relation to the outage (the trust insists its Datix systems remained available).

The report to the Northern Care Alliance board, which took over the former Pennine Acute Hospitals Trust sites, said detailed discussions are ongoing with software supplier, VMware, as well as hardware supplier Dell EMC, to establish the root causes.

Early indications suggest there was a “bug” in a software upgrade which was installed in January, the report says, and “there appears to have been a series of events which have conspired to expose a fault in code and fatally compromise the infrastructures’ ability to function effectively”.

It also noted that in the early stages of the outage, VMware suggested a further upgrade, because the January upgrade “was known to have issues”.

This sounds like there could be some questions about whether it could have been prevented altogether.

Despite using much of the same infrastructure, Salford Royal Hospital, also run by NCA, was unaffected because it installed a later upgrade of the software which did not contain the bug, the report said.

A spokeswoman for VMware told NxNW: “We are continuing to work with Northern Care Alliance NHS Foundation Trust following the continuity issue in May, in order to investigate and fully understand the root cause of the incident.

“VMware’s priority is always to support our customer and quick resolution of the issue was and continues to be of utmost importance to us.”

Dell EMC said it was also working with the trust to establish the root causes.

Instead of undertaking the further upgrade in the first few days of the outage, and after advice from independent consultants from Swedish tech firm Proact, the trust decided to migrate services to a new “simplified technology stack”, which is apparently stable though less efficient to use.

The trust said all this will be part of a trust-wide “infrastructure re-alignment” over the next two years, and the root cause analysis will feed into this. Next year, the Oldham, Bury and Rochdale sites are set to install the Allscripts electronic patient record, which is already used in Salford.

The trust added that more work is needed to establish the full impact of the outage, such as the effect on elective activity. It said there will be a full “lessons learned” report later this month.

North Manchester is now run by Manchester University FT, which is due to implement a separate EPR later this year.

High risk plans

Integrated care boards held their first formal meetings on Friday, which were largely spent rubber stamping constitutions, policies, and governance processes.

The Greater Manchester ICB did spend some time on its financial plan for 2022-23, though, with chief finance officer Sam Simpson outlining the high risks involved.

When the system initially submitted plans in April they were £187m short of breakeven, due largely to a significant rise in pay and price inflation since the start of the year.

GM has since been allocated a further £80m by NHS England, which narrows that gap, and has agreed to submit a balanced plan based on further savings being made.

Given providers were already tasked with delivering savings of 5 to 6 per cent under the original plans, that looks a tall order. Ms Simpson acknowledged it is “clearly extremely challenging” and the “position is at a high level of risk from any other in-year financial challenges”.

Shaking the balance sheet

It is set to be an interesting year on the financial front.

Many of these breakeven plans look impossible, but let’s remember that trusts were able to stuff their balance sheets with plenty of cash that was on offer during the pandemic.

They will be able to shake some of that out during the year, by shifting assumptions on items such as accruals, which will help deliver paper-based savings (watch out for some HSJ analysis on this over the summer).

This might get systems through 2022-23, but the longer term problem is none of those paper savings will be recurrent, which will simply increase the challenge in 2023-24, with less wiggle room available.

Patient choice

NHSE has always insisted that patient choice would remain in place under the new architecture of integrated care systems – with people retaining the ability to select an out of area provider for their treatment if they wish.

But what about when services are full to capacity?

In the quasi market, trusts would simply run their own waiting lists and accept referrals from any commissioner, which in theory meant there was no priority given to local patients.

But in the world of system-working and pooled waiting lists, trusts now have more of an interest in what’s best for patients in their own ICS.

West Lancashire MP Rosie Cooper has cited a recent example of a constituent struggling to access dermatology care in Cheshire and Merseyside, saying they were refused for being out of area.

Cheshire and Merseyside ICS said some of its trusts decided to temporarily restrict routine treatment for “external referrals”, due to supply and demand issues.

However, it said capacity and patient flows from West Lancashire are being urgently reviewed, with some insourcing support meaning that routine referrals will soon be accepted again. It said a community service was being worked on for the longer term.