The title of this post isn't some sort of engineering metaphor being used to explore the tribulations of the UK's National Health Service. Although it could be. No, this post is about a massive IT failure which has resulted in NHS employees not being able to access Patient Administration Systems (PAS) across Birmingham and the Black Country, Cheshire and Merseyside, Cumbria and Lancashire, Greater Manchester, Shropshire and Staffordshire and the southern part of the West Midlands. PAS systems are used to administrate hospital bookings, so as well as managing the patients, the systems schedule the facilities of the hospitals too. The systems, which are centrally hosted by CSC and their supplier Hitachi, have been off line since Sunday morning. I've only been able to find information on the BBC and The Guardian so far, where it is claimed that the problem was caused by a Network Area Storage failure. According to a spokesman from NHS Connecting for Health "The nature of the incident meant that service could not immediately be provided by the back-up systems." Now that is a strange failure mode.
It wouldn't be strange for a home user. It wouldn't be strange for a small business. However, it's quite strange for anything supporting more than several hundred users and extremely strange for as large a deployment as this.
I hope there's an enquiry into how and why this happenned. It's a terrible indictment of the state of IT in the NHS.
Tags: technology, healthcare, IT, healthcare IT, NHS, CSC, Hitachi, Connecting for Health, CfH, outage, failure