Oh, wait. These massive interruptions were somewhere else, not at Healthcare.gov?
You mean sometimes there are massive IT failures in non-government organizations? Even in well-established routine application systems that aren't brand new and ground-breaking? Even with vendors who are well-known, market-leading private sector companies?
Yeah, but... but... but... Obama is still incompetent or something for some reason because socialism Kenya detached golf.
Actual letter received today from the administration of a large, successful private university:
To the [large world-renowned private university] Community:
The network problems experienced over the past two days were corrected Monday at approximately 8 p.m., and all University and Healthcare systems were back online by 10 p.m.
We deeply regret the risk, cost and inconvenience created by these outages and are working with our networking vendor to minimize the impact associated with any future changes.
As background, over the weekend, IT took another step in replacing the core networking infrastructure that serves both Healthcare and the University. This conversion is necessary routine maintenance, replacing equipment that is over six years old. It is also a desirable upgrade to satisfy demands for increased capacity and advanced features that support [large world-renowned private university]’s multiple missions.
The changes over the weekend were intended to add the last set of features and capacity to move the healthcare networks to the new core with the already-migrated university networks. Last week, the networking team worked with our networking vendor (Cisco) to validate the new, planned configuration and performed multiple successful simulations with identical equipment in our testing lab. On Sunday, the upgrade was installed at 1:30 a.m. EST, and all indications were that it was successful. At approximately 7:45 a.m. on Sunday, network stability wavered in the face of some unseen error condition.
On Sunday afternoon, the networking team backed out a portion of the overnight changes as a workaround, and that led to a period of stability for both healthcare and university applications through Monday morning. Monday's additional load proved that not all issues were resolved by the workaround.
Continuous troubleshooting in consultation with the highest possible level of technical resources from Cisco were not able to identify a root cause, and at 6 p.m. the networking team began backing out the remainder of the overnight changes from Sunday. Our internal network engineers are now working with Cisco to further analyze the outages and formulate a revised plan for performing the upgrade. Cisco is in the process of building a lab with our exact configuration in order to identify the issue in their equipment or recommended design.
Thank you for your understanding as we work through these complex infrastructure changes.
[name], Deputy CIO, University Technology Services
[name], CIO [large world-renowned private university] Healthcare
[name], Enterprise CIO