OK

This is only a Preview!

You must Publish this diary to make this visible to the public,
or click 'Edit Diary' to make further changes first.

Posting a Diary Entry

Daily Kos welcomes blog articles from readers, known as diaries. The Intro section to a diary should be about three paragraphs long, and is required. The body section is optional, as is the poll, which can have 1 to 15 choices. Descriptive tags are also required to help others find your diary by subject; please don't use "cute" tags.

When you're ready, scroll down below the tags and click Save & Preview. You can edit your diary after it's published by clicking Edit Diary. Polls cannot be edited once they are published.

If this is your first time creating a Diary since the Ajax upgrade, before you enter any text below, please press Ctrl-F5 and then hold down the Shift Key and press your browser's Reload button to refresh its cache with the new script files.

ATTENTION: READ THE RULES.

  1. One diary daily maximum.
  2. Substantive diaries only. If you don't have at least three solid, original paragraphs, you should probably post a comment in an Open Thread.
  3. No repetitive diaries. Take a moment to ensure your topic hasn't been blogged (you can search for Stories and Diaries that already cover this topic), though fresh original analysis is always welcome.
  4. Use the "Body" textbox if your diary entry is longer than three paragraphs.
  5. Any images in your posts must be hosted by an approved image hosting service (one of: imageshack.us, photobucket.com, flickr.com, smugmug.com, allyoucanupload.com, picturetrail.com, mac.com, webshots.com, editgrid.com).
  6. Copying and pasting entire copyrighted works is prohibited. If you do quote something, keep it brief, always provide a link to the original source, and use the <blockquote> tags to clearly identify the quoted material. Violating this rule is grounds for immediate banning.
  7. Be civil. Do not "call out" other users by name in diary titles. Do not use profanity in diary titles. Don't write diaries whose main purpose is to deliberately inflame.
For the complete list of DailyKos diary guidelines, please click here.

Please begin with an informative title:

Oh, wait. These massive interruptions were somewhere else, not at Healthcare.gov?

You mean sometimes there are massive IT failures in non-government organizations? Even in well-established routine application systems that aren't brand new and ground-breaking? Even with vendors who are well-known, market-leading private sector companies?

Yeah, but... but... but... Obama is still incompetent or something for some reason because socialism Kenya detached golf.

Actual letter received today from the administration of a large, successful private university:

Intro

You must enter an Intro for your Diary Entry between 300 and 1150 characters long (that's approximately 50-175 words without any html or formatting markup).

To the [large world-renowned private university] Community:  

The network problems experienced over the past two days were corrected Monday at approximately 8 p.m., and all University and Healthcare systems were back online by 10 p.m.  

We deeply regret the risk, cost and inconvenience created by these outages and are working with our networking vendor to minimize the impact associated with any future changes.

As background, over the weekend, IT took another step in replacing the core networking infrastructure that serves both Healthcare and the University. This conversion is necessary routine maintenance, replacing equipment that is over six years old. It is also a desirable upgrade to satisfy demands for increased capacity and advanced features that support [large world-renowned private university]’s multiple missions.

The changes over the weekend were intended to add the last set of features and capacity to move the healthcare networks to the new core with the already-migrated university networks. Last week, the networking team worked with our networking vendor (Cisco) to validate the new, planned configuration and performed multiple successful simulations with identical equipment in our testing lab. On Sunday, the upgrade was installed at 1:30 a.m. EST, and all indications were that it was successful.  At approximately 7:45 a.m. on Sunday, network stability wavered in the face of some unseen error condition.

On Sunday afternoon, the networking team backed out a portion of the overnight changes as a workaround, and that led to a period of stability for both healthcare and university applications through Monday morning. Monday's additional load proved that not all issues were resolved by the workaround.  

Continuous troubleshooting in consultation with the highest possible level of technical resources from Cisco were not able to identify a root cause, and at 6 p.m. the networking team began backing out the remainder of the overnight changes from Sunday.  Our internal network engineers are now working with Cisco to further analyze the outages and formulate a revised plan for performing the upgrade. Cisco is in the process of building a lab with our exact configuration in order to identify the issue in their equipment or recommended design.

Thank you for your understanding as we work through these complex infrastructure changes.

[name], Deputy CIO, University Technology Services
[name], CIO [large world-renowned private university] Healthcare
[name], Enterprise CIO

Extended (Optional)

EMAIL TO A FRIEND X
Your Email has been sent.