Having worked in the software industry for decades now, none of these "launch failures" really surprise me. Especially when Project Stakeholders are making last-minute "structural changes," that can ripple through the code like 'a large rock hurled into a pond'.
Like termites discovered in a Fixer-up; Like a finicky buyer, in a seller's market ... well you pick the metaphor. The point is, "structural changes" late in a project, are usually bad news.
From the Start, Signs of Trouble at Health Portal
by Robert Pear, Sharon LaFraniere, and Ian Austen, NYTimes.com -- Oct 12, 2013
[...]
One person familiar with the system’s development said that the project was now roughly 70 percent of the way toward operating properly, but that predictions varied on when the remaining 30 percent would be done. “I’ve heard as little as two weeks or as much as a couple of months,” that person said. Others warned that the fixes themselves were creating new problems, and said that the full extent of the problems might not be known because so many consumers had been stymied at the first step in the application process.
Confidential progress reports from the Health and Human Services Department show that senior officials repeatedly expressed doubts that the computer systems for the federal exchange would be ready on time, blaming delayed regulations, a lack of resources and other factors.
Deadline after deadline was missed. The biggest contractor, CGI Federal, was awarded its $94 million contract in December 2011. But the government was so slow in issuing specifications that the firm did not start writing software code until this spring, according to people familiar with the process. As late as the last week of September, officials were still changing features of the Web site, HealthCare.gov, and debating whether consumers should be required to register and create password-protected accounts before they could shop for health plans.
[...]
Some people intimately involved in the project seriously doubted that the agency [the Medicare and Medicaid agency] had the in-house capability to handle such a mammoth technical task of software engineering while simultaneously supervising 55 contractors. An internal government progress report in September 2011 identified a lack of employees “to manage the multiple activities and contractors happening concurrently” as a “major risk” to the whole project.
[...]
Of course when huge bureaucracies are involved, often the Project Stakeholders change from year to year, and even month to month sometimes. Their grasp of what the contractors are really doing, is often little more than bluster, or a facade of episodic concern. Often simply they are not up to the task, of making sure the Project gets built "right." (Assuming they know what "right" really is.)
Imagine trying build an Office Building, without consulting an Architect, an Engineer, or vetting the General Contractor. That would be a "fool-hardy adventure" at best, or a very expensive "learning experience" worse. Sooner or later, you have to tell the Workers {Programmers}, what the features {problems} are, that you expect them to build {solve}.
Well the equivalent happens in large software projects, especially when "no single person" in the bureaucracy is tasked with gathering and VALIDATING the project specification, BEFORE the concrete is poured and the beams are set. As often is the case.
It's much, much easier {from the Builder/Programmer perspective} to make major structural changes, on paper -- BEFORE the concrete trucks show up, than after the fact, when the windows are being installed, and the tile grout-work is setting.
The same idea applies to Software Projects, as it does to Office Construction projects, roughly speaking. In software, as in most things in Life, Major Changes are far easier sooner, as opposed to later. Given the factors of momentum and inertia and interlinked logic.
Believe it on not there is an established "scientific theory" to "software construction." (well, actually several of them: Formal, Waterfall, RAID, Agile, Scrum, etc.) What they all have in common, is the need for an accurate and thoroughly thought out "Requirements Definition." (think "Blueprint," for the analog in "building construction.")
Functional specification -- From Wikipedia
[...]
Overview
In systems engineering a functional specification is a document that clearly and accurately describes the essential technical requirements for items, materials, or services including the procedures by which it can be determined that the requirements have been met. Specifications help avoid duplication and inconsistencies, allow for accurate estimates of necessary work and resources, act as a negotiation and reference document for engineering changes, provide documentation of configuration, and allow for consistent communication among those responsible for the eight primary functions of Systems Engineering. They provide a precise idea of the problem to be solved so that they can efficiently design the system and estimate the cost of design alternatives. They provide guidance to testers for verification (qualification) of each technical requirement.[1]
[...]
Purpose
There are many purposes for functional specifications. One of the primary purposes on team projects is to achieve some form of team consensus on what the program is to achieve before making the more time-consuming effort of writing source code and test cases, followed by a period of debugging. Typically, such consensus is reached after one or more reviews by the stakeholders on the project at hand after having negotiated a cost-effective way to achieve the requirements the software needs to fulfill.
Process
In the ordered industrial software engineering life-cycle (waterfall model), functional specification describes what has to be implemented. The next, Systems architecture document describes how the functions will be realized using a chosen software environment. In non industrial, prototypical systems development, functional specifications are typically written after or as part of requirements analysis.
When the team agrees that functional specification consensus is reached, the functional spec is typically declared "complete" or "signed off". After this, typically the software development and testing team write source code and test cases using the functional specification as the reference. While testing is performed, the behavior of the program is compared against the expected behavior as defined in the functional specification.
Believe it or not,
"I'll know it, when I see it," (as Bureaucrats are known to say)... is NOT a "Requirements Definition."
Imagine what a General Contractor would say, if those were his 'Marching Orders'. Something like, "You've got to be freaking-kidding me."
Sadly private Software Venders do not use that kind of language, often enough. Usually they just smile and nod, check the contract fine-print for their liabilities, and then cash the procurement paychecks.
Well "thankfully," to what degree the current problems with HealthCare.gov, are "Structural" and what are simply "Cosmetic" problems, seems still a bit hazy. In any event, the Project Stakeholders appear to be very well engaged now. Finally, now that the tower's front doors are jammed, and the roof is leaking ... and that the customer lines are forming, around the block.
Pouring "more Server Power" into it, certainly couldn't hurt the cause ...
HHS scrambles to fix glitches in insurance exchange roll out
by Andy Medici, FederalTimes.com -- Oct. 13, 2013
The Health and Human Services Department is scrambling to boost server capacity and fix software glitches that accompanied the rollout of the federal health care exchanges that opened Oct. 1.
The system was designed to support 60,000 simultaneous users but attracted more than 250,000 in its first few days of operation, according to the agency. The agency had based the traffic estimates on the Medicare Part D site, which attracts 30,000 users at the same time.
[...]
HHS employees at the Centers for Medicare and Medicaid Services -- which is in charge of the project -- have been working with contractors to add server capacity and fix any software glitches as they occur, according to spokeswoman Tait Sye. The primary contractors on the project are CGI Group Inc. of Montreal, which won a $93.7 million contract in 2011 to build the federal health insurance exchanges, and Quality Software Services Inc. of Columbia, Md., which won a $69 million contract in 2012 to set up a data services hub that would pull needed information from different agencies.
But we won’t stop until the doors to HealthCare.gov are wide open, and at the end of the six-month open enrollment, millions of Americans gain affordable coverage,” she said.
[...]
Now the coding "glitches" are truly another matter. They may be easily patched; they may be
easily unpatchable.
Imagine an Amazon at Xmas-rush, that suddenly started randomly routing your orders to the wrong drop-shipper ... is that a minor glitch, or a major FUBAR? A lot depends on the "how important" were your orders; and how "fixable" that random glitch really is.
In any event, Amazon in such a scenario, would most likely end up with a significant "black-eye" in the eyes of the consumers, the next time Xmas rolls around.
SO, be ready to break out the cold-packs, Software Stake-holders ...
HHS fixing Healthcare.gov glitches
by David Perera, FierceGovernmentIT.com -- Oct 7, 2013
Information technology workers behind Healthcare.gov spent the weekend fixing coding errors that resulted in customers finding difficulties with the federal online insurance marketplace, reports the Wall Street Journal.
[...]
An Oct. 5 Sunlight Foundation analysis says it's likely that the website's problems "are due to more expensive operations related to the insurance application process itself."
"Checking users' eligibility and filing their applications requires integration with a separate and more complex set of systems -- ones that have little to do with your web browser. Fixing those sorts of bottlenecks can be easy or difficult; the boring truth is that it's hard to say definitively from outside the system. Much harder than carping about uncompressed JavaScript, at any rate," writes Sunlight Labs Director Tom Lee.
Of course there is one "random factor" in this whole equation, that no one yet has the slightly clue how to fix,
-- that of re-programming Congress, so that they are willing and able, to adequately fund, competently define, and intelligently monitor -- the kind of Society, most of us simply expect them to deliver.
The person(s) who solve THAT "technical glitch" will have truly discovered 'the next killer app.'
We could call it simply: A Congress that Works. (We'd sell millions of copies of that app!)