Skip to main content

First, I do not work for HHS nor am I a tech "geek", so I don't have any inside information on what actually went on between HHS and its web contractors that caused or contributed to the problems.  But I do work in government and I do have experience with government contractors, and from that experience I do see some similarities between the problems I have seen with government contracting and the problems with the ACA HHS contracting that are now being exposed.  Therefore, I thought I would share some of my experiences and relate them to what might be behind the ACA web problems.

So if you're interested in some inside info on the flawed world of government contracting and the politically appointed Bureaucrats in government who support it, join me after the jump /.

First of all the amount of government contracting has exploded in recent years.  It used to be that the hiring of government contractors was limited to projects of short duration where there were not sufficient qualified government employees and where it did not make sense to hire and train new permanent employees for projects lasting only a few months.  Now however, thanks in large part to the Republicans zeal for austerity, their disdain for government workers and their belief that the private sector can always do a better job, many of the jobs performed by public employees are now going to private contractors leaving the remaining public employees as middle-men tasked with overseeing and managing contractors.  The logic being that the private contractor can do the work cheaper than the public employee because you don't have to pay for retirement, health insurance and other benefits.  But such logic doesn't take into account the fact that the private sector is profit driven and the fact that often the public employee overseeing the contractor has to revise or re-do the contractor's work meaning you're paying two people to do the work formally done by one.

Sorry to all those private contractors out there, but I had to get that out of my system.  As I've already disclosed, I am a government pubic employee and admittedly have a negative opinion about the rise in government contracting, since I've seen it result in public employees getting laid off or open positions not getting filled.  However, I will do my best to set my opinions aside and just give you the facts about my government contracting experiences.

Story #1:

My first experience involves getting a private contractor with personnel who have adequate qualifications (or just basic competency) to perform the work.  In this instance the work involves highly complex geo-technical environmental engineering that admittedly was beyond my engineering capabilities.  The task went to a "stand-by" private contractor ("stand-by" contractors are firms that are awarded government contracts to perform work on an "as-needed" basis) selected by management.  The selected firm did not have an engineer with adequate knowledge of geo-technical engineering, so they sub-contracted with another private firm for the geo-technical work.  BTW, anytime you sub-contract, you increase the cost by around 10% since the sub-contractor also has to make a profit.

Now the type of geo-technical engineering we are talking about involved reviewing an application for an environmental permit by a private company for a chemical containment structure.  The application contained, among other things, a design for the structure and an analysis to justify the company's claim that it will remain stable under seismic conditions.  This is a highly specified field, and while I had limited knowledge in it, I did know that there were only a handful of qualified experts available, and I knew who they were.  The company who submitted an application for a Permit to build this structure, an application our governmental agency was tasked with reviewing to determine that the structure has been properly engineered (such as to withstand an earthquake), had hired one of these experts, arguably the best of the small group.  But the combination of his knowledge in the seismic field, his extreme self-confidence (he was full of himself), and the fact that he was obviously biased, working for the company that wanted to build this containment structure at the lowest possible cost, I couldn't shake the feeling that I was being snowed.

The remaining experts I knew of existed as individual professional engineers in the private sector and one in the Environmental Protection Agency (EPA).  But the professional engineer we got through the sub-contractor was not one of these experts.  While this person held a doctorate in geo-science and seemed very knowledgeable in matters of soil properties, I soon discovered that in certain critical aspects of seismic engineering pertaining to this proposed containment structure, he knew less kowledge than I did, and far less than the company's expert, to the point where I started to feel that he was just as snowed by the guy as I was.

So the government contractor procurement process does not allow you to pick the experts  you want, so you don't always get the most competent people.  Sure you can establish minimum qualifications for the expert you are seeking, but the government contract laws are structured in a way that does not allow you to establish a specialized set of qualifications that only one or a few experts can met.  You have to keep your qualifications general enough to allow a significant number of contractors to participate in the process.  Then when it comes to awarding a contract, who gets it is most often related to costs not expertise, and when the contractor uses a sub-contracted expert, as in this case, you have no control at all over who that "so called" expert is.

Getting to the point of this story (finally) the government contracting process doesn't always result in getting the best and the brightest experts you need to get the job done properly.  While I can't prove that this happened in the case of the building of the web site because I was not inside that process, there are signs which suggest that the Federal government might of ended up with a web contractor of inadequate expertise which contributed to the failed roll out.  One sign that this may be the case is that news reports now say that the government has gone out to contract with the "best and brightest" in the web field (e.g., Google, Yahoo, etc.) to fix the failed web site, which leads to the obvious conclusion that they did not hire the "best and brightest" to build it.

Story #2:

This second story is about the relationship between government contractors, high ranking government bureaucrats who oversee them and DEADLINES.  Its sort of an "Emperor's New Close" type story in that no one wants to tell the "Emperor" or Head Haunch-o that more time is needed if the job is to be done right.  It deals with the project discussed in Story #1 where a government contractor was tasked with reviewing an permit application for a chemical containment structure (a task which used to be performed by government employees like me).  Because of the complexity of the structure's design, the application consisted of thousands of pages of technical drawings and design details, as well as detailed environmental evaluations of the design.  The idea was for the contractor to review the application and for government staff (me) to review the contractor's review, in order to make sure the application is technically adequate for public review.  While it sounds like this creates a situation for a thorough review of the application, such a review is only possible if high level government managers are interested in a thorough review.  However, in this case they were far more interested in completing the review quickly before the deadline, so they could tell the Grand Pu-Ba that the project was ready for public review and receive his "pat-on-the-back.

Throughout the project, the "Middle Manager" (the guy looking to impress the big cheese by getting the review done quickly) would put unreasonable deadlines on the government contractor for completing review of each piece of the application.  Deadlines which were arbitrary and which were not based on the size and complexity of the document to be reviewed.  Furthermore, when the contractor pointed out flaws in the Company's application, the "Middle Manager" would sometimes downplay there importance and brush them off.  Not because they weren't real flaws in the unit's design in some cases, but because they would take too long to fix.  So as time went on the contractor became more and more aware that the "Middle Manager" did not want to hear about any flaws, he simply wanted the contractor to say the application was acceptable, complete and ready for public scrutiny, regardless of its content.

To compound the problem, the "Middle Manager" gave even less time for government staff (me and my team) to review the contractor's work and double check the technical adequacy of the application.  This left staff with two choices, either do a quick inadequate review and meet the deadlines or perform a thorough review pointing out each and every flaw and keep missing deadlines.  Initially I opted for the latter option.  The fact that staff was taking longer then the "Middle Manager" arbitrarily was allowing and pointing out flaws in the application he did not want to hear about, quickly lead to a situation where staff had to face the "Wrath-of-Kahn" (the"Middle Manager") on a regular basis.  Rather than facing such wrath, staff, except for me, decided to simply tell the "Middle Manager" that everything was fine and the application ready for the public, without really reviewing any of it.  However, I still plodded along doing the thorough technical review I felt the public expected me to do, and which as a P.E. I felt I was ethically bound to do.  When the "Middle Manager" finally had enough of the added time I was taking and of all the flaws I was pointing out, he relieved me of reviewing all but a small portion of the application, and handed the remainder over to his "right-hand-man" who had no problem with "blessing" the contractor's work and the application without really reviewing it.  So now we are left with what I know is a technically flawed application being provided for public review which is likely to crash and burn soon after its published and make the government look incompetent.  Sound familiar?

All because a "Middle Manager" decided there was no way in hell he was going to ask the "Grand Pu-Ba" to extend his deadline to provide adequate time for a more thorough review of the application.  Instead, he wrongly decided (in my view) to meet the Pu-Ba's deadline and present a flawed application, crossing his fingers and burying his head in the sand, in hopes that nothing would go wrong.

I can't help but think that something very similar happened with the roll out of the web site.  Somewhere between the low level programmers who were building the site and President Obama was a "Middle Manager" (or group of "Middle Managers") who on October 1 had their fingers crossed and their heads buried in the sand.  This seems likely to me since I don't think the President would have allowed the site to go live on October 1 if he were told that there was a good chance it would crash and burn.  It also seems likely since E-mails from low level programmers have come to light warning their superiors of the flaws in the site before the October 1 roll-out.

So it seems entirely plausible to me that some "Middle Manager" (or group of "Middle Managers"), put pressure on programmers to get the site ready on time, ignored all the warnings of potential problems from programmers, and told those above him/her (or them), including President Obama, that the site was "fired up and ready to go".  Why would he/she (or they) decide to lie about the site's readiness?  I can only speculate, but two reasons come to mind that might have contributed to such a decision.

First, the "Middle Manager(s)" were undoubtedly aware of the political ramifications of having to delay the roll-out of the web site.  Republicans would have likely pounced and pushed hard to have the mandate deadline extended.  And the President would have been angry and rightly so.  The "Middle Manager(s)" could have felt that the President would go looking for someone down the chain who was responsible for the delay and possibly fired he/she or them.

Second, the "Middle Manager(s)" were likely not all that familiar with web site development and did not understand the seriousness of the flaws being pointed out by the low level programmers.  They probably thought that they were small glitches which could be fixed quickly and painlessly after the site went on-line.

Who that "Middle Manager(s)" is/are is anybody's guess (feel free to speculate in comments).  But I'm sure a good journalist could follow the e-mail trail from the programmers and find out who ignored their warnings.

Well those are my stories and how I think the failure of the web site was allowed to occur.


Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags


More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Tip Jar (10+ / 0-)

    "Some men see things as they are and ask, 'Why?' I dream of things that never were and ask, 'Why not?"

    by Doctor Who on Sun Nov 17, 2013 at 06:21:11 AM PST

  •  I think you're absolutely right about the role of (1+ / 0-)
    Recommended by:

    the contractors, and I personally think "privatization" in general has been a disaster for gov. services and for the people that need them. It's added the "make gov. small enough to drown it in a bathtub" to the personnel selection (Brownie, for instance), and affected the expectations of the parties involved, and as the system has added cost, it's been a "faith-based" element which has fueled denial of the true cost and has encouraged creative accounting, etc. in order to hide the problem under a rug.
    In health care dot gov you've got to also add the effect of the active gop obstruction and sabotage.
    I'm sure there will be a whole slew of books written on this subject in coming years. I hope we're able to focus real media attention on the parties who are actually responsible, and how to fix the process.

    I think the Pres and Sec Sibelius knew they couldn't wait past the deadline and made a calculation that they could ultimately win the political judo match over who was to blame.

    Ultimately they'll get it fixed. The fact is this rollout isn't any worse than some other big programs, what's different this time is that you have the media cutting and pasting gop talking points and you have the President deliberately taking the heat because he isn't running for office again. In the long run that may be very politically shrewd because they'll fix the sytem, millions will be better off, and this whole furor will be seen as more of the irrational "hatred for Obama" of the gop.

    You can't make this stuff up.

    by David54 on Sun Nov 17, 2013 at 06:45:28 AM PST

  •  Unfortunately, to the great unwashed, sound (3+ / 0-)
    Recommended by:
    pelagicray, Sunspots, Doctor Who

    bites like "government is the problem" trump all the well thought out, rational explanations in the world.  And that is why the general public thinks contracting out is just swell.  We have become a nation of tweets, or as I prefer, twits.

    Ted Cruz president? Pardon my Vietnamese, but Ngo Pho King Way.

    by ZedMont on Sun Nov 17, 2013 at 06:56:25 AM PST

  •  Consulting/contracting firms don't send their best (2+ / 0-)
    Recommended by:
    Dr Erich Bloodaxe RN, Sunspots

    ...& brightest to federal jobs which is pretty much part of the deep discount negotiations the gov't demands. No secret there. In the private sector when  a potential cons/contr firm had another large contract siphoning away staff it would work against that firm anticipating quality of staff assignments to be compromised, except if the potentially competing contracts were with federal agencies.    

    You're raising a real issue with its downstream problems interfacing with career staff but it's a pretty generic management issue, ubiquitous but no less real and is usually dealt with directly up front during negotiations, planned for and either accommodated or avoided. So the real issue you're raising is quality - project management and the overall quality of the management - from contracts to IT to post-implementation.  After all, the outsourcers do leave at some point, hopefully after the rolling out is done, and somebody does inherit a patched up behemoth to keep the lights on and get ready for next year... A hand off to career staff from a collegial but haphazard horde of cons/contr can leave behind a mountainous steaming pile of sh...

    •  Contracts have to have built-in consequences (1+ / 0-)
      Recommended by:

      But there's a lot of resistance to that, inside government.

      •  Bids would come in higher if that were done (1+ / 0-)
        Recommended by:

        That is not necessarily a bad thing if quality and on time performance improved for important projects.

        The most important way to protect the environment is not to have more than one child.

        by nextstep on Sun Nov 17, 2013 at 10:42:10 AM PST

        [ Parent ]

        •  Cheaper than repairing a disaster (0+ / 0-)

          and it makes realistic planning possible.  Speeds up the solutions.  

          Often it's easier to start over from scratch than to try to trace and correct all the errors in a bad system.   All the parts often affect all the other parts - and usually disasters aren't decently documented, to make it even harder.

  •  This, to me, is the real 'big story' about the (3+ / 0-)

    failure of the ACA website rollout.  Not 'a government that can't do anything big right', but a private sector that can't be trusted to do important work for the government, a private sector largely inserted as a way to discredit the public sector by shifting blame for problems that originate in the private sector onto government's shoulders.

    The 'government' did not fail here, except as intended by Republicans who fight tooth and nail to funnel public finds into private contractor pockets while painting 'government' with any failures that arise from these private contractors' incompetence (ACA) or overzealousness. (NSA)

  •  Your examples largely gave me a flashback to (4+ / 0-)
    Recommended by:
    Doctor Who, ybruti, Sunspots, nextstep

    old days when I was really looking at the problems and successes of federal contracting. Some observations—and one reason I lost daily contact with this stuff is the absolute ruin and debacle brought on federal contracting in the Bush years, particularly Iraq—from a distance of quite a few years.

    Your story #1 struck me as an example of an agency not being particularly skilled at contracting. Many are not for reasons I'll mention below. Unless things have changed far more than I think and see in some quick checks, most of these "failures" are failure of the agency's management and contracting office to use all the contracting tools in the law and regulations. Typically in my experience agencies get used to contracting a certain way, develop skills for that and sometimes wear blinders for other options. As can be seen in this piece on the FAR Part 15 Process Model you have to design the entire process to drive the required results—and yes, shit does happen sometimes even when done well. Still:

    One of an acquisition team's most important tasks is designing efficient competitive processes.
    If an agency does not fully understand its requirements and the process, then design and drive the selection process you will get contacting problems. Some agencies, due to internal prejudice or sometimes direction from above, simply eliminate the processes required to get the quality contracting they require.

    This thread, down in "Configuration control is one of the thorniest," touches on the problems with government management that does not understand consequences of failure to follow contracting/development best practices. I personally dealt with high level management so familiar with what is now called IT that they could not turn on the PC to get their internal e-mail but presumed to "require" schedules and performance that went beyond "cutting edge" into certain failure. Some had "read somewhere" they could get certain performance that had not yet reached prime time. Then they were shocked, just shocked, when the cost figures came in—or in one case every response noted "not possible with current technology" generating a whine about men on the moon and such.

    The expert use of the FAR and body of knowledge long existing on how to do these things reasonably well is as complex as many a technical matter I've known. So, why do we not put as much effort and money into acquiring and keeping an expert acquisition force as we would a technical team? Why do we keep tossing complex technical matters of IT acquisition into agency contracting offices that rarely if ever have done one of the complexity the suddenly must manage? Why do we reject what is discussed in "A digital core for the government?" (where I comment on some issues of skill and management meddling)?

    With some of these contracting issues it is as if the stakes are those of a pro championship game yet we toss the ball to a farm team, good in their right, but not experienced in the big league. I think there is merit in that "digital core" the Brits seem to have formed.

    The only foes that threaten America are the enemies at home, and those are ignorance, superstition, and incompetence. [Elbert Hubbard]

    by pelagicray on Sun Nov 17, 2013 at 07:22:58 AM PST

  •  I worked for several governments: US, California, (4+ / 0-)

    Australia, West Virginia, and UN. The issues you discuss occur in al of them.

    However, I think that the rollout was bound to "fail". They had to put together a site that would be handling millions of visitors on day 1 and that worked with dozens of other sites. They could have had the best and most experienced programmers from Google, Amazon, and Facebook working on it and there would have been significant glitches.

    I was in charge of a website for WVU several years ago that would be used by several hundred users. We spent years on the software and months testing it. when we opened it up there were all sorts of problems. People were trying to do things that had never occurred to us. In the words of the old engineer, "Every time I think I have finally designed an idiot-proof system someone comes up with a better idiot".

    The real test is whether they can fix the system rapidly. If it was well designed it will be just a matter of plugging holes.

  •  On management, a "PS" to my comment above. (1+ / 0-)
    Recommended by:

    Read the proposal by Steve Kelman and the comment thread. The proposal boils down to contracting staff pledging

    The idea is that each contracting professional in the office chooses a contract through which he or she will be buying something over the next year that is the same or similar to something bought in the past. They make themselves a personal pledge – like the goals regarding the contract this contracting professional at the meeting spoke about – to find at least one way to improve the process and (hopefully) the results of the buy the next time it is done.
    With two comments near the top noting (my emphasis):
    How about OFPP "pledge" to require that any OFPP Administrator, Agency Senior Procurement Executives, Heads of the Contracting Activity, and Agency Chief Acquisition Officers have a solid, credible background in contracting (as a former Contracting Officers, Directors of Contracting, etc.)!!!! The field is full of leaders who have not walked the walk in the federal contracting arena, yet want to make critical decisions about contracts in their organization. The failure to address this issue contributes to the decline of the federal contracting personnel.
    . . . why make the suggestion just to contracting professionals? What about the people who define requirements and write specifications and statements of work, develop proposal evaluation factors, evaluate proposals, provide input to cost analyses, perform contract quality assurance, and manage programs? Acquisition is a team effort, right? Why focus on only one part of the team? What if the technical folks pledged to meet their own schedule milestones instead of coming in late with their input and then pushing COs to meet obligation deadlines?
    As with too many things in our society there is much whining about a "broken" thing and damned little interest either in the society as a whole to demand and in the politicians to fix those problems. These discussions are much the same as those I was involved with twenty or more years ago. As with so many of our national "problems" I became very tired of seeing improvement and then wreckage.

    Until we as citizens pay real attention the cycle will continue until ruin is evident enough to start a crawl up again—or maybe just descend into third world status.

    The only foes that threaten America are the enemies at home, and those are ignorance, superstition, and incompetence. [Elbert Hubbard]

    by pelagicray on Sun Nov 17, 2013 at 08:14:36 AM PST

  •  Power trumps expertise, all too often (1+ / 0-)
    Recommended by:

    It's too widespread an underlying attitude these days, that if you're in control of something, reality is whatever you say it is.  Management is selected for agreeing with that (sucking up), and terribly often doesn't understand what it is supposed to be managing.  

    That and the emphasis on money, the idea that cost is all that matters, inevitably produces disasters.

  •  You didn't touch on the problem of underbidding (1+ / 0-)
    Recommended by:

    Companies will deliberately underbid a contract in order to be awarded that contract (20 years in the army and I became familiar with that ploy), with the hope of fulfilling the contract on a wing and a prayer.

    Furthermore, CGI, the company that was awarded the contract for the healthcare web site has a cadre of executives from a company that has a history of failed IT contracts, including government contracts.

    My impression is that CGI makes its money from big contracts, using the awarding of those contracts to get more contracts.  When you have companies that exhibit their "experience in getting government contracts" as their experience in the field you have a recipe for a failed outcome.  CGI is based in Montreal.

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site