Skip to main content

I'll start by introducing myself, because it has some small relevance to the topic: Hi, I'm Welt, long time lurker, very occasional poster and I almost never write diaries.  I'm a software engineer that has been in the industry for 15+ years.  Having cut my teeth during the crazy days of the dotcom bubble, I've worked at quite a few places. From giant chip manufacturers to mom and pop startups.  Five of my fifteen years were spent working on an earmark funded defense industry related startup.  Hindsight being 20/20 it was a project destined to fail from day one, for a variety of reasons, none of them being technology.  While this is neither here nor there, because of my time in the defense sector (so to speak), I look at much of our defense budget as wealthy white guy welfare.

I recently stumbled upon this write-up on some of the stumbling blocks that Healthcare.gov went through that has some excellent specifics about the process failures in government contracting that lead to the failures of Healthcare.gov (and plague the government):

If you want an IT project to fail, allocate a bunch of dollars to it. If you want it to be a total disaster, do that, plus let Congress design the requirements and set the deadlines. The disconnect between Congress’ view of software and the developer’s view of software has never been more vast. Then you have a force multiplier: Congress’ seemingly willful ignorance over how the procurement process actually works.
If you'll join me below the fold, I'll give you my somewhat snarky take on what I believe is wrong with the process:

Imagine doing your job while reporting directly to 536 bosses. Each is the equivalent of a VP level position and they each have a staff of workers who exist for no reason other than to help their boss advance within the company (i.e. they are not there to directly assist you in your job, they are only staff that exist to support their boss's job). About half of those 536 bosses are directly at war with one another. Everything in your job requirements is put together by a bare majority consensus of the group of warring bosses, and thus the process creates extremely vague requirements that do not in any way to take the process of actually doing your job into account.

Now add to that not only will every piece of your work will be checked by those bosses, but also every step you took towards making the work happen will be checked, and every step along the way will be scrutinized to make sure it was the most efficient possible way to make that step and followed company procedures with zero deviation. Not only will all of the bosses be scrutinizing your work, but the staff of all those bosses will be scrutinizing your work AND the staff's job is to find things wrong in your work and they are rewarded only for finding problems in your work (either the process or the end product, wherever they can find issues, it doesn't matter).

Further, each of your bosses has one to many other people who are vying for that bosses position. And each of those new job candidates has THEIR OWN staff that exists to scrutinize your work to make sure it doesn't reflect poorly on ANY ONE of your bosses. Also, each of your bosses and each of those people looking for your bosses' jobs has a budget (ranging from thousands to millions of dollars) that exists ONLY to look for issues with your work, no matter what you cannot tap into that budget to help your project.

Now finally imagine that your job is working with computers, and you have to do all of the above described work while following thousands of company guidelines and best practices, most of which were created before computers were common place (and many of which were created before computers existed).

All of the above may come off as a condemnation of the government and a call for privatization, but that couldn't be further from the truth. The private sector is no less prone to systemic failures of this kind than the public sector. I believe as long as we have polarized politics gnawing away at our government, it will be almost impossible for good people to get good things done.  That is not to say "both parties do it!" or anything of the like, there is clearly one group of anarchists that want to get in the way of progress, but we need to prepare ourselves for fiascos like Healthcare.gov's rocky launch for as long as we have these anarchists in our midst doing everything they can to destroy government.

Originally posted to welt on Fri Oct 18, 2013 at 09:59 AM PDT.

Also republished by Community Spotlight.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  I worked on the software acquisition strategy (22+ / 0-)

    and quality control specifications for Air Force some decades ago.

    They realized a long time ago that projects need essential structure and management of such - via whatever methodology and tools you will employ - to at least verify where you are at any given time.  Even so, they wanted a means to evaluate proposals up-front, gauge the likelihood that bids coming in had some basis in reality . . . and have a firm layout of how the claims being made in each bid could be tracked as the project moves along, especially between major milestones.

    Nonetheless, waste still happens and public + private business, requirements are not always managed well (often due to politics or hierarchical reasons) and scope creep happens, or definition of goals is never made clear until late in the game, etc.  Even with rapid prototyping or Agile methodologies being utilized.

    It did sound like the time from requirements to implementation was very short for the federal system - I can see how that rush may also have related to incomplete requirements clarity for awhile, because some time those offering the reqs need to see early work in order to modify their views of what truly is needed, etc.

    "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

    by wader on Fri Oct 18, 2013 at 10:11:18 AM PDT

    •  The delay was also created by nobody wanting to (43+ / 0-)

      do anything until 2012 was decided.  After all it's wasted work if Prez Romney and R-dominated congress repeals the law.

      So presume nothing real occurred until Nov 2012.  From the sound of it, things really didn't get out of discovery until spring this  year, the time in the middle likely eaten up by finding a vendor willing to do the job under the conditions described, and having said vendor vetted under all the procurement laws and such.

      In my company, 6 months discovery for choosing a new vendor from multiple choices is not at all unusual.  Then until you have a vendor, any work you did is largely wasted because they won't understand what you're saying until requirements are translated into their methodology.

      •  This explains the seemingly tardy initiation of (17+ / 0-)

        the project.

        And I think the ACA opponents knew this and introduced FUD into the start-up of the exchanges.

      •  This should have been a 120-day project. (2+ / 0-)
        Recommended by:
        walkshills, imchange

        In fact, easy to do.

        Instead of building it as a data warehouse implementation, they built a one-at-a-time web with individual queries off to mulyiple databases for each transaction.

        Dumb as a fence post.

        •  There are probably reasons they can't consolidate (12+ / 0-)

          the data. For example, HIPAA imposes very strict requirements on what healthcare-related data you can retain and how you need to secure it.

        •  No way this is done right as a 120 day project (20+ / 0-)

          You have a hell of a lot of personal information with lots of regulations about how they can be exposed and used.   Said information is stored and secured in different government agencies and perhaps credit rating companies.

          There is a reason companies like Cybersource pretty much do nothing but manage the legal part of storing credit card numbers...individual companies have to be audited YEARLY and compliance isn't easy.  I don't imagine things like income verification and identity verification are any less tightly locked down.  Unlike a private company you can't just ignore it and hope you don't get caught, or pay a slap-on-the-wrist fine if you get caught by an understaffed regulatory agency when hackers or bad actors in your own company get into it and expose it.

          You have dozens of insurance companies that have to be stitched in.  Most of these companies have their own ways of doing B2C and a lot of this will be completely out of their comfort zone...they'll all push back and cause problems.

          Even if you were to try to store everything in an ERP-like transactional database (not a data warehouse, you're doing two way transactions here, not simply dumping data in and reporting it), connections from everybody to and from the ERP database to populate it and keep it up to date, and to read data from it into the local insurance companies have to be written and debugged.

          There is a reason IT has been moving away from ERP models to service oriented architectures (which they appear to have tried to build, even if many of the services are buggy).   You don't have to hand craft every freaking thing, and there are less moving parts to break down.  (you'd think a centralized database to store the entire thing end-to-end would be simple.  It is not unless it owns every scrap of data from end-to-end and communicates with nobody but the people involved in the transactions it services)

          I've got over 20 years experience doing exactly the sort of thing you are proposing with complex and scattered systems..geographically, in tech they're built with, in time they were built under differing methodologies.   If you rush this sort of thing...you get exactly the kind of stuff they're dealing with now.

          And actually, 120 days is about what they spent on this, near as I can tell.  From what others have said the vendor and requirements weren't nailed down until this spring.  6 months (180 days) tops.

          •  It seems even many I/T folks here aren't getting (12+ / 0-)

            those fully relevant points: we don't know how many legacy systems they've had to stitch together in order to make this work.

            As you've offered, they probably went with service interfaces, because it's possible that some of those backend systems require real-time pulls based on heuristics for specific parameters provided - and, those systems may be pulling dynamically from other systems, as well.  Many times, the logic cannot be easily duplicated in a single system and all data dumped nearby for easy, fast access  - you just use the backend systems and hope that they can meet your service level requirements.

            It's amazing that it's processing as many applications as it has thus far, there must be lots of reuse from existing systems that gave them a leg up (which offers its own integration challenges, as implied here).

            "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

            by wader on Fri Oct 18, 2013 at 08:03:22 PM PDT

            [ Parent ]

            •  Except that the insurance front-end (2+ / 0-)
              Recommended by:
              Chi, wader

              software for enrollments and the once-a-year policy changes has been standardized for at least a decade.

              •  Except the rules have all changed (3+ / 0-)
                Recommended by:
                Oh Mary Oh, imchange, wader

                Just to pick one...you throw out all the stuff for medical history because they can't discriminate on pre-existing conditions.

              •  I suspect front-end software is not the biggest (1+ / 0-)
                Recommended by:
                waterstreet2013

                issue in their growth pains.

                Even so, the requirements for this system appear beyond standard enrollment setups that I've seen - perhaps due to political requirements going against the grain of industry common practice and (de facto or formal) standards.

                "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

                by wader on Sat Oct 19, 2013 at 04:33:20 PM PDT

                [ Parent ]

                •  Here's Some of the Code... (2+ / 0-)
                  Recommended by:
                  wader, waterstreet2013

                  Javascript ACA Code

                  = 'Error mapping the qualify enrollment group membersException in calling SubmitCustomEnrollmentGroups service'; resources['ffe.ee.shared.exceptions.500.20009'] = 'If retrieve Plan compare/enrollment groups data service written errorException in calling RetrieveNonFaPlanReviewConfirmDetails service'; resources['ffe.ee.shared.exceptions.500.20008'] = 'If retrieve Plan compare/enrollment groups data service written errorException in calling ConfirmNonFaEnrollmentTransaction service'; resources['ffe.ee.shared.exceptions.500.20007'] = 'If retrieve Plan compare/enrollment groups data service written errorException in calling ConfirmEnrollmentTransaction service'; resources['ffe.ee.shared.exceptions.500.20006'] = 'If retrieve Plan compare/enrollment groups data service written errorException in calling RetrieveQHPPaymentPortalData service'; resources['ffe.ee.shared.exceptions.500.20005'] = 'If retrieve Plan compare/enrollment groups data service written errorException in calling RetrievePlanReviewConfirmDetails service'; resources['ffe.ee.shared.exceptions.500.20004'] = 'If retrieve Plan compare/enrollment groups data service written errorException in calling RetrieveTaxHouseholdAPTC service'; resources[
                  Just a sample.  Remember, none of this is needed with Single Payer.  My Medicare card arrived a day before my 65th birthday.  I took my birth certificate into Social Security and signed the card!
                  •  I wish we were complaining about a Single Payer (0+ / 0-)

                    system . . .

                    "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

                    by wader on Sat Oct 19, 2013 at 05:28:21 PM PDT

                    [ Parent ]

                  •  Bingo !! (0+ / 0-)

                    None of that is needed with a data warehouse implementation.

                    If you have 500 distinct conditions to be checked (or retrieved) then the DW deisgn does them in parallel.

                    The shyte-code that uses hyper-complex queries ends up seeing the SQL processor do the conditions in series -- one after another for the 500 of them.

          •  That's off the shelf software (0+ / 0-)

            with proprietary security included.

            You buy it, not write it. Similar to doing sales tax in retail apps.

            •  Been there done that (6+ / 0-)

              "off the shelf software" still has to be integrated with everything else.

              "off the shelf software" that does taxes and credit cards has literally hundreds of settings and configurations, and you have to learn how to set it correctly for what you are doing, interpret responses, handle error messages etc.

              Just as an example, Cybersource will tell you very precisely why a credit card failed.  You can not just expose that to the customer because hackers use it to fake credit card numbers or use iterative methods to guess things like security codes.   Pretty much every possible response has to be mapped to "it worked"  "it didn't work" "the system is down".

              Don't even get me started on the complexity of tax software.  You have different rates in different states and even different counties for services, goods, freight, fees, etc.  You usually have to write exception code to handle a zip code that returns more than one county.   You need regression tests to handle all of this or your taxes end in the wrong bucket and you get sued.

              Finally there is installation.  Most software isn't like a windows "wizard", doing most of the work of a proper install for you with "download" and "run executable".  It is a giant mass of the equivalent of zip files, with terse instructions from the vendor that leave you with some binaries installed on your host, and phone-book sized manuals on configuration options.   There is a reason why the vendors make a lot of their money on consultants, to actually make the mess work in a given environment.

              Any piece of software complex enough to help with a complex business (without being simply unable to do things that you need it to do) will never, ever, "just work" out of the box.

            •  The reason for the above rant (12+ / 0-)

              Is I just spent the last year of my life working to make a "simple" "off the shelf" technology work without disrupting deadlines for literally dozens of other functional groups, because management believed as you seem to that when the vender said "it's simple and easy" that they were actually telling the truth.

              So they assigned a sysadmin and a dba to get it done, two individuals that had no time to even read the manuals, much less learn how to use the tool.

              My colleagues covered my work.  I read the manuals, talked to the consultant, worked out automation (we had a very complex use case as our first rollout), rewrote it after testing in dev, trained sysadmins and dbas in two datacenters how to do it, rewrote it again after we tried out the deployment in test, rewrote it again after we found flaws in the product that were not documented, worked out a strategy for data that didn't work with the product, did some final touches again when it interacted with the prod environment.  

              We finished just in time for the first adopter.  The stuff I was supposed to be working on in that period got dumped on the only other person on my team, so his workload was also doubled.

              The last time I had to install software that complex was back in 1999, when we had experienced a decade of expansion in IT, instead of a decade of belt-tightening followed by a recession.  I had two junior programmers working with me, a project manager, a consultant from the vendor, an independent consultant and me as lead developer.  We rolled out in half the time with no defects but it only worked for the early adopters and it took two years of tinkering to really bring as many folks onboard as we are trying to do in a big bang this time.

              The architecture and software we deployed then is still in use and survived an attempt to find a replacement vendor just a couple years ago.  I know what a successful installation of vendor software looks like and what an clusterfuck scramble because of bad management decisions and no staffing looks like.

              IT makes it work either way, eventually, but it goes a hell of a lot smoother with realistic assumptions and staffing.

              •  I hear you (3+ / 0-)
                Recommended by:
                imchange, wader, waterstreet2013

                I spent most of a year working at a company that had installed an off-the-shelf inventory control system that was intended for manufacturing. The software company had a team of programmers onsite for months tinkering it into working. When I left, it still had bugs, but it mostly worked.
                (Part of the problem was that the company owners wanted it to work exactly like the paper system they had put together. They didn't think through what that system was actually doing and how the pieces went together; some parts of the software version would have worked better when done just a little differently.)

                (Is it time for the pitchforks and torches yet?)

                by PJEvans on Sat Oct 19, 2013 at 08:21:54 AM PDT

                [ Parent ]

                •  LOL (0+ / 0-)

                  "the company owners wanted it to work exactly like the paper system they had put together"

                  I'm tempted to call them "idiots." My first talk to management groups installing Oracle emphasized that the system would not look like the old system, not by process or screens or check points.

                  Modifying a top-level OTC package to look like a semi-competent, older bag-of-onions system is idiocy. It wastes lots of money.

                  •  It isn't that simple (0+ / 0-)

                    Businesses do what they do because it works.  Changing to match a vendor's vision of how they should be doing their business at minimum involves a massive disruption - you have to redesign your data structures, retrain your workforce (warehouse, factory, sales, procurement, logistics etc) etc.

                    You also have to meet contractural obligations.  As an example, our customers have contracts with us, they have contracts in some cases that predate us, were made with companies that no longer exist but whose responsibilities we picked up.   Also our products become obsolete within about 12 months, in my industry Moore's law is for pussies.  That also affects pricing structures.  This is translated into a pricing system that is extremely flexible.

                    Oracle ERP's pricing system isn't up to the job.  For a while we had a homebrew system but eventually (with a lot of pain and very slow adoption for reasons mentioned in the first paragraph) adapted to the Vendavo system.  Which doesn't play especially well with Oracle's internals.  So we're still using our old custom pricelist and special pricing authorization logic inside the ERP.

                    That's just one tiny part of the business.  Credit checks are managed by one product, taxes by another.  Front ends to B2B, B2C and CSO are all different vendors, adapted to our needs.   They communicate with each other using a mix of SOA, older APIs, direct database links and message oriented middleware.   App servers and web servers add more vendors.  That doesn't even consider the factory systems that supply inventory, which, among other things, capture a couple dozen test results during fabrication helpful for tying field problems to individual production runs and to gather metrics on yield, scrap and long term product quality that informs future manufacturing decisions.

                    To create an order, ship it, fulfill it and get paid, an order involves products from at least 7 vendors, plus several apps we still wrote and maintain ourselves because NOTHING on the market approaches the feature set that evolved in house.

                    Or to paraphrase your last paragraph.  Assuming an off the shelf ERP can manage OTC in a business of any size is idiocy.   Assuming any off the shelf vendor software can fit into the complexity that is a modern global manufacturing system without significant effort is also idiocy.

                    •  Not exactly the optimal solution. (0+ / 0-)

                      I've written "proxy" style modules using Oracle's template to reach existing/external/other_vendor pricing. The key is using the Item Master system -- set quantity=0, set the switch allowing 0-quantity pricing, define a sub-inventory that matches your price list.

                      The sub-inventory can be a real table or a view that uses the "proxy." One can use another program altogether -- same as using a c program to set up a sales tax hook.

                      Further, this is optimistic: "Businesses do what they do because it works."

                      Business do what they do because nobody's got the balls to take the responsibility to make changes to improve the process. Very few processes remain anywhere near optimal as technology changes over time.

                      I wrote a planning system for The Hartford back decades ago. It survived for 15 years. Same hardware. At the time I thought that was wonderful, but really they were just too cowardly to stand up to ITT and insist on building a new spreadsheet-driven system.

                      There's also the issue of experience. Maintenance guys are generally not up to it for building new systems. The range of skills isn't there.

                      •  Not so much optimistic as self-evident (1+ / 0-)
                        Recommended by:
                        waterstreet2013

                        If what a business does fails to work, the business fails.

                        It might not be the best way, but it does actually work.

                        Change is disruptive.  It is not just a matter of courage.  It is a matter of risk.  The more useful and stable a legacy system or business process is, the more resistance there is going to be to change it.

                        As a rule, if you are going to change an existing business process (which includes most major software changes) the new process has to have some significant improvements over the old to encourage adoption and to allow the folks who are trying to do their job to overcome the inevitable problems and friction that occur whenever doing anything new.

                        If you are changing for the sake of change, you get a lot of resentful people who will passive-aggressively resist the change even if there is some future improvement payoff.

                        I've been in the role of business support, developer, system architect, process engineer (in several methodologies), QA guy and techie consultant.  I've supported my company transitioning its multinational manufacturing tech business from a "me too" product line to one of the two unquestioned leaders in the technology, absorbing or destroying what was literally hundreds of competitors at the start of my career to what is now a handful.

                        I've seen every argument for change or not change that you can imagine.  I've seen management get duped twice by vendors that couldn't deliver at all (so we had to build the whole thing in-house from the wreckage), one vendor that outright lied about capability, an 20ish year long relationship with Oracle and their entire product line, watched vendor after vendor get bought out and their product abandoned by the parent company.  I've also seen some really cool stuff written both in house and by vendors and stitched in.

                        We had a building full of order management people when I was hired.  Now we need only a few, to manage a business that has expanded 100 fold.  I've seen a contract system go from paper files to spreadsheets to a database to an actual half decent contract management system (once we got past the vendor that lied sigh).    I've seen factory systems that are absolutely astounding built and maintained by utter cowboys (that later went to more disciplined agile methodologies) because those systems must constantly change and can NOT have downtime (minutes cost millions, literally).      I've seen our product architecture change to match new marketing and promotion requirements to get us into the 21st century that required touching (and regression testing) nearly every IT system in the company.

                        I've seen financial systems running the same chart of accounts for 15 years because even though our business has transformed it is so disruptive to change it we had to have a major ERP upgrade occur simultaneously to even consider the risk of touching it.

                        It isn't a matter of being timid or a coward.  It's a matter of not fucking up in a way that costs the company millions of dollars in lost contracts, or causes an entire production run of defective product, or results in legal action taken against us, failing to meet regulatory requirements in a half dozen competing jurisdictions or pissing off your entire salesforce by screwing up their commission compensation formula.

                        As for maintenance guys not having the skills to build new systems, that might be true of 24x7 type support individuals who are trained to a different skills set, but where I work the "maintenance guys" are called "subject matter experts" because they know both the business and the tech inside and out.   They're the people that keep exposing all the missing functionality in what most of the software companies try to sell us, and the ones that figure out how to work in new products into our existing systems without breaking it.

                        •  "Subject matter experts" are not (0+ / 0-)

                          database engineers or acceptably knowledgeable for how to structure programming systems.

                          "Disruption" is a common excuse for running noncompetitive money-losing systems.

                          The old systems do their financial management functions well enough. Process management, not so much. Quality and value go lost so the whole company becomes vulnerable.

                          The usual result is that the laggard is absorbed in an M&A takeover. Crappy management systems make the company a target.

                          •  I've done process management (1+ / 0-)
                            Recommended by:
                            Oh Mary Oh

                            Years of it.  I've even made it work.

                            I've seen theories come and go.  Continuous Improvement, TQM, ISO 9000, Six Sigma, "Do more with Less", avoid hidden factories blah blah blah.  Trained in all of them at one time or another, actually did the "Black Belt" thing for a couple years.

                            The bottom line is that there are always "low hanging fruit" after a few years of using any methodology that are improved by using some other methodology.  Turns out different categories of processed do best under different process engineering.

                            To use software design as an example, in an environment of rapidly changing requirements but deep business engagement with a fairly robust test and prod environment that can tolerate rapid rollbacks of faulty code, Agile methodologies have clear advantages.

                            In an environment with apps where it isn't the end of the world if an outage occurs, where security is normally important but not critical and where the expense of properly staffing and equipping a data center (or multiple for disaster recovery) is prohibitive, Cloud architectures, even with current limitations, have clear advantages.

                            In an environment where requirements are stable (change slowly) but extremely complex, where downtime costs the company money instantly or leaves it exposed to lawsuits or regulatory punishment, old-school waterfall methodology combined with a robust QA and six-sigma like process monitors and statistical controls truly shine.

                            If you do not understand the advantages of all three, I question your understanding of process engineering.  

                            My company uses all three broad approaches, with variations for specific situations, and we're the kind of company that crushed all of our opposition by executing better than they did most of the time, while being resilient in years when we made bad decisions or the environment went toxic (as in 2009).   We're a tech company, so it isn't like we're limping along by having the secret Coke formula, an addictive chemical in our products and 80 years of marketing a brand.

                            So yeah, even though we change slowly in some areas, there is actually a reason for it, and it has not affected our agility where it matters.  Having a safe place to stand means you're more able to shift your weight without disaster.

                            Healthcare.gov does not fit neatly into any of these categories.   It was developed under conditions where only Agile does well, but it really is the kind of application that normally falls under massive production control for safety and reliability (similar to the third example).   Rolling out an app like that in a hurry means the usual evolution to true requirements happens after go-live, with a process very similar to what we're seeing now.  It ALWAYS sucks.    Apps like that usually aren't perfect on release because it is pretty much impossible to understand it well enough while developing it if nothing like it exists already.   But rushing the process to an arbitrary deadline while being extremely vague on requirements, expected user base, etc is pretty much guaranteed to experience problems.

                          •  Healthcare.gov was set up to fail. (0+ / 0-)

                            The left over moles from the Bush Administration got it rigged as a 55 contractor clusterfxck.

                            55.

                            That's utterly absurd. And these issues have nothing to do with AGILE vs. waterfall vs. the POC-with-stages approach favored by USAF Systems Command and Oracle Consulting.

                            (I don't see "cloud" as an architecture except at the hardware and connectivity functions.)

                            Identity Management should have gone to 1 vendor.

                            The rest of the web site was a 120-day project with no significant complexity -- it front-ends existing systems and there is no need for it to write into their tables or do extra logical filtering.

                            All would be well getting reports on sign-ups back at end-of-day.

                            55 @&^%$%^^&*amned separate contractors............

                          •  I agree it was set up to fail (0+ / 0-)

                            I'm not sure that was avoidable given that Congress has oversight and half of it is hostile to the idea.

                            I think we can agree that procurement was completely screwed up, the whole thing was sandbagged until after the 2012 elections and the entire project started entirely too late.    Given those front-loaded problems it's rather surprising anything was functional at all.

                          •  Congress had nothing to do with (0+ / 0-)

                            the procurement setup. That was a S.E.S. screw-the-pooch 100%.

                            Blaming democracy sucks.

              •  So many of us have been in that situation (1+ / 0-)
                Recommended by:
                ssgbryan

                "Off the shelf" often means, "tougher to get it to do what the requirements owner(s) really want," I've found.

                "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

                by wader on Sat Oct 19, 2013 at 04:37:32 PM PDT

                [ Parent ]

              •  Hiring consultants for the DBA and content (0+ / 0-)

                roles -- people with years of experience -- was what was needed.

                "two individuals that had no time to even read the manuals, much less learn how to use the tool...."

                And you're blaming the software ???????

                Really?

                I installed such as Oracle Financials for the GL and Receivables modules on the DBA side for years. And I never ever tried to play CPA on the accounting role tasks.

                Of course the health care packages are complicated. Experience is that sine qua non. But with the appropriate experience, you buy competent, user-group supported capabilities ASAP.

        •  No fucking way. (13+ / 0-)

          Speaking as a hardware engineer who interfaces with software engineers.  We build medical equipment, which among other things, records and stores patient information.

          Here you are trying to remake health care in the US of A.  If you think that's a 4 month project, good luck.

          Others in this thread have made cogent comments touching this point.

          The hungry judges soon the sentence sign, And wretches hang, that jurymen may dine.

          by magnetics on Fri Oct 18, 2013 at 11:13:42 PM PDT

          [ Parent ]

        •  ROFL. You're funny. Or trolling. (1+ / 0-)
          Recommended by:
          Dburn

          Its so easy, you should rung bids on this kind of project.

          You remind me of Six Weeks Steve.

          "What could BPossibly go wrong??" -RLMiller "God is just pretend." - eru

          by nosleep4u on Sat Oct 19, 2013 at 08:47:02 PM PDT

          [ Parent ]

          •  All this is doing is front-ending existing (0+ / 0-)

            systems.

            Nothing new.

            Really, nothing new but the eligibility system.

            Give me PowerCenter and the existing add-ons and OTC accounting for Identity Management -- yeah, 120-days is the real time line.

            This is a very easy problem. But they did not implement the data warehouse approach. They went with independent queries and real-time data acquisition for each session... which is a stupid alternative.

      •  Bad strategy-- outsource to wait & see vendors. (6+ / 0-)
        Recommended by:
        welt, MNPundit, greblos, Oh Mary Oh, imchange, wader

        Better not to privatize, and start working on the thing as soon as it's passed.  That would have made sense.

        To anyone with business experience in the real world.

        The hungry judges soon the sentence sign, And wretches hang, that jurymen may dine.

        by magnetics on Fri Oct 18, 2013 at 11:09:44 PM PDT

        [ Parent ]

        •  Probably true (4+ / 0-)
          Recommended by:
          Oh Mary Oh, imchange, AmazingBlaise, wader

          Almost certainly politically impossible.  The comment about having 500+ executives tinkering seems valid.  It is hard enough to decide on the build vs buy decision with only 2-3 executives (CIO, CFO usually) in the mix.   With over half of congress actively hostile to ACA, you don't think they couldn't kill any attempt to insource it, if only by reassigning team members periodically so nothing gets done?

  •  Are you saying a camel is an animal created by (7+ / 0-)

    committee.

    Only gun owners can control their guns and they say oopsie way too much. I lost it, I forgot it, it just went off. Support Gun Kill Speed Limits and Gun Ownership Speed Limits.

    by 88kathy on Fri Oct 18, 2013 at 10:31:41 AM PDT

  •  I used to be a contractor in Healthcare IT also... (24+ / 0-)

    I worked on technical mergers of large health plans. I lead and/or project managed some parts of implementation.

    I sympathize and agree with most of welt's insights.  It's hard to bring a bunch of techies, sub-contractors and users together to build something that works.  That's why the big deal contracting firms get paid hundreds of millions of $$ to do just that.  They leverage past contract awards (despite the results) to win future awards. That's business.  

    Year after year, a a select in-group of firms grab the big dollar projects. Sometimes they don't even have to bid. They pay no business price for failed projects. They remain key vendors.

    I don't have direct experience with CGI and/or Quality Software (lead firms for the ACA). I do know the firm that did much work on the California exchanges: Deloitte.  CA cooperated with ACA, and it's exchanges are doing better...but still not as intended.

    Deloitte is responsible for many public-sector software fiascos. But it still gets the big $$$, and it often gets $$$ more to clean up the mess it created. Just ask the teachers and retirees of LAUSD, EDD recipients and public agencies.

    Project management is challenging. And Sibelius' HHS surely didn't do it's job either.  

    •  I am working with Deloitte (6+ / 0-)

      on a project for the state I work for.

      If you don't handle Deloitte they will walk all over you, take your money and leave you with a maintenance nightmare (if you are lucky enough to have a completed project).  They have lots of experience taking advantage of trusting government officials.

      Republican tax policies have led to financial conditions which have caused Republicans to demand cuts to programs they have always opposed.

      by AppleP on Sat Oct 19, 2013 at 02:33:59 AM PDT

      [ Parent ]

    •  no h1bs - using americans improves communication (2+ / 0-)
      Recommended by:
      Radhika, ssgbryan

      qssi built the data hub that is failing and piece of crap.

      see -> http://www.myvisajobs.com/....  

      QSSI has 200 employees but over 62 H1bs ADDED in 2012 and 59 H1Bs added in 2011. It a indian body shop that the US government hired to do american healthcare web site. We have lots of good developers in america. Why do people keep hiring these Indians to do crap work, when will it stop that people hire bottom of the bucket for cheap work.  

      Our culture needs to start taking care of our own college students first. These Indians are able to use these government programs and underbid.

    •  We referred to them as "The Prime Evils" (0+ / 0-)
  •  There's so much that can go wrong in the process (18+ / 0-)

    It's little short of miraculous that that site even exists.

    Dueling agendas, specs written by people who don't have a clue as to what they want or even what makes sense, penny-wise-pound-foolish budgeting and contracting processes, you name it.

    About 13 years ago I served as a subject-matter-expert for a software development project in my agency. The contractors I worked with were generally decent and thoughtful people. But there was a serious disconnect between what we wanted and the platform we were required to use. Basically it took a database program and insisted on using it to do some rather serious number-crunching tasks. Needless to say it was an abject failure. And now guess what...I'm a SME on the replacement.

    •  good luck (2+ / 0-)
      Recommended by:
      Oh Mary Oh, imchange

      and I hope this time it goes better.

      (Is it time for the pitchforks and torches yet?)

      by PJEvans on Fri Oct 18, 2013 at 02:58:46 PM PDT

      [ Parent ]

    •  After I read an article on the causes (22+ / 0-)

      of the outages, especially too many last minute design changes,  I knew exactly what was going on.  The worst client a person or company can have is a uneducated disinterested client who only walks in when the deadline gets close and says something like "No No that's not what I wanted at all"

      In web based applications there is so much complexity going on behind the pages that are presented that this type of a client should be banned from any large scale project or even small ones that absolutely require their complete immersion into it from day one.

      Unfortunately, many of the people who are in power are are in their 50s to 60s (like me in my late 50s) and have zero desire to learn anything. Some of us have been drawn to it all of our lives but we have very little power.   I worked on one project for a non-profit with a client that had very little understanding.

       One of the things that try to do when I have this type of client is to explain in layman's terms what it is that I'm doing and how long it takes to get there. I literally got a email back from one client saying "stop sending me this tech crap, I'm not interested in it , I don't read it, it's annoying"

      I looked over some of my communications and tried to see if they sounded condescending or patronizing. No I 'm very careful about that even to people who admit upfront that they have no idea what it is that I'm doing. You have to be very careful because many of these people are experts in their field and still believe that past accomplishments inoculate them from learning anything new. "I have people that handle that for me"

      If you ask them, are you interested in getting to a level where you can write a complete RFP and understand what it is that you are asking the team to do? They answer "yes". That's just for public consumption.  They have no intention of learning anything new with rare exception.

      All this leads to is a very bad outcome and usually it's laid at the feet of the developers. I once told a client who new nothing to take everything a developer says about a project; multiply the time it will take by three and cut the features in half and triple the cost. I learned that when I had a relatively high position when dealing with Electronic design engineers. This massive case of optimism that leads to massive disappointment.

      I finally learned enough to take any task that is complex and do the math.  That isn't a slam at developers because at the outset you have all sides raring to go, but the developer rarely if ever, knows about all the different people they will have to deal with and their little power areas. It's office politics on steroids if you have aggressive people who want the top position and will do anything to get the top person fired.

      So when I was on the other side of the table I tried to relay that hard won experience and have had very little luck unless the person was younger say in their 30s and a few in their 40s. I hate to say it  but until my generation kicks off and most of the generation behind me does, this will be part and parcel of large and small projects. A developer has to be part psychologist to understand why people ignore request for key info or won't comment on how a prototype application works.

      I found that most people you send links too to have them look or try something out don't bother to even go there. You need their input before proceeding and it finally comes from a subordinate with a note "why is this taking so long?"

      As an aside: NASA was considered the gold standard for writing high quality code and hitting deadlines. Look at what they had to work with in the 60s.  Programmers in the 80s were used to working with very limted resources.

      By 2000 the cost of Memory, CPUs and storage kept going to new lows. Starting in the 1990s that['s when developers started writing code that was much more verbose. Example Photoshop 1.0 came on one floppy in 1989. By now it's multiple gigabytes yet 90% of users only use the features that were found in version 1.0.

      There is little that the developer can do once they are contractually bound for a set of deliverables and part way into the project they find that key people hadn't even been briefed that there input was required.

      There is easy solution that will never be used. Have the high ranking person with the most power take a test. Based on the results (Fail) Either eliminate that person who is "in charge". If not  up the charges by three times , cut the deliverables in half , and take the allotted time and multiply by three. Then have the client indemnify the developer not only from attempted legal remedies but also turning their business into a parking lot with bad PR.

      “ Success has a great tendency to conceal and throw a veil over the evil of men. ” — Demosthenes

      by Dburn on Fri Oct 18, 2013 at 04:16:12 PM PDT

      [ Parent ]

      •  I can relate. (3+ / 0-)
        Recommended by:
        Oh Mary Oh, Dburn, imchange

        I have always preferred to work directly with the end users and skip management as much as possible.  Some end users are motivated to get the best software possible to make their work lives easier and make themselves more efficient.  Others are so afraid of change they will lie and withhold important information and then beat you over the head and call you incompetent when you give them an incomplete product.  Management just wants to tell you what to do without knowing the ramifications of what they are asking so they can claim all the credit when it is done.

        It is a tough business.  

        Republican tax policies have led to financial conditions which have caused Republicans to demand cuts to programs they have always opposed.

        by AppleP on Sat Oct 19, 2013 at 02:45:40 AM PDT

        [ Parent ]

        •  Our answer to the incomplete product issue (2+ / 0-)
          Recommended by:
          Oh Mary Oh, imchange

          Is to make the users work with us early to tell us what success looks like.

          Give us all the use cases they need to do their job.  When we're nearing readiness for integration test we have them provide us a preferred set of data (customers, regions, etc) they'd like us to work with.  We go around with the IT subject matter experts to see what can be done in test environment.   Loop back and forth till all agree...if these tests run with this data succeed, we accept the software.

          Then IT does the integration test as a dress rehearsal for user acceptance testing, getting all the freaking configuration/setup/we never connected that in test envt BS out of the way before any user sees it.

          Done a world of good.  But the most important step in all of that is the first one...beating the use cases out of them and getting a signoff saying "yup, that's all of them"

        •  In one case I was told to put up a website (3+ / 0-)
          Recommended by:
          imchange, linkage, T Maysle

          in 30 days with all kinds of interactive procedures. I promised them a working prototype nothing more. I got a CMS (Joomla) Then bought a $hundreds  of extensions for it. Then they told me I had to put content in. HShit...

          aI somehow managed to fill the site with Content. The person who ordered it sent a subordinate to look at it. He didn't particularly like UI. So the CEO  moved to hire another group of developers, a group their investors insisted on.

          I was told they had a proprietary system that NO ONE ELSE HAS!. I was shaking my head. Then comes the part that tops them all. I asked to look at the features of the site they were proposing. Huh? it had all the same features that I put in.

          Then I realized what happened. The people running the place either never looked at it or did but didn't try it. Then they directed the developers to my site because they couldn't articulate what they wanted. The developers went through it and took all the features I had put up, put them in their proposal and told them it was proprietary.

          I was still working as a independant as somewhat of a webmaster. It was 2009 and as I watched jobs being lost and the economy nose dive , I told myself, "get whatever you can and don't let ego get in the way"

          8 Months passed. No site. Just for the hell of it I searched for the site and found the Beta. Not only was the UI worse than mine (except for the writing. They had a first rate writer) but none of the features that they had "proposed" were up. Apparently they had run into the same thing I had until the group disbanded. They operated like a Movie team. They would all come together for one project and then disband to take on other small projects until a big one came in.

          I kept telling the CEO's subordinate that nothing was happening with the site. After 9 months they tell me that the "developers begged them to fire them". That of course was their ego talking. So I called the lead and asked him what happened. Same thing. Send designs in and wait for a two months before they would get a hazy reply and then when one of there deadlines approached they did to them what they had done to me. They had enough and were smart enough to get a kill fee upfront.

          I remember the Lead  asking me: "Is this a real Company? " I said "Huh. I never thought anything like that".  Turns out, when they decided they were through with me, that this was a vanity non-profit. I did research on their 990s. I contacted a few of the many that had gone before them and as soon as I mentioned the name, they had either anxiety attacks or just raw anger. Apparently it was trigger for anxiety disorders and many other problems.

          I had used Cold fusion for some parts of the site for small apps. I had to clean the site of them because it was moving into an environment that didn't have Cold Fusion. Then I would get calls "The site isn't working the way it was. I told them what I had to do. Whoops it flew right over their head. Then I was contacted by their lawyer.  He told me that he was a litigator formerly a securities enforcement attorney.  I said "So you got laid off and figured since you were an attorney you would be able to handle anything that had law behind it.  I then told the guy to put the complaint/threat in writing as most professional attorneys do. He  asked weakly "Put it in writing?" "Yeah" Then I hung up on him.

          It never went legal. I was trying to get the last months check out of them plus reimbursement for many out of pocket costs.

          The end? The site that had planned on never appeared and their regular site went dark for 7 months. They had replaced the old site with a wordpress site.

          All of this is a true story with no spin. Can anyone top it?

          “ Success has a great tendency to conceal and throw a veil over the evil of men. ” — Demosthenes

          by Dburn on Sat Oct 19, 2013 at 08:26:06 AM PDT

          [ Parent ]

          •  My horror story (1+ / 0-)
            Recommended by:
            Dburn

            I was moved from one department to another because they loved the application I had created for the department and wanted something similar but more complex.  A couple months after I started IT hired a "Team Lead" that only had one member of the team...me.  Then I found out that I was only 75% on the new department and 25% on the old department and had to complete the project in less than a year while maintaining my old application and answering to the IT manager, the new team lead (who know nothing but had a lot of ideas for my time), my former business area manager who wanted 100% of my time and the business area manager who expected a completed application without bothering to stand behind me when I requested time to work on it.  Her strategy was to wait until the application didn't show up on the scheduled date and throw a fit at that point.
            Long story short...While everyone else played politics, threw me under the bus and spread every lie possible about me I was able to successfully deploy a web application by myself with over 120 pages to over 1K dedicated users with few problems.  Today, a year later, I am working elsewhere.  They have 4 developers (all making more than I ever did) doing my previous job....badly.  The "team lead" got my boss fired and took his position...and has had to explain to the board why a 3.5 million dollar project has failed.  I am sure he will be promoted to CIO in a couple of years.

            As I said:  It is a tough business.

            Republican tax policies have led to financial conditions which have caused Republicans to demand cuts to programs they have always opposed.

            by AppleP on Sat Oct 19, 2013 at 05:56:15 PM PDT

            [ Parent ]

      •  a friend worked for (0+ / 0-)

        Rocketdyne as a programmer. He said that when they upgraded the computer hardware for his project, the older engineers insisted that they put in everything that was in the original software, including the comments.

        (Is it time for the pitchforks and torches yet?)

        by PJEvans on Sat Oct 19, 2013 at 08:26:08 AM PDT

        [ Parent ]

  •  How would NSA be any different? (10+ / 0-)

    I used to work at Fort Meade, and you describe the work practices there to a Tee.

    •  their advantage is they're INVISIBLE! (16+ / 0-)

      See, the beauty of security contracts is that nobody knows how incompetent they are. In fact, they can be completely FUBAR but they can use various levels of clearances to hide it.

      My current startup is using MVP (minimally viable product) metrology. That's a concept that would never fly for a federal project of this scope.

      I would have proposed back in January that the project be released with an inverted-Turing test model. That is, you use a computer to access, but the reality is that there are people on telephones on the back end typing response.

      14 million unique visitors in 3 weeks is pretty scary for a launch of this complexity.

      I'm a serial tech entrepreneur in Silicon Valley of several decades with patents and a couple exits. My expertise is in business models and systems analysis, with lots of project management. I get to build the demo and GUI, and let the developers work without micromanaging.

  •  What is the collective, professional opinion (5+ / 0-)

    on when it might actually work?

    I'm in NH;  highest healthcare premiums (for independents) in the country and no tax base to develop its own network.

    The Revolution has already been televised. This is just the mopping up operation.

    by opticnerver on Fri Oct 18, 2013 at 01:45:59 PM PDT

    •  If I had to guess (18+ / 0-)

      I would say it will probably continue to keep getting better and better in the coming weeks.  There are more and more people reporting success with it right now, I believe (at least that's what Ezra Klein's twitter machine told me earlier).  I just window shopped to try to debunk some facebook myths a few minutes ago without any issues.

      I would suspect that the datacenter has a lot of guys sweating a lot of bullets hunched over machines desperately fixing bugs and working long hours.  And behind those guys stand a lot of bosses with a lot of anxious looks on their faces! :)

      •  Thanks, that's encouraging, (6+ / 0-)

        and I'm certainly willing to wait and support the effort (painful as it is for me, given the long fight and anticipation), and they really can't afford, politically, to link a blank page after login into 2014!

        The Revolution has already been televised. This is just the mopping up operation.

        by opticnerver on Fri Oct 18, 2013 at 02:08:02 PM PDT

        [ Parent ]

        •  Hey, Opticnerver (9+ / 0-)

          Just in case it helps you or anyone else - I had that problem on Firefox - I get a blank page instead of profile page after I log in. I used the live chat function after a couple of days of this and she suggested using another browser. That did the trick, I have no trouble with Explorer.

           Still can't get anywhere using firefox, though.

          “Good things don’t come to those who wait. They come to those who agitate!” Julian Bond

          by Dvalkure on Fri Oct 18, 2013 at 02:27:38 PM PDT

          [ Parent ]

          •  What version? Have any plugins installed? (5+ / 0-)

            If it uses a lot of scripting and you have no script or something similar, those won't necessarily work unless you enable them.

            And if you use chrome, it won't bring in scripts or any content from a non-ssl site on a ssl page. It won't tell you, either.

            We've run into this recently. I wouldn't be surprised if they did as well. It's pretty new. The pages we had that broke were working fine last week.

          •  You can't log in with Chrome either (1+ / 0-)
            Recommended by:
            Oh Mary Oh

            They should use javascript to detect the browser and notify the user to switch to IE.

            I just checked and they lost all the information I typed in last night so i have to go through the whole process again.

            I am an internet software developer so I understand the complexity...but really.  Saving and retrieving data is a basic requirement.  As is testing for browser compatibility.  

            Republican tax policies have led to financial conditions which have caused Republicans to demand cuts to programs they have always opposed.

            by AppleP on Sat Oct 19, 2013 at 03:02:13 AM PDT

            [ Parent ]

          •  they seem to believe (2+ / 0-)
            Recommended by:
            nargel, T Maysle

            that there are only two browsers in the US: IE and Safari. (FWIW, I use a commercial site where you can do stuff with Firefox, but some things won't work unless you use IE. And they don't seem to understand that they have a section of their site that's browser-specific, because I've told them that it's a problem at least twice.)

            (Is it time for the pitchforks and torches yet?)

            by PJEvans on Sat Oct 19, 2013 at 08:36:58 AM PDT

            [ Parent ]

          •  Thanks for this! nt (0+ / 0-)

            The Revolution has already been televised. This is just the mopping up operation.

            by opticnerver on Sat Oct 19, 2013 at 11:55:50 AM PDT

            [ Parent ]

      •  Well, it worked (7+ / 0-)

        just fine for me when I visited the site today.  I gave it a few weeks just as I'd planned from the git-go; waiting for the crush to lessen and the glitches to be revealed.  I guess I'm not an Early Adopter.

      •  I also suspect that they're (4+ / 0-)

        Going to have to do a major database cleanup before too long. Tons of duplicate accounts, failed accounts, looky-lous. You can't leave that in there forever.

        Not sure how they'll do a purge though. But they need to.

        •  Maybe, maybe not (3+ / 0-)
          Recommended by:
          JerryNA, ladybug53, Oh Mary Oh

          You can do a purge based on activity and that's pretty safe.

          Something like if they haven't finished setting up the account in 30 days, purge it.  There probably are not duplicate accounts at the database level (ie, accounts with the same identity information in all key fields).   People may have duplicates, but to the database they are unique.

          If an account is set up but not used, you set the password to expire after a while and present a screen to get a new password in the usual way.   Normally you do not purge real accounts on an app like this, not ever, but if for some reason they wanted to, the way to do it would be to email the account owner and ask if it is ok to purge it (or perhaps purge if they don't log in in XXX time, something like 30 days).    Only account owners who have not signed up for health plans would be on that mailing list.

          Disc space is really, really cheap, and modern databases with proper indexes have no difficulty processing multiple tables with tens of millions of records per table.  Hell they've even become pretty good at processing hash joins of two hundred million row tables if you've got big enough iron.    Most folks these days don't really purge data except for legal reasons or when they switch over to a new application and don't want to data convert all the history.

      •  Fixing bugs.... or workarounds? (1+ / 0-)
        Recommended by:
        Oh Mary Oh

        Desperate for results, instead of fixing the bugs there's band-aid piled on band-aid until it collapses under the strain of all that tugging?

        •  Bandaids are normal (3+ / 0-)
          Recommended by:
          Oh Mary Oh, Kristjan Wager, nargel

          Production down is always fixed like this:

          1.  Get the system up and running
          2.  Diagnose what went wrong (sometimes needed for #1)
          3.  Put in a temporary fix or work-around with the emphasis on SAFE.  Usually the minimum possible to avoid the issue.
          4.  Code a proper fix in with all other bugs/features discovered recently into a build, with proper project management, test cycles etc to ensure the proper fix won't break anything else.
          5.  Deploy that fix along with numbers of other fixes, replacing the bandaids.

          Only if all of your IT is spending all of its time on work-arounds does the scenario you describe occur.  That's a trap, and it happens sometimes with understaffed IT, but it is fairly well understood in most IT organizations these days that each bandage/work-around makes a system more fragile.   I've spoken with one person involved in the back end a little bit offline and he was talking about "Phase 2" two weeks in.  That's IT code for "all the stuff we triaged out of the first build, plus the first round of proper bug fixes".

          It could happen the way you describe, but I judge it unlikely because they've got people working on proper phased rollouts.

          •  My wife is an IT project manager. (0+ / 0-)

            And her previous job was in a place where they WERE spending most of their time on work-arounds because the company bosses hated to let them spend the time to fix it properly and demanded more content to sell. They were a media company.

            If the bosses are really as bad as depicted here, then maybe that will happen.

    •  It isn't going to "just work" one fine day (2+ / 0-)
      Recommended by:
      welt, Oh Mary Oh

      The thing that was broke yesterday will be fixed.

      Some problem users will succeed.  Others will expose a new problem hidden behind the first one.

      Then that will be fixed

      Lather rinse repeat.

      Just when you think it's stable, people will push new functionality in, causing new unintended consequences.

      It will do just what it has been doing.  It'll get better.  More people will have it work for them, and the nature of the beast is that once it works for somebody they won't log back in for another year.   It'll look WAY better a year from now.

      For those struggling, it will suck and suck and suck less and finally work.   The universe of people it will work for will broaden.  At some point it will include most people, those with the worst problems will have given up and done it by phone or paper.

      The only real question is how fast it will improve.  It's been tough so far because step 1 "set up an account" is blocking a lot of bug discoveries.  If they actually get a browse feature in, that'll help a lot, as casual viewers will then help bug test some of the deeper features.

    •  Here's your link to find docs (2+ / 0-)
      Recommended by:
      Oh Mary Oh, ybruti

      http://www.nh.gov/...

      I'd assume that filling in forms is faster than trying to deal with crap software.

      Finding out which policy you want is another matter. Betcha the telephone help is pretty good.

  •  This is a program management problem (20+ / 0-)

    It's not a software problem. The problem is doing it on the cheap assuming that a whiz bang web site is going to magically appear and it will take the place of administration and operations.  

    I just got a postcard today from local police asking if I want to complete a survey on-line.   I had a similar experience with getting some information in the mail about some fairly complicated changes in garbage and recycling.  In both cases, my suburb knows its citizens well enough to know that they needed BOTH paper instructions and a web site.

    They didn't assume that if they built a web portal their their mission would be successful.

    People needed simple options.  They needed a simple web site and a simple paper alternative.  They needed both delivered to them so that if one method wasn't working or was confusing they could easily use the other.  Instead, they have a not ready for prime time web site, no forms in hand, and a list of assisters who they have to track down and trust.

    This is NOT a software engineering problem.  This is a program management problem.

    This will become even more evident on the back end when people do not get insurance and policies and cards delivered accurately and on time.

    •  I agree completely (17+ / 0-)

      The failure is on the management side, which is what I tried to outline above.  The reality is that half the management wants the project to fail (technically half the management have bet their careers on the project failing).  That means any deviations from the project plan have to go through a group of people who are capable of stopping any change in the plans.

      What's wrong with Healthcare.gov isn't the contractors or the engineers working on the product, it's the GOP in congress standing in the way of the project and refusing even the slightest modification to the project because they want it to fail.

    •  Best summary I've read yet. (7+ / 0-)

      The website being in full failure for me, they're claiming I'll get a paper application in the mail by "about Oct 30th" (that was from Oct 7th). I doubt they'll reach that estimate, or that it's anywhere near finished yet; even my US Senators, one of each flavor, don't have access to copies.

      Tax law changes constantly, yet you can get forms at nearly all libraries and post offices in plenty of time to file. But that's a process with a huge motivation factor for both major parties behind it - pay your taxes!

      "All war is stupid" - JFK

      by jorogo on Fri Oct 18, 2013 at 03:36:51 PM PDT

      [ Parent ]

    •  Agreed (3+ / 0-)
      Recommended by:
      JerryNA, greenbell, Oh Mary Oh

      This is absolutely on the program manager.

      The software vendor may have enabled them by making optimistic predictions, but given how this was bidded out, I assume you don't show up to the table unless you think you have a chance of succeeding the unrealistic timeline and expectations.   So only the folks inclined to be overly optimistic bothered to bid, likely.

      One can at least hope between April 2010 and November 2012 the program manager at least did the data mapping and similar preparatory activities before waiting to see if there would be any Obamacare after the 2012 election.  So the vendor probably wasn't working from a cold start.

      But if my management told me to be in a lead technical role on this project, showed me the deliverables, timelines and above all, complexity of systems (all owned by different agencies or insurance companies) I'd ask them for some of whatever they were drinking or smoking.   Then I'd try to pare down features to the bare minimum needed to get it running.

      •  And I would go a step back from there (1+ / 0-)
        Recommended by:
        T Maysle

        to the business end of it and maybe that's why they're failing because they're not in the insurance business.  I spent a few years in the hell of corporate HRIT - payroll, open enrollment etc.  So many of the issues were business issues, dealing with carriers and getting accurate information from them and accurate specs to feed data.  It was also having the HR people to handle questions from employees so IT didn't have to get buried with questions about policy.

        They're trying to implement a system which interfaces with millions of customers who have QUESTIONS not to mention technical problems.  Yes, they have "assisters" but they are not tightly integrated in the project and they don't have skin the game in the sense that their careers depend on it succeeding or any long term relationship with their customers.  I mean some may but it's all over the map.

      •  Is the program manager an imbedded "Bushie"? (0+ / 0-)

        Torture is for the weak. After all, it is just extended wheedling.

        by nargel on Sat Oct 19, 2013 at 01:15:41 PM PDT

        [ Parent ]

    •  It's a software problem. (0+ / 0-)

      This shouldn't need more than 40 web pages for everything. And they can't do the ID process because they failed to build a warehouse ahead of time, up front.

    •  Program management problem - I agree... (2+ / 0-)
      Recommended by:
      Oh Mary Oh, greenbell

      I read an article a couple of weeks ago that said that companies that get government projects for big computer rollouts like this are much more experinced at playing the POLITICS of getting such contracts than they are at EXECUTING such projects.

      BIG problem there!!

      Existence always was and always will be.

      by Seattle Mark on Sat Oct 19, 2013 at 12:13:18 AM PDT

      [ Parent ]

    •  I disagree somewhat (2+ / 0-)
      Recommended by:
      Oh Mary Oh, welt

      I don't think it is really a program management problem, though that's definitely part of it. Rather, it is a procurement process problem (which the link in the OP also makes clear).

      Until the procurement process has been fixed, then there is nothing much program management can do. Once the procurement process is fixed, then it is time to look at program management (and the actual development processes etc.)

      My perspective is from working with farily large projects in the Danish finance sector and public sector. I am currently working on a somewhat large project for the Danish public sector. The most spetacular failure I've ever been involved in, was a $200 million project in finance - the public sector projects I've worked with, have all ended up being successes.

      Here in Denmark, there have been a number of large scale public sector projects that has failed, and there is a lot of work put into changing the procument process, as to allow better, more agile projects, without removing the transparency. In other words, we are quite a bit further down the line in making needed changes than the US. The contractors, including the company I work for, are quite involved in this process, and are working together with the public sector to continuously improve the process.

      Also, as an aside, I've seen several articles mention that the projects in the US public sector should start using Agile. I find it amazing that anyone at this day and age isn't using Agile in one form or other. I can't think of any project in Denmark, public or private, within the last 5 years, that hasn't been Agile.

      •  In my world (2+ / 0-)
        Recommended by:
        Oh Mary Oh, welt

        procurement is an early task in program management, which is why I rate this whole thing as a program management failure.

      •  Procurement is absolutely an obstacle but that (1+ / 0-)
        Recommended by:
        welt

        should be a known constraint in how the program is planned.  

        And there I guess you might say it is also a political failure or an executive failure because the program manager given the horror of the federal procurement process was probably given an unrealistic timeline.  If Obama could manage like Steve Jobs he probably could have made the constraints disappear but given that he can't manage that way someone needed to tell the executive staff that they weren't going to get insurance policies delivered by Iphone.

  •  Why is California poise too register (0+ / 0-)

    100 thousand people a  month ,if they had the cooperation from some holdout  states,we would not need  so much of a centralize computer database ,maybe Republican are using botnets trying to bring down the server ,i am not into conspiracies but maybe it just a thought ,they can process million of lottery ticket per minute ,Republican desire too  deny people that need life saving care will stop at  nothing to derail Obamacare

    •  California is a different beast (8+ / 0-)

      Unfortunately federally we had one shot at the requirements, budgeting and planning process and the intransigence of the Republicans to pass ANY new law or budget means that whatever was set in stone on day one was the end all be all of the entire project.  In California they have a state legislature that is amiable to the law, and thus they can react accordingly.

      The number one thing I've seen kill software projects is rigidity.  If there is no flexibility, either through reducing requirements or increased budgeting (mostly more time) then the process is doomed to fail unless it was meticulously planned (and it's NEVER meticulously planned) and small in scope (and Healthcare.gov is not in any way small in scope).

      Every successful project I've been on has had some flexibility, whether that is reducing the feature set or having the ability to delay the release, etc.  No matter how well you plan, projects of this size that require years to complete will have some things that change the scope of requirements over the process.  If you can't react to that due to systemic organizational problems (i.e. our Congress) then you are destined to fail.

      I've shipped products that sold millions despite having features stripped at the last minute because we couldn't finish them in time, and I've worked on projects that failed spectacularly and wasted millions of dollars because marketing insisted that every feature was maximum priority and that meant every feature worked equally poorly when we overran the cost estimates.

      •  If you feel you have the solution to the problem (0+ / 0-)

        Why do not you make your servies available to get it fix,i had too retool my megamillion formula ,because of they added 19 more numbers too the  draw

        •  I'm not sure I follow? (0+ / 0-)

          I don't think I indicated I had a solution to the problem.  You can understand what the problem is without having a solution.  In fact, my entire point is that it's a difficult problem without a clear and easy solution.

          I'm sorry if I came off as someone that was armchair quarterbacking, I certainly do not envy the poor engineers who are likely working 16+ hour thankless days and I definitely do not have a silver bullet to fix their problems (though I wish I did, and if I did I'd gladly offer it up).

  •  "Wealthy white guy welfare." Perfect! (9+ / 0-)

    You're correct.  Just look at all the fat cats (no disrespect to pooties) face down in the Defense trough.   We're talking BIG money.

    The darkest places in hell are reserved for those who maintain their neutrality in times of moral crisis. - Dante Alighieri

    by Persiflage on Fri Oct 18, 2013 at 02:32:57 PM PDT

  •  Fiefdoms are a problem (8+ / 0-)

    in any large-scale IT project, but they are especially problematic in government.

    Having worked as a Business and Quality Assurance Analyst for quite some time for NY State, I've seen it first hand.  Even in a state as seemingly homogenous as NY without the Democrats versus Republican politician nonsense, there are always a multitude of politically appointed bigwigs trying to build themselves and their own fiefdom up while simultaneously tearing down the fiefdoms of the other politically appointed asshats.  To add insult to injury, most of the politically appointed asshats have zero background in IT and make pathetically stupid decisions.

    Add in consulting firms that want to get paid as much as possible, and have an incentive to make sure things don't work just quite right so they can stay on to help maintain and fix the software after it goes to production, and it's a recipe for just what we see with the ACAs implementation:  A huge cluster** being "tested in production."

    •  Well said. I have a longer comment up above (3+ / 0-)
      Recommended by:
      chantedor, Darth Stateworker, welt

      but yours gets it down to it's essence.  Working with people who know nothing but have much power is just asking for huge trouble. Unfortunately, we don't get exposed to them until well into a project. In any type of project, large or small, one can't believe how many obstacles will purposely be thrown up to make someone in the organization look bad.

      “ Success has a great tendency to conceal and throw a veil over the evil of men. ” — Demosthenes

      by Dburn on Fri Oct 18, 2013 at 04:25:23 PM PDT

      [ Parent ]

      •  I had an argument (5+ / 0-)
        Recommended by:
        tofumagoo, JerryNA, Cobbler, welt, greenbell

        with an attorney once who was a political appointee about infrastructure needs.  The idiot thought because he was able to run a tiny little Access database on his home desktop we didn't need much to run a huge Oracle database with the capacity for a huge number of transactions per second in production due to having a large number of concurrently logged in end users.  No matter how I tried to explain that the hardware needs were vastly different, he just didn't get it.

        Luckily, a deputy commissioner higher up did understand and overturned his decision allowing us to purchase the right hardware.  The attorney was pissed.

        •  Oracle compared to Access (1+ / 0-)
          Recommended by:
          welt

          That's like comparing a 747 with an ultralite. Not only do you have to deal with heavy duty hardware you have to know SQL/PL.  I can't help but laugh then break out with a anxiety attack.  Holy shit.

          “ Success has a great tendency to conceal and throw a veil over the evil of men. ” — Demosthenes

          by Dburn on Sat Oct 19, 2013 at 08:32:40 AM PDT

          [ Parent ]

  •  lousy diary requiring me to 'imagine' (1+ / 0-)
    Recommended by:
    welt

    what the process and problem is

  •  Whatever the reasons (1+ / 0-)
    Recommended by:
    welt

    for the failure of the website rollout (and it's only the HHS website that's a disaster--not the state sites and not any other aspect of ACA), it needs to be resolved somehow.  It's possible that the website isn't fixable.  Less harm would come from shutting down and setting up a whole new platform than trying to make a bug-riddled website work when it may be a hopeless cause that could cripple the ACA politically and functionally. And of course republicans who say they're shocked, shocked are actually enthralled that they may now have a way to sabotage a law and a system that they could care less about improving.

  •  I understand Completely... (1+ / 0-)
    Recommended by:
    chantedor

    I spent 22 yr as an Oracle DBA both in the private sector and as a Gov. Contractor..

    I also learned my trade in the Navy, and when I was contracting for the Gov, It was a Cluster F^&*

    U.S. Navy '64-'85 | The man who knows and knows he knows not is a wise man.. - The ink of scholars is worth more than the blood of martyrs.

    by OpherGopher on Fri Oct 18, 2013 at 04:15:01 PM PDT

  •  Sebelius must be held accountable (0+ / 0-)

    For her to go on Daily Show and laugh it off is inexcusable. yes, I work for a vendor that provides healthcare related software to the government. And I see a lot of mismanagement because your average employee - private or public - is not super efficient. Having said that, someone must have ownership of this. They had to know the Republicans would find every opportunity to bash the ACA and then for this to happen? If they were not ready, they should have had a slower rollout.

    I hope enough penalties were built into these contracts. And if the private company accepted the terms, they need to pay for it. And if they don't , whoever in government did a bad job conveying requirements need to be fired. All of them need to be disciplined or you will continue to have the same stereotype of public inefficiency being ridiculed by republicans (and deservedly, I might add). Yes, the same crap can happen with private companies too , but that is not on the tax dollar dime. So that is not my concern.

    •  Some of your suggested solutions are extreme. (4+ / 0-)
      Recommended by:
      JerryNA, welt, Kristjan Wager, PJEvans

      A process like this won't benefit from "penalties" and "firing."

      I do agree that politics and optics probably played too large a role in the date of "going live." I'll give you that.

      Any private organization that experiences an "upgrade" usually has a month of crap dealing with the new system. You can't fire and penalize everyone every time.

      That seems to be how technology works and doesn't. Some unanticipated stuff happens.

      "Jersey_Boy" was taken.

      by New Jersey Boy on Fri Oct 18, 2013 at 07:56:08 PM PDT

      [ Parent ]

    •  Not on the tax dollar dime? (1+ / 0-)
      Recommended by:
      PJEvans

      I keep hearing people say this about private sector projects, but of course it is, just indirectly.

      Private companies pay taxes after all, and big IT projects that fail, costs those companies a lot of money, which means that they pay less tax. In other words, part of the cost is covered by tax deductions.

      Also, given the number of companies that have had to be bailed out by the government in recent years, I think we should all by now realize that private company costs can easily turn into tax payer costs down the line.

  •  Sean Hannity has been doing his best... (7+ / 0-)

    ...to come up with horror stories about Obamacare (even if they happen to be faked).

    I'd love to hear his response to a caller who could get through his screeners and point out the Republicans missed a real opportunity when they refused to set up exchanges in their own states - because what I've heard so far suggests the states who did are doing a bit better at this.

    They're always claiming it's better to leave things to the states - yet when they had a real chance to demonstrate it, they balked. It makes it very clear they don't give a damn about the people they're supposed to represent.

    "No special skill, no standard attitude, no technology, and no organization - no matter how valuable - can safely replace thought itself."

    by xaxnar on Fri Oct 18, 2013 at 04:20:36 PM PDT

  •  Thank you for spotlighting the nut-and-bolts (4+ / 0-)
    Recommended by:
    elfling, chantedor, JerryNA, welt

    It is very difficult to successfully run a development project of this magnitude even if everyone is well-intentioned and is involved at the the appropriate times in the best ways.  The GOP is so invested claiming that government is the problem that they spend most of their efforts making government the problem, and it's surprising that anything works at all.

  •  Yup (1+ / 0-)
    Recommended by:
    welt

    that sounds about right. I am amazed very day that anything anywhere actually works.

  •  Let me understand. (1+ / 0-)
    Recommended by:
    magnetics

    Healtcare.gov and the state exchanges are fucked up because they were mandated by a bill passed by Congress and signed by the President?

    No, they were fucked up because every single government contractor working on them failed and because there was no governmental oversight -- not because there were too many bosses and not enough lead time.

    If you hate government, don't run for office in that government.

    by Bensdad on Fri Oct 18, 2013 at 04:50:23 PM PDT

    •  The government prohibits high-rank (0+ / 0-)

      employees from doing actual programming.

      Which means the best stop being the brightest. They go into management and vegetate.

      But still, spending $88,000,000 on this front-end system was too much money from the get-go.

      It should have been 40 pages and $2,000,000 for each of three parallel proof-of-concept systems.

    •  This goes to POTUS. It's his baby. He needs to (0+ / 0-)

      act like he has a dog in this fight.

      Not micromanage, but surely take a hand in big-picture management, particularly assuring that key positions are filled with top people.  

      It's called 'effective leadership.'

      I work in a business that does big projects --medical equipment.  Not as big as ACA, but big.  I don't expect the CEO to micro-manage, but he must, at some level, manage-- if things are going to work at all.  If you don't have the specific knowledge and skills required, make damn sure get a subordinate who does.

      The hungry judges soon the sentence sign, And wretches hang, that jurymen may dine.

      by magnetics on Fri Oct 18, 2013 at 11:24:19 PM PDT

      [ Parent ]

  •  Good shakedown. (1+ / 0-)
    Recommended by:
    nargel

    I am inclined to agree with the diary, in the idea that a monkeywrench gang could be involved. Particularly when you consider the staggering lengths the GOP has gone to derail Obamacare.

    After the way the republicans have behaved for the last six years, I believe it.

    A true craftsman will meticulously construct the apparatus of his own demise.

    by onionjim on Fri Oct 18, 2013 at 04:50:45 PM PDT

  •  apparently they're going to take it down 4 a while (0+ / 0-)

    Politics is like driving. To go backward put it in R. To go forward put it in D.
    Drop by The Grieving Room on Monday nights for support in dealing with grief.

    by TrueBlueMajority on Fri Oct 18, 2013 at 05:10:32 PM PDT

  •  Maybe they should hire some hackers to work out (0+ / 0-)

    the bugs.
    Is Snowden available?

    •  The last thing you want to fix bugs is hackers (8+ / 0-)

      People who fix bugs and don't introduce more are highly disciplined people willing to do boring, repetitive, careful work.

      They're not always the best folks to give a blank page project to, but they color within the lines and do boring things like give realistic estimates and hit their deadlines.

      What I'm seeing on the improvement side in both state and federal sites indicates they've got the right kind of folks fixing them.  New problems do not seem to be introduced by fixes to old problems.  New problems seem mostly to be masked by larger scale older problems that didn't let you get to the new thing that broke.

  •  In the 1979 SNL skit about Three Mile Island, (1+ / 0-)
    Recommended by:
    waterstreet2013

    Dan Aykroyd's Pres. Carter says "I'm a nuclear engineer, and I'm pretty worried right now."

    Well, I design, implement and maintain medical clouds in my town, and I'm pretty worried right now.

    My strongest advise is this: don't reinvent the platform. Admit that the scale of HealthCare.gov is beyond the scope of your teams' available time and skillset, and buy into an existing PaaS. Focus on the apps, the interconnect, the interoperability, and leave the backend heavy lifting to an existing, successful, supported platform.

    It's time to move HealthCare.gov to a working platform which is ACA friendly. Amazon cloud services would be my choice.

    I get it now. It's not the Tea Party. It's the Neo-Confederate Party.

    by DavidHeart on Fri Oct 18, 2013 at 05:30:25 PM PDT

    •  They can not do that. (2+ / 0-)
      Recommended by:
      Kristjan Wager, Oh Mary Oh

      The databases have personally identifiable information that is protected by law (HIPAA), requires very high security so can not be shared, and so can not be put on the Amazon cloud.  Your utterly simplistic solution does not meet legal requirements.  It's so nice that you took 30 seconds to come up with it, though.

      •  Yes they can (0+ / 0-)

        As SysAdmin, I sign off on HIPAA regs which keep the medical networks I design, go-live and manage in compliance. My most complex medical cloud has just been granted PCMH certification. No security shortcuts getting there.

        My designs are based on PaaS. I privately host using Microsoft Azure guidelines. Private hosting means I can run an established PaaS as the infrastructure of a enterprise, while guaranteeing public/private LAN/WAN segmentation.

        There is nothing in the current guidelines which preclude hosting in a private cloud structure. Private clouds using PaaS are inherently no less secure than 1990's style bandwidth heavy server-side solutions.

        Likewise, there is nothing that prevents HealthCare.gov from buying into an existing PaaS. Having a mature, common platform on which to design apps and interconnect from should have been a no brainer.

        So here's my nice new utterly simplistic 30 second response for you: Got a huge project? Pick one: Platform or application.

        I get it now. It's not the Tea Party. It's the Neo-Confederate Party.

        by DavidHeart on Sun Oct 20, 2013 at 10:45:43 AM PDT

        [ Parent ]

    •  I use AWS (3+ / 0-)
      Recommended by:
      Kristjan Wager, PJEvans, Oh Mary Oh

      The kind of tools it brings to the party are very definitely NOT the problem here.

      The most serious problems with healthcare.gov are system integration problems.  AWS will not solve those problems.  At best, it will substitute other problems.

      And many if not most of the systems they need to integrate are not easily integrated into the cloud, and cannot be moved in a year or two, much less than in a few weeks.

      The cloud is not magic.  Creating scalable sites on it is hard, especially if you actually do it right.

      Wrong, always, is easier.

      Quote of the week: "They call themselves bipartisan because they're able to buy members of both parties," (R. Eskow, Campaign for America's Future.)

      by mbayrob on Sat Oct 19, 2013 at 12:45:53 AM PDT

      [ Parent ]

      •  I use Azure (0+ / 0-)

        The clouds I live in are scalable, redundant, portable, supported, common and secure. The cloud is magic. Especially when hosted privately by a competent team with clear budget and guidelines.

        Interconnect with all those external systems will remain difficult and asynchronous until a common set of platform tools are established. Something like HL7 is not enough. Until then, go async and queue up. Don't design for real-time.

        Platform or application. Pick one.

        I get it now. It's not the Tea Party. It's the Neo-Confederate Party.

        by DavidHeart on Sun Oct 20, 2013 at 11:00:02 AM PDT

        [ Parent ]

  •  Developing software is like baking a cake (2+ / 0-)
    Recommended by:
    Cobbler, welt

    Cranking up the heat in the oven will not bake it faster it will probably ruin it.

    The same thing when you develop software, throwing money at it will not get it done faster.

    The Air Force scrapped a $1 billion software project last year.

    I hate to thing that the ACA software will not be humming by November.

    Daily Kos an oasis of truth. Truth that leads to action.

    by Shockwave on Fri Oct 18, 2013 at 07:29:06 PM PDT

  •  Amazon Cloud Services SLA (5+ / 0-)

    Is actually very vague and puts most of the risk on the customer.

    Is actually quite inadequate for mission critical apps.  The problem with cloud is  when something breaks you'll be just one in a giant line of people who can't run their business.  You can't wake up your worldwide IT force and throw manpower at a problem till it is fixed, dropping all other priorities.

    By the time Amazon gets its act together, you are out of business.   It is fine if your company can endure an outage without massive disruption, or for non-mission-critical apps where you don't want to fuss with all the infrastructure.

    There are security issues as well that are still being worked out.  Healthcare.gov has some of the most sensitive information imaginable, aggregated in a form extremely valuable if it got loose.   Cloud vendors assure a certain degree of privacy from other customers and from outsiders, but little or none internally (because they need root to maintain their infrastructure)

    Again, speaking as an It professional that works for a global manufacturing company which has looked at cloud repeatedly.  I've seen the analysis, attended the conferences, read the white papers and tried to apply the theory in the real world.  

    The problem here is NOT the data center.  The problem is that the services in their architecture have to talk to a zillion different agency and heath insurance companies, while also providing a UI for every imaginable kind of user in the universe, while keeping everything 100% safe in terms of transactional integrity and data security.

    That is a hard problem, and if you were working with Amazon, somebody still has to write all those interfaces...they just do it to Amazon instead of to your own data center.   All the problems we have now AND finger pointing with another vendor in the mix.

  •  This is a great diary on this topic. (5+ / 0-)
    Recommended by:
    welt, PJEvans, Oh Mary Oh, nargel, T Maysle

    I do not do this for a living, but am currently reading a book called, "No Better Time," about Danny Lewin and the roll out of the Akamai technology that makes the internet work.

    It's great read for anyone who likes the New Yorker style of journalism, where you get the background of everyone involved. And, as a non-programer, I've come to understand more about how the internet works by reading it.

    I otherwise would have not slogged through the tech details on this diary, but currently feel a fleeting understanding of how complicated this all is.  I empathize with everyone involved in this roll out of Healthcare.gov, and thank you all, who work in the tech field, for your work in the tech field.

    So, sort of off topic, but a good book.

    http://www.amazon.com/...

    "Jersey_Boy" was taken.

    by New Jersey Boy on Fri Oct 18, 2013 at 08:13:53 PM PDT

    •  Thanks! (1+ / 0-)
      Recommended by:
      New Jersey Boy

      Thank you for the book recommendation.  Having been in the industry for far too long now (15 years makes me ancient in the software world!), I'm always looking for new material to read.  "No Better Time" is now officially next on my reading list!

      •  You're welcome. What makes it more interesting (0+ / 0-)

        is that Danny Lewin was "the first victim of 9-11."

        It's not really a spoiler, but he was on one of the planes that crashed into the world trade centers.

        He had been a member of an elite anti-terrorism unit in Israel when he was young.  Based on cell phone calls from people on the plane, they determined that he attempted to disarm the terrorists and was killed before the planes hit the towers.

        The guy was a computer genius and a badass.

        "Jersey_Boy" was taken.

        by New Jersey Boy on Sat Oct 19, 2013 at 06:59:26 PM PDT

        [ Parent ]

  •  I've built front-end enrollment (2+ / 0-)
    Recommended by:
    magnetics, Samulayo

    systems and fixed back-end systems that couldn't handle volume. This PPACA system should have cost $2,000,000 for each of three proof-of-concept prototypes. The fundamental design should have built a data warehouse for all the look-up tables.

    Obviously, $88,000,000 is too much money to spend building a 40-page web site. That's $2,200,000 a page.

    The simple design for enrollment uses the database to store information for validations, but from there it is published to in-memory tables in a warehouse format. Updates happen at the database. Changes are the posted to the application servers automatically. Typically the whole of the look-up data fits in memory.

    Here, the PPACA reportedly generates multi-target queries that have to go out to remote links. A query to a dozen sources has to run serially to a dozen linked resources. That's beyond stupid.

    We've been consolidating large-scale lookup data and compressing it where needed for decades.

    I had a 2-terabyte system up supporting B2B purchases -- the look-up data in memory on "pizza box" SPARCs -- in the mid-1990s. The database sat on a pair of E450 Suns and published to the SPARCs' memory. The SPARCs needed very clean electric power and temperature stability, but they handled bulk info requests at 5,000/second tested capacity.

    By 2006 we got a similar system running with one 8-processor Dell box and a half-dozen front-end boxes. By that point a 4-gig front-end box was nothing unusual.

    PPACA received a couple million requests the first morning. O.K., fine. If they'd had a warehouse+publish-it system with suitable front-ends, that would have taken running 5,000/second systems at 18,000,000 transactions an hour. I don't see the reason for the choke.

    Typically we would have installed 6 servers with frontends around the country to balance out network load and provide failover. Also, typically, most PPACA pages are simple things. Verifying format could be done at the data-entry user computer.

    But you can't do it right and spend $88,000,000.

    You gotta know they spent every dime of the $88,000,000.

    •  You are very sure of yourself. (4+ / 0-)

      I'm glad you have such confidence, but my experience has proven to me that pretty much anyone who has as much confidence as you do that they can do a better job than someone else is almost assuredly missing most of effort required to finish the project.

      If I had a dollar for every contractor/employee who promised me they could do it faster and cheaper than someone else had promised, I wouldn't currently have to commute to work in the morning. ;)

    •  The 40 pages are not the hard part (5+ / 0-)

      I find it interesting that you're focusing on the number of "pages" as it was some kind of static website that just needed to handle the volume. It's not - there are a lot of integrations going on - integrations to other, much older systems. Implementing such, in a way that can handle high volume traffic, is not a trivial task.

      •  That's why there's PowerCenter or (0+ / 0-)

        other commercial ETL tools in play.

        And Oracle as the backend database can do transaction loads 1,000 times greater than 2,000,000 a week. No problemo.

        Army Purchasing has 1,400,000 users and they make transactions happen at very high volumes. No problemo.

        This was not a hard problem. A simple 40-page system that got screwed at the design phase.

        There's nobody active in the peer review pools at the SIGs at OAUG couldn't have built this site competently. It was an easy problem and we've all seen enrollment in our daily tasks.

        •  You've a simplistic view of the problem area (4+ / 0-)

          Every domain has its own unique problems, and to think that it is as easily fixed as you claim, shows me that you don't really understand the problems connected to this domain.

          When I am talking about integrations, I am obviously not talking about database integrations (where 2 million visitors a week would genereate many more than 2 million transactions, but that's beside the point). Rather, I am talking about connections to external systems, which are not build to handle this kind of load.

          There are ways to handle this, but the simplistic solutions you are sprewing are not the way to go, and would only cause more problems to appear.

          Also, unless we're talking about a static site, there is no such thing as a "40-page system".

          •  I've designed the warehouses (1+ / 0-)
            Recommended by:
            Samulayo

            to support this type of system.

            I know exactly what is involved.

            And there are no "new" source systems feeding PPACA with intrinsically difficult or undocumented data definitions. That's why you buy PowerCenter -- to get known sources up and running asap.

    •  In my experience gubment (1+ / 0-)
      Recommended by:
      fleisch

      project sponsors are loath to pilot project/proof-of-concepts.

      They would rather spend as much money as possible all at once.

      Also, a $2mm project as you suggest would not get any respect or support from agencies or legislators because the budget is too small. They want big $$ projects. Not much concern about good engineering practices.

  •  I knew all along it was the fault of the (0+ / 0-)

    republicans in Congress, and no fault of our Administration, our HHS Secretary, or our President.  Thanks for validating this.

  •  Software is hard to do (2+ / 0-)
    Recommended by:
    PJEvans, welt

    As someone who is not a programmer, but has worked through several implementations of software tools for a Fortune 10 company all I can say is, "It's hard, and there are always bugs on initial roll outs."  It is doubly so, when there really has been little, time and as welt has pointed out their are lots of requirements and too many bosses.

  •  I'm a software engineer too (2+ / 0-)
    Recommended by:
    Gorette, welt

    I looked at the technical and managerial issues in my blog. Just so happens that I manage a highly trafficked interactive federal website for a living, so I can speak with some authority, although I obviously was not part of developing this system. (Maybe they needed an oversight board of federal engineers. If so they should have asked me.)

    http://www.occams-razor.info/...

    Many an insightful opinion and observation can be found on my blog Occam's Razor. UID: 875

    by Guy Noir on Sat Oct 19, 2013 at 08:10:21 AM PDT

  •  This goes with what I read years back about (0+ / 0-)

    a couple other govt. projects one of which was the Navy or Marines I believe, and had to be scrapped, never used, at the cost of something like 60 million if I remember correctly. So apparently, yes, as you explain so well, these systems are very hard to do with all that bureaucracy and special interests.

    I didn't realize that determining the amount ahead of time was in itself problematic. That adds to the corruption or what? Do military projects always cost double or triple the original price? Rich white man's welfare, I think you called it!

    "extreme concentration of income is incompatible with real democracy.... the truth is that the whole nature of our society is at stake." Paul Krugman

    by Gorette on Sat Oct 19, 2013 at 02:40:25 PM PDT

  •  Looks like $1B was lost in Air Force data project. (0+ / 0-)

    One of our military's largest and most expensive computer systems project was scrapped last year.

    I. Money Down the Drain

    Dubbed the "Expeditionary Combat Support System" (ECSS), the project involved enterprise resource planning" (ERP). ERP refers to efforts to merge external and internal data flows, such as expenses, manufacturing metrics, logistics, contractor relationships, and unified messaging into a single flow of data.

    ...... International Business Machines, Inc. (IBM ), Google Inc. (GOOG ), and a handful of other top players in the data-mining sector.

    The U.S. Air Force  (USAF) ..... poured a lot of money into the effort -- $1.03B USD since 2005.  But five years later an cool billion out of the pocket, and a USAF spokesperson's diagnosis of the project's health is:

    [ECSS] has not yielded any significant military capability.  We estimate it would require an additional $1.1B for about a quarter of the original scope to continue and fielding would not be until 2020. The Air Force has concluded the ECSS program is no longer a viable option for meeting the FY17 Financial Improvement and Audit Readiness (FIAR) statutory requirement. Therefore, we are cancelling the program and moving forward with other options in order to meet both requirements.

    "extreme concentration of income is incompatible with real democracy.... the truth is that the whole nature of our society is at stake." Paul Krugman

    by Gorette on Sat Oct 19, 2013 at 02:52:55 PM PDT

  •  qssi and h1bs main source of buggy code (0+ / 0-)

    qssi built the data hub that is failing and piece of crap.

    see -> http://www.myvisajobs.com/...  

    QSSI has 200 employees but over 62 H1bs ADDED in 2012 and 59 H1Bs added in 2011. It a indian body shop that the US government hired to do american healthcare web site. We have lots of good developers in america. Why do people keep hiring these Indians to do crap work, when will it stop that people hire bottom of the bucket for cheap work.  

    Our culture needs to start taking care of our own college students first. These Indians are able to use these government programs and underbid.

  •  I was part of a multi-milion dollar (0+ / 0-)

    Software design project.. that ran 18 months behind, had serious service failures and to date hasn't really met the goal.

    That's Ok, I'm not there any longer, but let's not kid 'development hell' exists everywehre.

    Gandhi's Seven Sins: Wealth without work; Pleasure without conscience; Knowledge without character; Commerce without morality; Science without humanity; Worship without sacrifice; Politics without principle

    by Chris Reeves on Sat Oct 19, 2013 at 09:00:54 PM PDT

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site