Skip to main content

View Diary: A Software Engineer's take on Healthcare.gov (145 comments)

Comment Preferences

  •  I worked on the software acquisition strategy (22+ / 0-)

    and quality control specifications for Air Force some decades ago.

    They realized a long time ago that projects need essential structure and management of such - via whatever methodology and tools you will employ - to at least verify where you are at any given time.  Even so, they wanted a means to evaluate proposals up-front, gauge the likelihood that bids coming in had some basis in reality . . . and have a firm layout of how the claims being made in each bid could be tracked as the project moves along, especially between major milestones.

    Nonetheless, waste still happens and public + private business, requirements are not always managed well (often due to politics or hierarchical reasons) and scope creep happens, or definition of goals is never made clear until late in the game, etc.  Even with rapid prototyping or Agile methodologies being utilized.

    It did sound like the time from requirements to implementation was very short for the federal system - I can see how that rush may also have related to incomplete requirements clarity for awhile, because some time those offering the reqs need to see early work in order to modify their views of what truly is needed, etc.

    "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

    by wader on Fri Oct 18, 2013 at 10:11:18 AM PDT

    •  The delay was also created by nobody wanting to (43+ / 0-)

      do anything until 2012 was decided.  After all it's wasted work if Prez Romney and R-dominated congress repeals the law.

      So presume nothing real occurred until Nov 2012.  From the sound of it, things really didn't get out of discovery until spring this  year, the time in the middle likely eaten up by finding a vendor willing to do the job under the conditions described, and having said vendor vetted under all the procurement laws and such.

      In my company, 6 months discovery for choosing a new vendor from multiple choices is not at all unusual.  Then until you have a vendor, any work you did is largely wasted because they won't understand what you're saying until requirements are translated into their methodology.

      •  This explains the seemingly tardy initiation of (17+ / 0-)

        the project.

        And I think the ACA opponents knew this and introduced FUD into the start-up of the exchanges.

      •  This should have been a 120-day project. (2+ / 0-)
        Recommended by:
        walkshills, imchange

        In fact, easy to do.

        Instead of building it as a data warehouse implementation, they built a one-at-a-time web with individual queries off to mulyiple databases for each transaction.

        Dumb as a fence post.

        •  There are probably reasons they can't consolidate (12+ / 0-)

          the data. For example, HIPAA imposes very strict requirements on what healthcare-related data you can retain and how you need to secure it.

        •  No way this is done right as a 120 day project (20+ / 0-)

          You have a hell of a lot of personal information with lots of regulations about how they can be exposed and used.   Said information is stored and secured in different government agencies and perhaps credit rating companies.

          There is a reason companies like Cybersource pretty much do nothing but manage the legal part of storing credit card numbers...individual companies have to be audited YEARLY and compliance isn't easy.  I don't imagine things like income verification and identity verification are any less tightly locked down.  Unlike a private company you can't just ignore it and hope you don't get caught, or pay a slap-on-the-wrist fine if you get caught by an understaffed regulatory agency when hackers or bad actors in your own company get into it and expose it.

          You have dozens of insurance companies that have to be stitched in.  Most of these companies have their own ways of doing B2C and a lot of this will be completely out of their comfort zone...they'll all push back and cause problems.

          Even if you were to try to store everything in an ERP-like transactional database (not a data warehouse, you're doing two way transactions here, not simply dumping data in and reporting it), connections from everybody to and from the ERP database to populate it and keep it up to date, and to read data from it into the local insurance companies have to be written and debugged.

          There is a reason IT has been moving away from ERP models to service oriented architectures (which they appear to have tried to build, even if many of the services are buggy).   You don't have to hand craft every freaking thing, and there are less moving parts to break down.  (you'd think a centralized database to store the entire thing end-to-end would be simple.  It is not unless it owns every scrap of data from end-to-end and communicates with nobody but the people involved in the transactions it services)

          I've got over 20 years experience doing exactly the sort of thing you are proposing with complex and scattered systems..geographically, in tech they're built with, in time they were built under differing methodologies.   If you rush this sort of thing...you get exactly the kind of stuff they're dealing with now.

          And actually, 120 days is about what they spent on this, near as I can tell.  From what others have said the vendor and requirements weren't nailed down until this spring.  6 months (180 days) tops.

          •  It seems even many I/T folks here aren't getting (12+ / 0-)

            those fully relevant points: we don't know how many legacy systems they've had to stitch together in order to make this work.

            As you've offered, they probably went with service interfaces, because it's possible that some of those backend systems require real-time pulls based on heuristics for specific parameters provided - and, those systems may be pulling dynamically from other systems, as well.  Many times, the logic cannot be easily duplicated in a single system and all data dumped nearby for easy, fast access  - you just use the backend systems and hope that they can meet your service level requirements.

            It's amazing that it's processing as many applications as it has thus far, there must be lots of reuse from existing systems that gave them a leg up (which offers its own integration challenges, as implied here).

            "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

            by wader on Fri Oct 18, 2013 at 08:03:22 PM PDT

            [ Parent ]

            •  Except that the insurance front-end (2+ / 0-)
              Recommended by:
              Chi, wader

              software for enrollments and the once-a-year policy changes has been standardized for at least a decade.

              •  Except the rules have all changed (3+ / 0-)
                Recommended by:
                Oh Mary Oh, imchange, wader

                Just to pick one...you throw out all the stuff for medical history because they can't discriminate on pre-existing conditions.

              •  I suspect front-end software is not the biggest (1+ / 0-)
                Recommended by:
                waterstreet2013

                issue in their growth pains.

                Even so, the requirements for this system appear beyond standard enrollment setups that I've seen - perhaps due to political requirements going against the grain of industry common practice and (de facto or formal) standards.

                "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

                by wader on Sat Oct 19, 2013 at 04:33:20 PM PDT

                [ Parent ]

                •  Here's Some of the Code... (2+ / 0-)
                  Recommended by:
                  wader, waterstreet2013

                  Javascript ACA Code

                  = 'Error mapping the qualify enrollment group membersException in calling SubmitCustomEnrollmentGroups service'; resources['ffe.ee.shared.exceptions.500.20009'] = 'If retrieve Plan compare/enrollment groups data service written errorException in calling RetrieveNonFaPlanReviewConfirmDetails service'; resources['ffe.ee.shared.exceptions.500.20008'] = 'If retrieve Plan compare/enrollment groups data service written errorException in calling ConfirmNonFaEnrollmentTransaction service'; resources['ffe.ee.shared.exceptions.500.20007'] = 'If retrieve Plan compare/enrollment groups data service written errorException in calling ConfirmEnrollmentTransaction service'; resources['ffe.ee.shared.exceptions.500.20006'] = 'If retrieve Plan compare/enrollment groups data service written errorException in calling RetrieveQHPPaymentPortalData service'; resources['ffe.ee.shared.exceptions.500.20005'] = 'If retrieve Plan compare/enrollment groups data service written errorException in calling RetrievePlanReviewConfirmDetails service'; resources['ffe.ee.shared.exceptions.500.20004'] = 'If retrieve Plan compare/enrollment groups data service written errorException in calling RetrieveTaxHouseholdAPTC service'; resources[
                  Just a sample.  Remember, none of this is needed with Single Payer.  My Medicare card arrived a day before my 65th birthday.  I took my birth certificate into Social Security and signed the card!
                  •  I wish we were complaining about a Single Payer (0+ / 0-)

                    system . . .

                    "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

                    by wader on Sat Oct 19, 2013 at 05:28:21 PM PDT

                    [ Parent ]

                  •  Bingo !! (0+ / 0-)

                    None of that is needed with a data warehouse implementation.

                    If you have 500 distinct conditions to be checked (or retrieved) then the DW deisgn does them in parallel.

                    The shyte-code that uses hyper-complex queries ends up seeing the SQL processor do the conditions in series -- one after another for the 500 of them.

          •  That's off the shelf software (0+ / 0-)

            with proprietary security included.

            You buy it, not write it. Similar to doing sales tax in retail apps.

            •  Been there done that (6+ / 0-)

              "off the shelf software" still has to be integrated with everything else.

              "off the shelf software" that does taxes and credit cards has literally hundreds of settings and configurations, and you have to learn how to set it correctly for what you are doing, interpret responses, handle error messages etc.

              Just as an example, Cybersource will tell you very precisely why a credit card failed.  You can not just expose that to the customer because hackers use it to fake credit card numbers or use iterative methods to guess things like security codes.   Pretty much every possible response has to be mapped to "it worked"  "it didn't work" "the system is down".

              Don't even get me started on the complexity of tax software.  You have different rates in different states and even different counties for services, goods, freight, fees, etc.  You usually have to write exception code to handle a zip code that returns more than one county.   You need regression tests to handle all of this or your taxes end in the wrong bucket and you get sued.

              Finally there is installation.  Most software isn't like a windows "wizard", doing most of the work of a proper install for you with "download" and "run executable".  It is a giant mass of the equivalent of zip files, with terse instructions from the vendor that leave you with some binaries installed on your host, and phone-book sized manuals on configuration options.   There is a reason why the vendors make a lot of their money on consultants, to actually make the mess work in a given environment.

              Any piece of software complex enough to help with a complex business (without being simply unable to do things that you need it to do) will never, ever, "just work" out of the box.

            •  The reason for the above rant (12+ / 0-)

              Is I just spent the last year of my life working to make a "simple" "off the shelf" technology work without disrupting deadlines for literally dozens of other functional groups, because management believed as you seem to that when the vender said "it's simple and easy" that they were actually telling the truth.

              So they assigned a sysadmin and a dba to get it done, two individuals that had no time to even read the manuals, much less learn how to use the tool.

              My colleagues covered my work.  I read the manuals, talked to the consultant, worked out automation (we had a very complex use case as our first rollout), rewrote it after testing in dev, trained sysadmins and dbas in two datacenters how to do it, rewrote it again after we tried out the deployment in test, rewrote it again after we found flaws in the product that were not documented, worked out a strategy for data that didn't work with the product, did some final touches again when it interacted with the prod environment.  

              We finished just in time for the first adopter.  The stuff I was supposed to be working on in that period got dumped on the only other person on my team, so his workload was also doubled.

              The last time I had to install software that complex was back in 1999, when we had experienced a decade of expansion in IT, instead of a decade of belt-tightening followed by a recession.  I had two junior programmers working with me, a project manager, a consultant from the vendor, an independent consultant and me as lead developer.  We rolled out in half the time with no defects but it only worked for the early adopters and it took two years of tinkering to really bring as many folks onboard as we are trying to do in a big bang this time.

              The architecture and software we deployed then is still in use and survived an attempt to find a replacement vendor just a couple years ago.  I know what a successful installation of vendor software looks like and what an clusterfuck scramble because of bad management decisions and no staffing looks like.

              IT makes it work either way, eventually, but it goes a hell of a lot smoother with realistic assumptions and staffing.

              •  I hear you (3+ / 0-)
                Recommended by:
                imchange, wader, waterstreet2013

                I spent most of a year working at a company that had installed an off-the-shelf inventory control system that was intended for manufacturing. The software company had a team of programmers onsite for months tinkering it into working. When I left, it still had bugs, but it mostly worked.
                (Part of the problem was that the company owners wanted it to work exactly like the paper system they had put together. They didn't think through what that system was actually doing and how the pieces went together; some parts of the software version would have worked better when done just a little differently.)

                (Is it time for the pitchforks and torches yet?)

                by PJEvans on Sat Oct 19, 2013 at 08:21:54 AM PDT

                [ Parent ]

                •  LOL (0+ / 0-)

                  "the company owners wanted it to work exactly like the paper system they had put together"

                  I'm tempted to call them "idiots." My first talk to management groups installing Oracle emphasized that the system would not look like the old system, not by process or screens or check points.

                  Modifying a top-level OTC package to look like a semi-competent, older bag-of-onions system is idiocy. It wastes lots of money.

                  •  It isn't that simple (0+ / 0-)

                    Businesses do what they do because it works.  Changing to match a vendor's vision of how they should be doing their business at minimum involves a massive disruption - you have to redesign your data structures, retrain your workforce (warehouse, factory, sales, procurement, logistics etc) etc.

                    You also have to meet contractural obligations.  As an example, our customers have contracts with us, they have contracts in some cases that predate us, were made with companies that no longer exist but whose responsibilities we picked up.   Also our products become obsolete within about 12 months, in my industry Moore's law is for pussies.  That also affects pricing structures.  This is translated into a pricing system that is extremely flexible.

                    Oracle ERP's pricing system isn't up to the job.  For a while we had a homebrew system but eventually (with a lot of pain and very slow adoption for reasons mentioned in the first paragraph) adapted to the Vendavo system.  Which doesn't play especially well with Oracle's internals.  So we're still using our old custom pricelist and special pricing authorization logic inside the ERP.

                    That's just one tiny part of the business.  Credit checks are managed by one product, taxes by another.  Front ends to B2B, B2C and CSO are all different vendors, adapted to our needs.   They communicate with each other using a mix of SOA, older APIs, direct database links and message oriented middleware.   App servers and web servers add more vendors.  That doesn't even consider the factory systems that supply inventory, which, among other things, capture a couple dozen test results during fabrication helpful for tying field problems to individual production runs and to gather metrics on yield, scrap and long term product quality that informs future manufacturing decisions.

                    To create an order, ship it, fulfill it and get paid, an order involves products from at least 7 vendors, plus several apps we still wrote and maintain ourselves because NOTHING on the market approaches the feature set that evolved in house.

                    Or to paraphrase your last paragraph.  Assuming an off the shelf ERP can manage OTC in a business of any size is idiocy.   Assuming any off the shelf vendor software can fit into the complexity that is a modern global manufacturing system without significant effort is also idiocy.

                    •  Not exactly the optimal solution. (0+ / 0-)

                      I've written "proxy" style modules using Oracle's template to reach existing/external/other_vendor pricing. The key is using the Item Master system -- set quantity=0, set the switch allowing 0-quantity pricing, define a sub-inventory that matches your price list.

                      The sub-inventory can be a real table or a view that uses the "proxy." One can use another program altogether -- same as using a c program to set up a sales tax hook.

                      Further, this is optimistic: "Businesses do what they do because it works."

                      Business do what they do because nobody's got the balls to take the responsibility to make changes to improve the process. Very few processes remain anywhere near optimal as technology changes over time.

                      I wrote a planning system for The Hartford back decades ago. It survived for 15 years. Same hardware. At the time I thought that was wonderful, but really they were just too cowardly to stand up to ITT and insist on building a new spreadsheet-driven system.

                      There's also the issue of experience. Maintenance guys are generally not up to it for building new systems. The range of skills isn't there.

                      •  Not so much optimistic as self-evident (1+ / 0-)
                        Recommended by:
                        waterstreet2013

                        If what a business does fails to work, the business fails.

                        It might not be the best way, but it does actually work.

                        Change is disruptive.  It is not just a matter of courage.  It is a matter of risk.  The more useful and stable a legacy system or business process is, the more resistance there is going to be to change it.

                        As a rule, if you are going to change an existing business process (which includes most major software changes) the new process has to have some significant improvements over the old to encourage adoption and to allow the folks who are trying to do their job to overcome the inevitable problems and friction that occur whenever doing anything new.

                        If you are changing for the sake of change, you get a lot of resentful people who will passive-aggressively resist the change even if there is some future improvement payoff.

                        I've been in the role of business support, developer, system architect, process engineer (in several methodologies), QA guy and techie consultant.  I've supported my company transitioning its multinational manufacturing tech business from a "me too" product line to one of the two unquestioned leaders in the technology, absorbing or destroying what was literally hundreds of competitors at the start of my career to what is now a handful.

                        I've seen every argument for change or not change that you can imagine.  I've seen management get duped twice by vendors that couldn't deliver at all (so we had to build the whole thing in-house from the wreckage), one vendor that outright lied about capability, an 20ish year long relationship with Oracle and their entire product line, watched vendor after vendor get bought out and their product abandoned by the parent company.  I've also seen some really cool stuff written both in house and by vendors and stitched in.

                        We had a building full of order management people when I was hired.  Now we need only a few, to manage a business that has expanded 100 fold.  I've seen a contract system go from paper files to spreadsheets to a database to an actual half decent contract management system (once we got past the vendor that lied sigh).    I've seen factory systems that are absolutely astounding built and maintained by utter cowboys (that later went to more disciplined agile methodologies) because those systems must constantly change and can NOT have downtime (minutes cost millions, literally).      I've seen our product architecture change to match new marketing and promotion requirements to get us into the 21st century that required touching (and regression testing) nearly every IT system in the company.

                        I've seen financial systems running the same chart of accounts for 15 years because even though our business has transformed it is so disruptive to change it we had to have a major ERP upgrade occur simultaneously to even consider the risk of touching it.

                        It isn't a matter of being timid or a coward.  It's a matter of not fucking up in a way that costs the company millions of dollars in lost contracts, or causes an entire production run of defective product, or results in legal action taken against us, failing to meet regulatory requirements in a half dozen competing jurisdictions or pissing off your entire salesforce by screwing up their commission compensation formula.

                        As for maintenance guys not having the skills to build new systems, that might be true of 24x7 type support individuals who are trained to a different skills set, but where I work the "maintenance guys" are called "subject matter experts" because they know both the business and the tech inside and out.   They're the people that keep exposing all the missing functionality in what most of the software companies try to sell us, and the ones that figure out how to work in new products into our existing systems without breaking it.

                        •  "Subject matter experts" are not (0+ / 0-)

                          database engineers or acceptably knowledgeable for how to structure programming systems.

                          "Disruption" is a common excuse for running noncompetitive money-losing systems.

                          The old systems do their financial management functions well enough. Process management, not so much. Quality and value go lost so the whole company becomes vulnerable.

                          The usual result is that the laggard is absorbed in an M&A takeover. Crappy management systems make the company a target.

                          •  I've done process management (1+ / 0-)
                            Recommended by:
                            Oh Mary Oh

                            Years of it.  I've even made it work.

                            I've seen theories come and go.  Continuous Improvement, TQM, ISO 9000, Six Sigma, "Do more with Less", avoid hidden factories blah blah blah.  Trained in all of them at one time or another, actually did the "Black Belt" thing for a couple years.

                            The bottom line is that there are always "low hanging fruit" after a few years of using any methodology that are improved by using some other methodology.  Turns out different categories of processed do best under different process engineering.

                            To use software design as an example, in an environment of rapidly changing requirements but deep business engagement with a fairly robust test and prod environment that can tolerate rapid rollbacks of faulty code, Agile methodologies have clear advantages.

                            In an environment with apps where it isn't the end of the world if an outage occurs, where security is normally important but not critical and where the expense of properly staffing and equipping a data center (or multiple for disaster recovery) is prohibitive, Cloud architectures, even with current limitations, have clear advantages.

                            In an environment where requirements are stable (change slowly) but extremely complex, where downtime costs the company money instantly or leaves it exposed to lawsuits or regulatory punishment, old-school waterfall methodology combined with a robust QA and six-sigma like process monitors and statistical controls truly shine.

                            If you do not understand the advantages of all three, I question your understanding of process engineering.  

                            My company uses all three broad approaches, with variations for specific situations, and we're the kind of company that crushed all of our opposition by executing better than they did most of the time, while being resilient in years when we made bad decisions or the environment went toxic (as in 2009).   We're a tech company, so it isn't like we're limping along by having the secret Coke formula, an addictive chemical in our products and 80 years of marketing a brand.

                            So yeah, even though we change slowly in some areas, there is actually a reason for it, and it has not affected our agility where it matters.  Having a safe place to stand means you're more able to shift your weight without disaster.

                            Healthcare.gov does not fit neatly into any of these categories.   It was developed under conditions where only Agile does well, but it really is the kind of application that normally falls under massive production control for safety and reliability (similar to the third example).   Rolling out an app like that in a hurry means the usual evolution to true requirements happens after go-live, with a process very similar to what we're seeing now.  It ALWAYS sucks.    Apps like that usually aren't perfect on release because it is pretty much impossible to understand it well enough while developing it if nothing like it exists already.   But rushing the process to an arbitrary deadline while being extremely vague on requirements, expected user base, etc is pretty much guaranteed to experience problems.

                          •  Healthcare.gov was set up to fail. (0+ / 0-)

                            The left over moles from the Bush Administration got it rigged as a 55 contractor clusterfxck.

                            55.

                            That's utterly absurd. And these issues have nothing to do with AGILE vs. waterfall vs. the POC-with-stages approach favored by USAF Systems Command and Oracle Consulting.

                            (I don't see "cloud" as an architecture except at the hardware and connectivity functions.)

                            Identity Management should have gone to 1 vendor.

                            The rest of the web site was a 120-day project with no significant complexity -- it front-ends existing systems and there is no need for it to write into their tables or do extra logical filtering.

                            All would be well getting reports on sign-ups back at end-of-day.

                            55 @&^%$%^^&*amned separate contractors............

                          •  I agree it was set up to fail (0+ / 0-)

                            I'm not sure that was avoidable given that Congress has oversight and half of it is hostile to the idea.

                            I think we can agree that procurement was completely screwed up, the whole thing was sandbagged until after the 2012 elections and the entire project started entirely too late.    Given those front-loaded problems it's rather surprising anything was functional at all.

                          •  Congress had nothing to do with (0+ / 0-)

                            the procurement setup. That was a S.E.S. screw-the-pooch 100%.

                            Blaming democracy sucks.

              •  So many of us have been in that situation (1+ / 0-)
                Recommended by:
                ssgbryan

                "Off the shelf" often means, "tougher to get it to do what the requirements owner(s) really want," I've found.

                "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

                by wader on Sat Oct 19, 2013 at 04:37:32 PM PDT

                [ Parent ]

              •  Hiring consultants for the DBA and content (0+ / 0-)

                roles -- people with years of experience -- was what was needed.

                "two individuals that had no time to even read the manuals, much less learn how to use the tool...."

                And you're blaming the software ???????

                Really?

                I installed such as Oracle Financials for the GL and Receivables modules on the DBA side for years. And I never ever tried to play CPA on the accounting role tasks.

                Of course the health care packages are complicated. Experience is that sine qua non. But with the appropriate experience, you buy competent, user-group supported capabilities ASAP.

        •  No fucking way. (13+ / 0-)

          Speaking as a hardware engineer who interfaces with software engineers.  We build medical equipment, which among other things, records and stores patient information.

          Here you are trying to remake health care in the US of A.  If you think that's a 4 month project, good luck.

          Others in this thread have made cogent comments touching this point.

          The hungry judges soon the sentence sign, And wretches hang, that jurymen may dine.

          by magnetics on Fri Oct 18, 2013 at 11:13:42 PM PDT

          [ Parent ]

        •  ROFL. You're funny. Or trolling. (1+ / 0-)
          Recommended by:
          Dburn

          Its so easy, you should rung bids on this kind of project.

          You remind me of Six Weeks Steve.

          "What could BPossibly go wrong??" -RLMiller "God is just pretend." - eru

          by nosleep4u on Sat Oct 19, 2013 at 08:47:02 PM PDT

          [ Parent ]

          •  All this is doing is front-ending existing (0+ / 0-)

            systems.

            Nothing new.

            Really, nothing new but the eligibility system.

            Give me PowerCenter and the existing add-ons and OTC accounting for Identity Management -- yeah, 120-days is the real time line.

            This is a very easy problem. But they did not implement the data warehouse approach. They went with independent queries and real-time data acquisition for each session... which is a stupid alternative.

      •  Bad strategy-- outsource to wait & see vendors. (6+ / 0-)
        Recommended by:
        welt, MNPundit, greblos, Oh Mary Oh, imchange, wader

        Better not to privatize, and start working on the thing as soon as it's passed.  That would have made sense.

        To anyone with business experience in the real world.

        The hungry judges soon the sentence sign, And wretches hang, that jurymen may dine.

        by magnetics on Fri Oct 18, 2013 at 11:09:44 PM PDT

        [ Parent ]

        •  Probably true (4+ / 0-)
          Recommended by:
          Oh Mary Oh, imchange, AmazingBlaise, wader

          Almost certainly politically impossible.  The comment about having 500+ executives tinkering seems valid.  It is hard enough to decide on the build vs buy decision with only 2-3 executives (CIO, CFO usually) in the mix.   With over half of congress actively hostile to ACA, you don't think they couldn't kill any attempt to insource it, if only by reassigning team members periodically so nothing gets done?

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site