Skip to main content

View Diary: MUST WATCH: health insurance then vs. now (19 comments)

Comment Preferences

  •  You're not the only one with I/T experience (2+ / 0-)
    Recommended by:
    Catte Nappe, kerplunk

    here.

    Your prescription does not sound like a solution to current issues, but an alternative way to develop the system - from scratch, apparently.  For example, PaaS is not a requirement for infrastructure scalability.  "Rewrites" of things not fully broken are overkill and can simply recreate what went wrong in a different fashion.

    Project management methodologies, scalability built into the design, synchronization of service levels/availability across back-end systems, etc. are often more important aspects of ensuring reliable,correct performance than technology, per se, IMHO.

    I doubt that anyone at CGI is feeling complacent about handling the startup issues of their system.  I'm not sure what you men by "noisy", but I think the situation and internal pressure is already there when the President has already mentioned the situation publicly.

    Most of the problem mentioned with the federal site thus far sound like back-end synchronization issues under load, to me.  I'm sure it's more complex than that, but given the issues with what appear to be partly-saved profiles, messed up sessions and the like, I'd say that they are having timing issues that were not encountered in test, at the very least.

    "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

    by wader on Fri Oct 18, 2013 at 09:59:36 AM PDT

    [ Parent ]

    •  Pick one: platform or application. (0+ / 0-)

      Unless you have unlimited time and funds you shouldn't bet the farm on a project that does both.

      I get it now. It's not the Tea Party. It's the Neo-Confederate Party.

      by DavidHeart on Fri Oct 18, 2013 at 10:45:02 AM PDT

      [ Parent ]

      •  Do we know from what they built this system? (0+ / 0-)

        I've worked projects that have built successfully from scratch and others that are able to reuse + grow existing programs - via some combination of infrastructure/hosting, application(s), support, processes, etc.

        In my experience, you don't have to choose only one or the other to find success - but, you should still make a decision up-front based upon expected needs multiplied by a reasonable growth factor, etc.

        Until we hear more, I can't offer a prescription for their ails besides suggesting processes to assess their issues and relative severities of such - as I mentioned above, it's likely an issue from the combination and synchronization of systems that only one area.  That doesn't imply we necessarily should throw all of their platform or application out the window to solve the problems.

        "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

        by wader on Fri Oct 18, 2013 at 12:02:38 PM PDT

        [ Parent ]

        •  Have you ever picked up on someone else's code? (0+ / 0-)

          If so, multiply that by 50 different contractors on the same project. Imagine the spaghetti code, the lack of documentation, the utter hopelessness of trying to make sense of design by committee.

          And what do you mean by "synchonization" and "timing" issues? The internet is a set of protocols, not some RS232 handshake.

          And those garbled records? Christ on a pogo stick. How can you be complacent about that? World class backends never drop records because their T-SQL based. Transactions either complete, or they don't.

          What kind of Mickey Mouse bullshit is going on?

          Every IT professional in the US should sitting up taking notice. And then demanding action.

          I get it now. It's not the Tea Party. It's the Neo-Confederate Party.

          by DavidHeart on Fri Oct 18, 2013 at 05:56:57 PM PDT

          [ Parent ]

          •  For one thing, they didn't get the initial set of (0+ / 0-)

            requirements until last Spring.  Due to politics.

            If you have never had to deal with the fact that one click on the front-end system may depend on multiple back-end systems to have their data in-synch for successfully returning what is required to form the content of a response, then I'm not sure why you are even commenting here.  These systems are bringing together data from all sorts of sources and the front-end likely expects all the data to be available on-demand.  From what I'm reading, multiple requests at a certain level of load seem to be hanging up and sometimes returning a completed requests (e.g., for a registration of a family member), but later attempt to retrieve that record from the credentials originally provided are not returning correct data - or, returning any data at all.  Those are examples of timing and synchronization issues under load, most likely.  This has been a rush job since the beginning, so I'm assuming they are reusing as much as possible to make it work - which likely led to back-end service interface redesigns and a gazillion BDAs scratching their heads in parallel with the UA teams.

            Some of the end use issues reported sound like problems trying to match identities/profiles with credentials from prior registration attempts.  This could be caused by anything from corrupted database entries to a hard reliance on persistent cookie information and so forth.  So many other things at possible causes - much of them likely related to the fact that people gave up when systems took too long to visually respond and the system is handling so many incomplete flows that it wasn't fully designed to correct for.  This is not something you or I can prescribe an answer for without knowing more of what's actually happening in the system flows.

            Nothing I've heard implies that the entire thing needs to be thrown out at all: registrations are going through, issues are being worked and I hope that they are seeing a path through the next few months to alleviate most of the Sev 1 and 2 issues in a realistic manner.  I've seen far worse move-to-production issues for large systems and great recovery for most of them (not all).

            Unlike you, I don't presume to know all of what's going on or at fault - I'm just looking at symptoms and making guesses, based on my history from smaller to immense electronic and web-based transaction systems.

            "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

            by wader on Fri Oct 18, 2013 at 07:57:07 PM PDT

            [ Parent ]

            •  Think asynchronous. Get queued. (0+ / 0-)

              "one click on the front-end system may depend on multiple back-end systems to have their data in-sync"

              At this time, HealthCare.gov should not aspire to be a real time medical records provider. Right now it's best function is that of serving ACA. That is, to be a facilitator for Insurance Exchange. For now, don't even think of being a medrecs provider.

              HealthCare.gov system does not have to provide services in real-time to achieve first generation success.

              I get it now. It's not the Tea Party. It's the Neo-Confederate Party.

              by DavidHeart on Fri Oct 18, 2013 at 08:56:18 PM PDT

              [ Parent ]

              •  It's offering registration with rates based on (0+ / 0-)

                information both provided by users and queried from the backend.  We aren't aware of the heuristics involved in such calculations - I've seen plenty of cases where existing "fulfillment" systems also hold the pricing routines, for example.

                That could very well require some real-time flows, depending upon the architecture that they had to hurriedly put in place.  I'm willing to bet that since Spring they didn't have time to do much more than hook up a bunch of legacy systems and hope the front-end could handle the waits, minimizing asynchronous/isochronous calls in their design as best possible and caching whatever they could closer to the front end.  This would imply possible design of new service interfaces for backend systems, service level upgrades for systems not previously requiring realtime access, etc.  At the same time, I'm sure there's much which is being replicated for easier access and use by the front-end system, but would easily bet money it's far more complex than that.

                No idea if we'll ever find out what the internal architecture looks like, though I'd be interested to take a look.

                "So, please stay where you are. Don't move and don't panic. Don't take off your shoes! Jobs is on the way."

                by wader on Fri Oct 18, 2013 at 09:22:03 PM PDT

                [ Parent ]

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site