Most webapps aren't immediately released to millions of users
When Google or Facebook are releasing a new app, do you think they just flip a switch and release it to the world? Absolutely not! They slowly release it to beta testers, or add the feature to a subset of users. They encounter bugs, fix them, slowly scale up their infrastructure, and wait until they've seen it succeed before opening it up to the general public.
HealthCare.gov, on the other hand, launched to the entire nation at once. It didn't just have to deal with traffic from thousands (hundreds of thousands? millions?) of people looking for health plans for themselves; I'm sure a lot of curious people (like myself) went there just to check it out. There was no chance to work out the bugs, and no chance to gradually scale the infrastructure. This was almost guaranteed to be a problem.
Now, perhaps they should have slowly rolled out the site. I suspect, however, that the reasons were political. How do you decide who gets to sign up for healthcare first? By state? By last name? Random drawing? People who signed up ahead of time? And how do you explain to everyone else that they have to wait? People expect this from Google, but a highly charged political issue like the ACA makes it dicey.
Much of what happens in the exchange is out of their control
HealthCare.gov is an extremely distributed application. John McDonough, a health policy professor at the Harvard School of Public Health, described it as:
"an enormously complex task. The number of systems that have to work together – federal, state, insurance companies, the Internal Revenue System – the number of systems that have to align here is pretty daunting."If there's a bug in the IRS endpoints, there's going to be a problem. If one of the hundreds of private insurance companies' systems can't handle the increased workload, there's going to be a problem. And as someone who has built plenty of distributed systems, I know that end users don't say "oh, it must be a problem with system X that this app depends on". They think it's a problem with whatever webapp they're using, and don't care what's going on in the backend.
Nor should they care. But professional IT people should know better.
Most webapps aren't serving as a referendum on a heated national controversy
Frankly, there are a lot of people that want HealthCare.gov to fail primarily because they want the ACA to fail. And many people who oppose the ACA will still have a need to use HealthCare.gov. New Google apps are not typically tied to such a heated political issue, and are generally not used by people who don't want to use them.
So let's have some empathy for the people who build HealthCare.gov
This application was an enormous undertaking, with many challenges that few (if any) systems out there need to deal with. My expectation and hope is that the bugs will be worked out in the coming weeks, as happens with any new system. Whether the Affordable Care Act will be a success is yet to be seen; but the healthcare exchanges that support it were built by regular IT professionals that are just trying to do their jobs. Let's cut them some slack.
I'd have empathy if they weren't awarded a massive federal contract to do this. I feel bad for the programmers who'll get blamed, but not the people making money off of this. Disgusting.
ReplyDeleteWell yeah, that's who I'm talking about: the people who built it. The programmers, sysadms, etc. Not the executives and lawyers who got most of the money.
DeleteI dig it.
ReplyDeleteAsking for empathy from autistic nerds is like asking for rain in a desert.
ReplyDeleteI wasn't able to look at healthcare.gov during the first week of operations, before the upgrades. However, I was able to take a quick look at coveredca.com the day after its launch. Please take a look at the network activity generated by page views, and tell me if it looks professional. Is there a way to justify serving images and that many JavaScript files from the app server?
ReplyDeleteIs there a way to justify 404s on a production site?
CoveredCA received about $900,000,000 in grants from the Federal government to build that exchange.
The number of hits (in the millions), the number of visits (compounded by people trying and retrying) seriously overestimate the actual number of people who went to those sites. Both numbers are inflated due to the plumbing not being done well.
Part of the problem with the plumbing is a design problem: Requiring people to create accounts and validate information before they can do anything useful creates a bottleneck. However, part of the problem, during the launch, was that not everything that could have been done to handle high demand had been done.
I agree, the number of HTTP requests is a problem; there should be some seriously minifying and combining going on. Hopefully they are at least serving those from a CDN (I read somewhere that the federal site does).
DeleteHowever, poor packaging of front-end resources was not the cause of most (if any) of the early problems. Without access to the backend code, we can only guess at what the problems were. Perhaps they really did a terrible job; perhaps my points above explain the problems. My point is that we don't know.
Also, most of the cost figures out there are garbage. I don't know anything about the CoveredCA ones, but some of the federal numbers being thrown around were made by combining all contracts to that company, most of which had nothing to do with HealthCare.gov, some going back to 2008.
The number I am citing is from http://www.ncsl.org/issues-research/health/state-actions-to-implement-the-health-benefit.aspx and refers to the grants given to each state to set up exchanges. That is, those numbers represent a lower bound on the total amount of resources available to build exchanges. I don't care if they burned 99% and spent only 1% on actual stuff ... What matters is the total amount of resources that are now not available to do other things.
DeleteThe total amount of Federal grants to states to implement exchanges is somewhere in the neighborhood of $3.5 billion. Note, that is a lower bound on the total cost of implementing all exchanges.
Until there is transparency about what the exchanges are doing in the background, I am perfectly willing to take 404s in the front-end as an indicator of what's going on behind the scenes.
You forget the sad state of the visible user-end code and development. The front page makes 60-odd HTTP requests, not including images! What the hell? Did you look at the Javascript code too? The CSS is a nightmare as well.
ReplyDeleteThere isn't an excuse for this shoddy work besides an incompetent contractor being far over-paid ($631mil was it?!).
Indeed, agreed. I'd just add that
ReplyDelete1. Nothing's been officially disclosed but the problems with the servers reek of denial-of-service attacks.
2. Every state has their own insurance companies. Every insurance company probably has their own set of APIs, and many of whom probably exposed for the first time. Mapping hundreds of disparate APIs is a herculean task, comparable to getting all the of exchanges, trade houses, etc... to build current Wall Street stats. There's a reason Bloomberg doesn't have too much competition for their consoles.
3. As a developer, so long as your site doesn't spit out white-screen 404 or 500 errors but at least keeps the site up during the initial roll out, I count that as a win.
4. It's pretty clear that this is the first time the administration really tackled a large development task. And don't diminish it, this would be a huge product even if handled by Google or Microsoft. I guarantee you that there are administration officials who now know the difference between "unit" and "stress" testing. In retrospect Secretary Kathleen Sebelius really needed a cadre of advisors with good industry experience
Thank you for this article.. Travelers from Canada who wish to travel the Kenya need to apply for visa for Kenya for Canadian citizens through online e visa portal. That offers the fast and secure visa services.
ReplyDeleteThe gambling industry in the United States, and the impact
ReplyDeleteLas Vegas, Nevada: A study by a researcher 안전 공원 at the University of California has found that if 베트맨 토토 넷마블 people play 슬롯 slots 슬롯 online, 다음 드 they're probably as likely to