Inside the Company That Built Healthcare.gov
December 27, 2013 11:33 AM Subscribe

That lack of expertise explains why in building healthcare.gov, the government turned to industry contractors; in particular, to CGI Federal, a subsidiary of CGI Group, a Canadian company. To those uninitiated in the dark art of government contracting, it seems scandalous that CGI, a company most Americans had never heard of, a company that is not located in Silicon Valley (where President Obama has plenty of Internet superstar friends who could have formed a dazzling brain trust to implement his signature legislation) but rather in Montreal, could be chosen as the lead contractor for the administration’s most important initiative. While right-wing news outlets have focused on the possible relationship between Toni Townes-Whitley, senior vice president for civilian-agency programs at CGI Federal, and Michelle Obama, both of whom were 1985 Princeton graduates, CGI’s selection is probably more an example of a dysfunctional system than it is a scandal. “A lot of the companies in Silicon Valley don’t do business with the government at that level [the level required for federal contracting],” explains Soloway. “It is very burdensome, and the rules make it very unattractive.” Indeed, government contractors have to meet a whole host of requirements contained in a foot-thick book, including cost accounting and excessive auditing, to prove that they are not profiting too much off the American taxpayer. Hence, there tends to be a relatively small, specialized group of companies that compete for this work, even on such critical matters as healthcare.gov. - Accounting for Obamacare
posted by beisny (106 comments total) 26 users marked this as a favorite

"of the 80,000 information-technology workers in the federal government, for every one person under 30, there are 10 people over 50."

That's kinda... top heavy, I think.
posted by pwnguin at 11:37 AM on December 27, 2013 [3 favorites]

dark art of government contracting

You know when a phrase is so right it kind of hurts?
posted by EvaDestruction at 11:40 AM on December 27, 2013 [33 favorites]

There are massive problems with government procurement and contracting at all levels of government; the rules are meant to protect taxpayers from waste and fraud, but IMO, they end up perversely encouraging it in practice, and the procurement/contracting processes are often in tension or directly at odds with good project management practice. The simplest solution would be for the various governmental entities to build their own capabilities up, but that would mean investing more in government labor and expanding the scope of services, which seems politically impossible right now. Just skimmed the article but looking forward to reading in-depth later...
posted by saulgoodman at 11:43 AM on December 27, 2013 [18 favorites]

This is not a remarkable story. Pretty much every big government IT implementation that I can think of anywhere is a huge debacle. Go read just about any 10-K for EDS or CSC. There are lots of reasons for this.

Also accounting for roll-ups and big long multi year projects the recognize revenue on a % of completion basis is fraught with difficulties. These guys always have issues with over stating profitability.
posted by JPD at 11:46 AM on December 27, 2013

CGI does IT for my employer and is cursed by virtually every computer-using employee on at least a weekly basis. Based on my experience with their roll-out of numerous company-wide software upgrades, the fact that healthcare.gov didn't work at launch was entirely foreseeable.
posted by cardboard at 11:47 AM on December 27, 2013 [3 favorites]

In hindsight the greatest mistake was believing this would, or even could, launch cleanly.
posted by mcstayinskool at 11:50 AM on December 27, 2013 [2 favorites]

the rules are meant to protect taxpayers from waste and fraud,

That's their nominal purpose. Their actual purpose is to prevent any individual politician or government employee from being blamed for waste and fraud.
posted by jon1270 at 11:51 AM on December 27, 2013 [37 favorites]

While I only work for a state government, we sort of act as a local branch of the federal government. From my experience, it's not so much "dark arts" that go into contracting, but a lot of regulations and legal agreements that require a ridiculous amount of back-and-forth. And if the agreements aren't really well written in the beginning, you spend a ton of time trying to go back and re-negotiate aspects of the original contract(s).

Again, from my limited experiences, the real "waste, fraud and abuse" comes from needing 3rd party contractors. Instead of building an internal support system, contractors actually create another level of bureaucracy and delay, incurring additional costs accordingly. But in the end, the government supported private businesses, and as jon1270 noted, any blame can be shifted to the contractor. "We, the government, set up a good program, but it was the contractor who squandered resources and didn't produce a viable product on time!"
posted by filthy light thief at 11:55 AM on December 27, 2013 [15 favorites]

That's kinda... top heavy, I think.

And is it top heavy because the federal government is a stodgy club for people who think the Internet is a series of tubes, or is it top heavy because like all federal departments it's been starved of funds to bring new hires on board and limps along with a handful of people who had just enough seniority to avoid being let go during the last budget reduction?
posted by RonButNotStupid at 11:56 AM on December 27, 2013 [33 favorites]

it's bizarre how deeply "free market" ideology has penetrated everything. you start with the assumption you have to outsource a vital governmental program i.e. the ACA website and when there are problems with the product, it must be because the outsourcing market wasn't free enough.

the problem with big governmental infrastructure projects is that there aren't enough government engineers on permanent staff who understand the problems well enough and have enough authority to both decide what parts of a project will be outsourced and be able to manage the resulting contractors. it's the same reason why private sector outsourcing contracts go bad. the US needs a Corps of IT Engineers, but you will never hear anyone advocate for it.
posted by ennui.bz at 11:57 AM on December 27, 2013 [63 favorites]

is it top heavy because the federal government is a stodgy club for people who think the Internet is a series of tubes, or is it top heavy because like all federal departments it's been starved of funds to bring new hires on board and limps along with a handful of people who had just enough seniority to avoid being let go during the last budget reduction?

I'd say it's a perfect storm of both.
posted by philip-random at 11:59 AM on December 27, 2013 [2 favorites]

JPD: This is not a remarkable story. Pretty much every big government ~~IT implementation~~ contracted job that I can think of anywhere is a huge debacle.

Big jobs are never "normal," so there is no prior examples to use as templates. Small jobs are great, because clear procedures can be set up, and a contractor can be efficient and effective at getting the job done. But big things are never completely understood from the beginning, so the scenario changes as the understanding improves.

RonButNotStupid: is it top heavy because the federal government is a stodgy club for people who think the Internet is a series of tubes, or is it top heavy because like all federal departments it's been starved of funds to bring new hires on board and limps along with a handful of people who had just enough seniority to avoid being let go during the last budget reduction?

Probably more of the latter, which creates a situation of the former. Government jobs don't pay well, especially compared to the free-market contract jobs that government now relies upon. So you get a bunch of old-timers who finally make decent pay, and people who are content with a safe job that provides reliable pay and benefits.
posted by filthy light thief at 12:00 PM on December 27, 2013 [2 favorites]

I would say they're also pretty light on the non-governmental side of the relationship, too. Their effect can be to put unrealistic constraints on the project team but they also often also make it too easy for contractors to blow off their contractual obligations while still getting paid. Ultimately, there's just a lot about these processes that are kind of arbitrary and that allow politicaly connected individuals to have too much influence over the outcomes. It's a mess from all angles and there is culpability for that mess on both the private and public sides of the issue.
posted by saulgoodman at 12:00 PM on December 27, 2013

ennui.biz nailed a major part of the problem, but it's a structural problem ultimately,because the decision-makers (pols and their appointees) ultimately aren't qualified to make competent technical decisions.
posted by saulgoodman at 12:03 PM on December 27, 2013 [1 favorite]

people who had just enough seniority to avoid being let go during the last budget reduction?

I'm just thinking out loud here, but maybe seniority isn't a very good way to make staffing decisions?
posted by pwnguin at 12:04 PM on December 27, 2013 [4 favorites]

actually government projects usually are less profitable. The issue really is the scale is just so massive and the ways in which money leaks out.
posted by JPD at 12:06 PM on December 27, 2013

It's been about 20 years since I bid on contract jobs for the software company I worked for. Looks like the process still needs a massive overhaul. But so much in the government does, it must be difficult to figure out where to start.
posted by _paegan_ at 12:13 PM on December 27, 2013

It's always good to hear that my "don't attribute to malice what can be explained by incompetence" mantra is once again accurate.
posted by Joey Michaels at 12:19 PM on December 27, 2013 [1 favorite]

I'm just thinking out loud here, but maybe seniority isn't a very good way to make staffing decisions?

In the case of government jobs, the calculation was made a long time ago that paying less salary but ensuring greater job security was the better bet. Seniority is a huge part of job security.
posted by Etrigan at 12:23 PM on December 27, 2013 [5 favorites]

...maybe seniority isn't a very good way to make staffing decisions?

Thing is, because of the sheer size, age, and complexity of government systems, much of the time the senior people are the only people with adequate knowledge to fix/maintain them. I mean, I don't know too many 20-something geeks who have COBOL in their toolbox, do you?
posted by Thorzdad at 12:23 PM on December 27, 2013 [7 favorites]

Well

Close enough for contracting work.
posted by duffell at 12:25 PM on December 27, 2013 [1 favorite]

So: allow the lowest profits in the industry, and mandate an arcane set of rules that mostly exist to ensure you're only getting the allowed profits. What other outcome should people be expecting? Perhaps our country's flag should be a giant stick and a paltry carrot.

I'd like to think that in an ideal world, government contracts would be less profitable, but that difference would be more than made up for by the prestige and bragging rights that would come from having successfully implemented a Really Big Thing.
posted by RonButNotStupid at 12:26 PM on December 27, 2013

I'd like to think that in an ideal world, government contracts would be less profitable, but that difference would be more than made up for by the prestige and bragging rights that would come from having successfully implemented a Really Big Thing.

I'm sure most companies and people would prefer cold, hard cash instead of bragging rights.
posted by Brandon Blatcher at 12:34 PM on December 27, 2013

...of the 80,000 information-technology workers in the federal government, for every one person under 30, there are 10 people over 50.

Insider view: This is a terrible statistic. While you would think the hipster coders might bring 'new' and 'fresh', a lot of the over-50 IT guys are who know the org structure of govt hosting, the information assurance guidelines (AND how to implement them), and have a clearance.

I dunno specifically about healthcare.gov and it's environment, but lots of times I have to direct the younger guys to back up and start over because they rush forward with open source components and frameworks that just aren't going to pass security guidance.

A bunch of bullshit php is no way gonna fly.

Of the significant contributors, 'over 50' is not one.
posted by j_curiouser at 12:36 PM on December 27, 2013 [35 favorites]

If you enjoy reading epic sagas of government IT procurement failure, here's another one (involving Fujitsu and the Northern Territory government's asset management system). It's basically a political rant but still fascinating.
posted by A Thousand Baited Hooks at 12:38 PM on December 27, 2013

I'm sure a Google-scale company would jump at the chance to do something that would both earn it money and substitute for some lobbying. They probably know a thing or two about data security, too. It really is the regulations.
posted by topynate at 12:39 PM on December 27, 2013 [1 favorite]

As a government IT worker, the problems aren't very different from what I've seen at very large risk-averse organizations: the focus is on rules and upfront planning, rather than responsibility, and that pushes strongly towards a waterfall process where every detail is decided before work begins so the cost and – far more importantly, risk – can in theory be fixed. In practice this is always a disaster because, as Clay Shirky memorably put it, this “amounts to a pledge by all parties not to learn anything while doing the actual work”.

The fix is actually quite simple on the macro level: give managers more discretion and in-house staff but hold them accountable for results. On the micro level, having in-house staff – i.e. people who work closely with the ultimate users and learn the agency business and culture – and smaller, iterative projects is much more likely to produce a good system. This is somewhat common in the private sector (academia is somewhere in between) and isn't unprecedented in government (I work in a fairly agile group and word has it that most of CFPB functions this way) but it's a major change from both the way most agencies function and, far more importantly, how Congress tends to structure things. Even healthcare.gov is unlikely to be big enough to get a voting majority to hire more government programmers because the debate is predicated on distrust for government employees and thus features more micromanagement and less discretion, which is basically the polar opposite of should happen.
posted by adamsc at 12:39 PM on December 27, 2013 [17 favorites]

I'm sure most companies and people would prefer cold, hard cash instead of bragging rights.

Bragging rights turn into a whole lot more cold hard cash over the long term. "Oh," says Google, "You just implemented Big Cool Tech Thing on a massive scale. We'll buy you at a massively overinflated price, k?" or "Hey look at Big Tech Company," says a VP at McDonalds, "we should get them to implement our Big Tech Thing."

Lather, rinse, repeat.

I'm sure a Google-scale company would jump at the chance to do something that would both earn it money and substitute for some lobbying.

Also, this.
posted by feckless fecal fear mongering at 12:41 PM on December 27, 2013

I'm sure most companies and people would prefer cold, hard cash instead of bragging rights.

interesting that I'm watching The Right Stuff right now c/o this recent FPP. One thing that quickly comes clear is that the guys who broke the sound barrier, made it to the moon, pulled off all manner of other wild and wonderful stuff were NOT in it for the money.

Indeed, if the movie's to be believed, Chuck Yeager (who first broke the sound barrier), refused extra payment for the extra dangerous work -- just took his normal (and paltry) air force (or was it army?) salary, while a high-priced civilian pilot watched jealously from the sidelines.

guess times have changed.
posted by philip-random at 12:42 PM on December 27, 2013 [2 favorites]

Bragging rights turn into a whole lot more cold hard cash over the long term.

Isn't this a rewording of "it'll be good for your resume/portfolio" spec-work ploy?
posted by rhizome at 12:42 PM on December 27, 2013 [4 favorites]

A large part of the issue is that there are two axes of success: success at creating a stable and robust computer system and success at navigating the requirements of the government. That's what the article is getting at, but it lacks a little bit of nuance in terms of understanding the "state of the art," not in technology, but in technology project planning (which is often what counts). The healthcare.gov debacle is just the latest and highest-profile failure of an attempt to do "big design up front," to build a massive monolith and plant a flag atop it. Understanding of IT project management has evolved over the decades, outside the government at least, to a realization that this just doesn't work. You build in increments. Your first iteration can tell people whether they'll be allowed to claim government subsidies. Then, perhaps, you add functionality to tell people plans that they can purchase within their state, and who to contact. Then you add functionality to allow insurers to sell their plans through your site if they wish, perhaps offering incentives for early adopters. Then you have a cut-off date where insurers must register plans for sale through the site.

You just cannot build a perfect system, all at once, and expect it to work for any meaningful value of "work."
posted by sonic meat machine at 12:44 PM on December 27, 2013 [12 favorites]

guess times have changed.

The military has its own idiosyncratic set of motivations. Perhaps you could say that the free market has destroyed the nation's sense of civic duty, but I don't think that's particularly germane here: it's already gone, and the system is not (or no longer) set up to encourage it.
posted by rhizome at 12:44 PM on December 27, 2013 [1 favorite]

j_curiosier: I really like the distinction between someone with 30 years of experience or 1 year repeated 30 times. I've worked with both and while the difference in results is obvious, it can be surprisingly hard to handle within an organization: at a private employer, they were extremely disinclined to fire anyone, which is perhaps pleasingly loyal but enormously inefficient if the retired-in-place were left in charge of key decisions. This was corrosive on newer staff: the ambitious younger employees tended to leave but those intending to coast similarly would stick around because it was the path of least resistance.
posted by adamsc at 12:46 PM on December 27, 2013 [1 favorite]

CGI, a company most Americans had never heard of, a company that is not located in Silicon Valley (where President Obama has plenty of Internet superstar friends

This was a big fuck up, but as an employee of a small tech company that most Americans have not heard of and is not located in Silicon Valley, I cannot even begin to describe the number of things that are wrong with this sentence.

For one, we're quite good at what we do. But, enough of that....

Creating products in Silicon Valley is a completely different ballgame from engineering a government service. While it's easy to shout "Herp Derp Government Inefficiency!," it's also easy to forget that a government service has many (very difficult) requirements that need to be in-place on Day 1, in order to provide that service to the entire population the most fair and egalitarian manner possible. In many cases, it's complicated and inefficient for some very good reasons.

In some cases, that means supporting IE8, in others it means religious adherence to accessibility standards, in others it means translating everything into Cherokee, in others it means answering to a wide net of stakeholders where no clear hierarchy is defined. It can also mean that you need to make the system work for people who have no "traditional" first or last name, or don't know their date of birth. It's hard.

In the case of Healthcare.gov, you can also throw hundreds of competing insurance carriers into the mix.

In the private sector, you can eschew 0.01% of your potential customer base, to avert a large engineering effort to support that 0.01%, especially when the payoff would be minimal. When you're a government, it's not an acceptable compromise to throw 0.01% of your citizens under the bus.

Obviously, CGI weren't able to meet these goals, but I'm not sure that a Silicon Valley firm would have fared much better. It requires a completely different approach. There are many ways in which the government is wrong-headed about technology, but the private sector is not a meaningful or helpful comparison.

The government is not a business, and I'm perplexed that people keep demanding it to run like one (without being willing to accept any compromises that would come from living under an all-powerful CEO).

Also, it doesn't seem crazy to me that we'd hire a foreign firm to build our national healthcare platform, given that other countries clearly have a lot more experience with this stuff than we do.
posted by schmod at 12:49 PM on December 27, 2013 [66 favorites]

Nice to see a dose of xenophobia mixed in there too. CGI isn't just incompetent, they're incompetent and Canadian.
posted by GuyZero at 12:50 PM on December 27, 2013

Isn't this a rewording of "it'll be good for your resume/portfolio" spec-work ploy?

I suppose, but sometimes there's strategic value in taking less money; reputation and goodwill also count towards a company's valuation. I mean it's often used as a way for employers or clients to cut down on costs any way they can (see: interns), but that doesn't mean it's always a bad move for the employee.

And sometimes the challenge is really all that matters--see the bit about Chuck Yeager above.
posted by feckless fecal fear mongering at 12:53 PM on December 27, 2013

...but that difference would be more than made up for by the prestige and bragging rights that would come from having successfully implemented a Really Big Thing.

But, does the tech sector have any respect for gov work, no matter how well delivered? There seems to just enough free-market worshipfulness in SiliValley to make me think that stooping to work on a .gov project (that wasn't actually rocket science) would be seen as a stain on your reputation.
posted by Thorzdad at 12:58 PM on December 27, 2013

Bragging rights turn into a whole lot more cold hard cash over the long term.

They can, but don't always and in the meantime, you can't use bragging rights to pay, well, anything.

I hear what you're saying, but I can't blame a company or person for deciding not to step into this particular minefield. Implementing this website wasn't just a straight forward technical challenge, it sounds like there were all sorts of political hot rocks to juggle.

None of this excuses the familiar of the company or the government for the problems the site had, but in the end it's not surprising that there were large problems.
posted by Brandon Blatcher at 1:01 PM on December 27, 2013

does the tech sector have any respect for gov work, no matter how well delivered?

The relationship between technology and the mechanism of government is one of the key issues for the next decade. While there might be prominent market fundamentalists in Silicon Valley, as schmod says, they're in a different line of work.

At the same time, you're also seeing people take notice of the example of the UK GDS -- which is slowly ending an era of large outsourced IT contracts -- while Code For America is starting to gather talent and momentum of its own.
posted by holgate at 1:06 PM on December 27, 2013 [1 favorite]

Building a huge system does not confer "bragging rights," because huge systems are prone to failure and are a cost center. The bigger the system you have, the more complex its inner workings become, the more powerful the forces of entropy acting on it, and the more expensive it is over time.

This is the source of the unix philosophy, where complex systems are built from dozens of small systems wired together. (See The Unix Philosophy: A Brief Introduction.) It's also the motivating factor behind Service Oriented Architecture, and it was the idea behind object-orientation (although that never panned out as OO was eventually just used to build massive monolithic projects).
posted by sonic meat machine at 1:10 PM on December 27, 2013 [7 favorites]

I hear what you're saying, but I can't blame a company or person for deciding not to step into this particular minefield.

Oh goodness, me neither. Unless you are, as noted, doing (the equivalent of) actual intergalactic rocket science, I don't see why anyone would want to engage with the government to deliver any project of scale at all. I guess some people probably think:

1) Bid for huge money govt contract

2) Fail miserably

3) ~~?????~~ Get paid double what you estimated anyway

4) Profit. Retire to an island somewhere.

It's kind of like the startup version of do one thing well, get paid some ridiculous sum by one of a handful of companies to buy you out, then buy yourself a personal jet thing, only with the added bonus that you not only do you not need to be competent, you don't even really need to try too hard because so much government oversight everywhere is so lax.

But even if you're doing it for the glory or whatever, you get snared in these Byzantine rules and regulations... so you're screwed anyway.

It's an ugly mess. Government needs to provide certain services to the people, and the people deserve to have the best possible minds behind those services, at a reasonable cost.

Instead we're getting... this.
posted by feckless fecal fear mongering at 1:14 PM on December 27, 2013

Instead we're getting... this.

Remember, too, that "this" began life as a terribly compromised and fucked-with program, thanks to a lot of professional-grade obstruction and roadblocking by parties opposed to the concept in general. A bit like trying to build a home on top of a combination minefield and quicksand bog.
posted by Thorzdad at 1:24 PM on December 27, 2013 [10 favorites]

Creating products in Silicon Valley is a completely different ballgame from engineering a government service. While it's easy to shout "Herp Derp Government Inefficiency!," it's also easy to forget that a government service has many (very difficult) requirements that need to be in-place on Day 1, in order to provide that service to the entire population the most fair and egalitarian manner possible. In many cases, it's complicated and inefficient for some very good reasons.[list of reasons]

I disagree. First, there are other reasons for inefficiency besides what you mentioned; you just listed the ones with an upside. Second, if the mandatory additional costs and complexities are too much, then all those marginal groups can't actually be served even though they are supposed to be on paper. This project is an example of that because a lot of people were not able to use the exchange when they were supposed to be able to use the exchange.

I dunno specifically about healthcare.gov and it's environment, but lots of times I have to direct the younger guys to back up and start over because they rush forward with open source components and frameworks that just aren't going to pass security guidance.

If you are not exaggerating, I think you are part of the problems with projects like this one. You tell your young guys to start over? How'd they even get started; shouldn't you have been proactively educating them about acceptable technology choices? Or, if you don't get involved soon enough to prevent them from making mistakes, but have authority to force a redo, doesn't that mean some dysfunction keeps you from working with the manager of these developers early on? You should try to do a better job! You count.
posted by michaelh at 1:35 PM on December 27, 2013 [2 favorites]

CGI isn't just incompetent, they're incompetent and Canadian.

Um - so, (no comments on competency) - however, considering that the various times I was called-out to support CGI activities in the USA, they have quite a few American offices with quite a few American staff. (Just as I was Canadian, working for an American company with offices in Canada)

Look... Once your company has turned into a "3-letter" behemoth and makes every attempt to procure these government engagements, nothing good will happen... This is an IT-story as old as the hills, you can also find public news of debacles with EDS(HP) in the U.K., IBM in Canada, etc...
posted by jkaczor at 1:35 PM on December 27, 2013

adamsc - i see your POV. yes, anybody who is stagnating technically or can't provide leadership or consistently makes poor technical choices should be cut loose. more generally, that goes for anyone, regardless of age.
posted by j_curiouser at 1:38 PM on December 27, 2013

It is possible to make big systems that perform well on first launch. It is possible to meet all the regulations imposed on government contracted developers. Maybe you could do both, but it would probably take decades.
posted by idiopath at 1:41 PM on December 27, 2013

hi michaelh - i am often called in only after a project is in trouble and uncover these earlier decisions during a review. in a greenfield effort, i know how to prep the team. my point is, there are a lot of reasons you might want seasoned engineers who have seen many projects large and small fail/succeed in different ways in the Fed IT space. [fyi - i'm not over 50. it's not the years, it's the mileage.] </agederail>
posted by j_curiouser at 1:45 PM on December 27, 2013 [4 favorites]

Oracle is as Silicon Valley as they come and yet they managed to make a complete mess of Oregon's health care exchange. The Federal Exhange was an order of magi other larger than Oregon's and it is actually working today. It was an incredibly complicated project that required integrating hundreds of vendors and numerous federal state and local health care regulations and agencies. How many companies have totally fucked up their ERP and CRM solutions? Heck even tech companies often have scary bad back office systems. Most of those systems are starting with a lot more established software and data standards. A month of shakeout after going to prod and things are mostly working. That's a miracle. The cost variance to tax payers was minimal. Compare that to the crazy cost overruns and delays in most private sector software consumer web implementations.
posted by humanfont at 1:49 PM on December 27, 2013 [19 favorites]

I work some with supplier quality, and while I'm in industry, I bet the principles are the same. All suppliers/contractors are subject to an approval process whose rigor is based upon the risk of the situation. This is a time-consuming process that does not favor new suppliers, for why would you blow a couple weeks on a systems and financial audit when you have a pre-approved group? Considering the political criticality of national healthcare, this task probably got a "high" risk rating, meaning more work and up-front evaluation for a new contractor.

It sounds like CGI did not do a good job, but I can understand why it makes a lot of sense to use a supplier who is pre-approved for government work. I'm sure that the audits, security clearance, and testing of a new contractor would have eaten valuable months. The final product is far from perfect, but I wonder if this would be considered a colossal screw-up if CGI was in charge of releasing, say, the website for the National Basketball Association.

At the end of this, CGI's supplier rating may be bumped down, and perhaps they'll be investigated for fiscal malfeasance. They made a product that could not meet the demand, but I'm not sure anybody would have, which is made more salient by an antagonistic rival party who looks for any excuse to pounce.
posted by Turkey Glue at 1:51 PM on December 27, 2013 [1 favorite]

j_curiouser: strong agreement – the underlying problem is the incentive system in most of the government. If excellence had more rewards and underperformance more risk, it'd go a long way towards improving the situation across the board.
posted by adamsc at 1:53 PM on December 27, 2013 [1 favorite]

There seems to just enough free-market worshipfulness in SiliValley to make me think that stooping to work on a .gov project (that wasn't actually rocket science) would be seen as a stain on your reputation.

Depends. On the consumer side? Most likely. But even though consumer software is most of what people think of when they think of Silicon Valley, enterprise software huge too. (Think of where Larry Ellison made all of his money.) And at least some enterprise companies would *love* government projects.
posted by asterix at 2:03 PM on December 27, 2013

Thing is, because of the sheer size, age, and complexity of government systems, much of the time the senior people are the only people with adequate knowledge to fix/maintain them. I mean, I don't know too many 20-something geeks who have COBOL in their toolbox, do you?

This doesn't contradict the idea that seniority might be a bad idea. Institutions that don't operate by way of seniority also try to keep the people with adequate knowledge to fix/maintain their systems. They do it because they need those people. In contrast, institutions that strongly rely on seniority are prone to keeping senior people without adequate knowledge to fix/maintain their systems, because, well, seniority. Seniority can also set up a perverse incentive that discourages senior people from becoming essential skilled employees, because their positions are inherently more secure.
posted by 2N2222 at 2:03 PM on December 27, 2013 [1 favorite]

I'm just thinking out loud here, but maybe seniority isn't a very good way to make staffing decisions?

Seniority in itself isn't the problem. The unspoken premise here is that you shouldn't get seniority if you're incompetent because a well run office either trains up or fires the incompetent. If you have a lot of know-nothings with seniority, then obvious question to ask is why they didn't get fired or corrected along the way. And nothing about having seniority should protect incompetence right now, either.

In other words, don't blame rules like seniority for your failure to adequately manage your staff.
posted by fatbird at 2:04 PM on December 27, 2013 [5 favorites]

if the mandatory additional costs and complexities are too much, then all those marginal groups can't actually be served even though they are supposed to be on paper.

And your proposed way to resolve this is... what, exactly? Private-sector tech projects have greater freedom to choose when and how to resolve certain edge cases; public-sector ones don't. More importantly, those edge cases and "marginal groups" are often the ones for which government services simply have to work, because the private sector doesn't have much motivation to cater to them.
posted by holgate at 2:05 PM on December 27, 2013 [4 favorites]

The idea that "Internet superstar friends" want anything to do with bureaucratic systems that handle things like health insurance is kind of laughable. I think of that software as in the same realm as ERP or CRM systems -- very rule-based, process-oriented, unsexy software. It's profitable, but it's not something that most software developers are going to jump out of bed in the morning to work on.
posted by mikeh at 2:20 PM on December 27, 2013 [3 favorites]

> They made a product that could not meet the demand, but I'm not sure anybody would have

Oh, come on!

First, as I've argued here before, this isn't rocket science. The scale is large, but there all sorts of technology companies deal with that sort of size all the time.

Moreover, it's an almost perfect problem for parallelization - "buying a lot of cheap computers to run at the same time" - because there are NO dependencies between accounts, no inventory, you can simply prevent the same account from running twice at the same time, and, once you have a framework to represent the rules, you can hire 50 x ? contractors to work independently on each state's rules.

And also - you don't actually buy anything, so there isn't any payment system needed; you get a free additional verification stage because the finished form goes off to an insurance company who can catch mistakes, or fraud (because isn't that the point? otherwise, why are they getting so much money?); the amount of data per person is tiny; there is no rich media like audio, video, or animations needed (except a few "fixed assets" which you could perfectly well host on YouTube for free...)

America's the country that went to the fucking moon - and they had to invent all the technology from scratch to do this, and they couldn't afford trial and error! It constantly prides itself as the greatest nation in the world - but now we're saying, "Oh, we can't even do a routine if somewhat large project, because we're institutionally inept, and no one can do these things, and this can never change."

It's pathetic.
posted by lupus_yonderboy at 2:25 PM on December 27, 2013 [6 favorites]

It wasn't very long ago that we elected a President who promised us change. He talked about the problems in government, problems with health care, problems with climate change, problems with militarization, problems with indefinite detention, problems with surveillance, and promised us change.

I'm an older guy. I didn't believe in miracles - but I did believe that the way the US government ran desperately needed change and that it was so bad that we couldn't avoid it.

Again and again we see not just "no change" but not even an attempt at a change.

Healthcare is a perfect example. It was crippled before it even got out of the gate by Mr. Obama's relentless refusal to allow any actual change to even be discussed - and by secret deals with the pharmaceutical companies. Then, for no rational reason, Mr. Obama wasted a year trying to get the Republicans to buy into the deal - even though every day we heard over and over again from them that they wouldn't, and they never did.

Finally we got something mediocre - but we got something, something we desperately needed.

But then what does the Administration do to implement this, Mr. Obama's signature legislation? More business as usual.

The fact that the government is very bad at implementing IT is not a secret - it hasn't been a secret for decades now. But there was no attempt whatsoever to make any change in the system. Why wouldn't Mr. Obama rationally expect just the results he in fact got?

Please remember that this was a huge bill - there would have been little difficulty in putting in specifics about how this was to be implemented, because that's not what got debated, and in the end, the Republicans simply didn't get a say in the matter anyway. They could simply have set up a government technology corps (as described above, love that idea) or a host of other things - put it right into the bill - but in fact there seems to have been zero thought into how it was going to be implemented at any point.

They could have made an end run around these "bureaucratic systems" that people here seem to think are both completely unavoidable and also absolutely ensured the failure of this project. At the very very least they could have asked for it, brought public attention to the crappy job that the government does at IT and say, "WE NEED THIS in order to make this project a success."

But what do we get? Obviously, no one had even started to test it before a few weeks before it went out. Obviously, no one higher up even looked at it before it went out to the public. There's just ZERO evidence of any adult supervision going on there.

Now, I'm sure I'm going to hear, "Well, that's the way it is, there's no way to change it," or "He doesn't have the attention to deal with every single detail," but that's why we hired him - remember, he was the guy who knew the Internet? Twenty first century, voter registration apps, have you forgotten?

And Mr. Obama seems to have lots of time for drones, lots of time for surveillance, lots of time for getting involved in every tinpot little war in every godforsaken shithole in the world. What he doesn't seem to have any time for is that change we were promised.
posted by lupus_yonderboy at 2:33 PM on December 27, 2013 [5 favorites]

There's also that bit where the congress wouldn't say whether or not the system was actually going to be built, leading to a much shorter deadline. We can leave it on Obama's conscience if you like, though.
posted by mikeh at 2:37 PM on December 27, 2013 [4 favorites]

Pathological hatred of Obama is generating absurd levels of criticism well out of proportion with actual impact. Over a million people signed up for subsidized health insurance in 90 days. That's a good thing.
posted by humanfont at 2:55 PM on December 27, 2013 [9 favorites]

Pathological hatred of Obama is generating absurd levels of criticism well out of proportion with actual impact. Over a million people signed up for subsidized health insurance in 90 days. That's a good thing.

Sure, but how much of the federal exchange website did he code himself?
posted by crank at 2:57 PM on December 27, 2013 [2 favorites]

I agree with those saying that Silicon Valley doesn't have any particular expertise that would be relevant to this; I would expect that the most difficult and complex part of the project is the back-end connections to the insurance companies, the part that still isn't working well, and it's probably also the part that would trip up any more general technology company that didn't properly anticipate the dependencies between the back-end and consumer-facing sides of the project.

I've got acquaintances who worked in the "provider portal" industry, which makes web applications that are the front-ends for health insurance companies that the staff of a doctor's office would use to obtain insurance authorizations and submit claims. Not only are the internal computer systems at health insurance companies and other related healthcare institutions that have to be integrated stuff that was originally designed on mainframes and minicomputers in the 1970s and 1980s, and was based on healthcare industry information exchange "standards" and byzantine computing industry interoperability standards from the same era, but individual insurance companies usually don't have personnel who comprehensively understand their own jury-rigged quirky systems much less documentation, and the companies often don't really have a thorough or centralized understanding of their own business processes - the stuff the humans do. Different parts of the same company will have conflicting understandings of how and why the different procedures and work the company does connect to each other, the healthcare providers like doctors and hospitals, Medicare, and other healthcare institutions, they tell me.

Consequently contractors and internal IT teams have historically had a high failure rate for projects making a web front-end for a single insurance company, and only a handful of products on the market provide a platform that properly abstracts out everything so that the portal can interact with multiple insurance companies on the back-end. And only a smaller proportion of those actually provide a unified interface so that the doctor's office and hospital staff can do every transaction with the private insurance they'd need to do over the web.

So, the few dominant technology companies in that industry and any similar industries are probably the ones that actually have the expertise and experience that would have been necessary to plan ahead of time and successfully build healthcare.gov, but consequently they'd also have known well enough not to bid on the project in the first place.
posted by XMLicious at 2:59 PM on December 27, 2013 [7 favorites]

> There's also that bit where the congress wouldn't say whether or not the system was actually going to be built, leading to a much shorter deadline.

Here's a timeline of the ACA. Note that healthcare.gov launched in July 2010...

Some other highlights:

Early 2011: The Centers for Medicare and Medicaid Services — a large federal agency spread among locations in the District, Bethesda and Baltimore — takes over responsibility for building the insurance marketplaces from the unit that had reported directly to Sebelius. As a result, the work of building the federal exchanges becomes fragmented.

February 2011: The White House official overseeing work on the law, Nancy-Ann DeParle, is promoted and begins to spend only a small fraction of her time on it.

[...]

Late November 2012: Standards for insurance coverage under exchanges aren’t proposed until after Obama’s reelection, just before Thanksgiving, and do not become final until February 2013.

It seems to me that the Administration simply didn't give it much attention until it was far too late. Can you point in that timetable to "that bit where the congress wouldn't say whether or not the system was actually going to be built" and explain how this was the key problem?

> We can leave it on Obama's conscience if you like, though.

Snarking is never a good contribution to the conversation.
posted by lupus_yonderboy at 3:00 PM on December 27, 2013 [3 favorites]

But then what does the Administration do to implement this, Mr. Obama's signature legislation? More business as usual.

I just picked up health insurance, even in a non-Medicaid-extension-accepting red state, for just under $7 a month. Comprehensive insurance. I do contract (freelance content and academic adjunct) work, and do not get insurance through any employer, haven't since the recession of '08. I was paying $145 for higher-deductible insurance before. Things are going much better for me now, economically, than at any time since the recession. Even so, the lower insurance costs are huge relief for me, a significant change, and millions of others are having similar experiences. You speak for yourself.
posted by raysmj at 3:00 PM on December 27, 2013 [7 favorites]

> Even so, the lower insurance costs are huge relief for me, a significant change, and millions of others are having similar experiences. You speak for yourself.

Perhaps you missed the part where I said that this legislation was desperately needed...?

I'm very much in favor of universal health care. I'm very glad we got as much as we did. What's appalling is the shoddy job done and the lack of attention paid to it by the Administration.
posted by lupus_yonderboy at 3:03 PM on December 27, 2013

Snarking is never a good contribution to the conversation.

Neither are "Obama broke my heart SO HARD" laments that get reliably trotted out whenever... well, whenever something involving Obama comes up here. It adds little to the discussion to parade your disappointment for us, except offering us a nice derail that we can have yet again.
posted by fatbird at 3:04 PM on December 27, 2013 [2 favorites]

I'm saying it worked great for me, and the site worked fine in the end, despite everything. I have zero complaints.
posted by raysmj at 3:04 PM on December 27, 2013

The role of integration with both insurance agencies and federal agencies like the IRS keeps getting gestured at but not fully explored in this. All those different parties with disparate systems... at the beginning, I would have put giant red flags around it every chance I got. And I've seen a few talk about how there were issues like having you put in your full name including middle initial, without considering normalization of identifiers at all. The only way I could have seen this working is to build and release healthcare.gov three years ago in fixed form, and then send out fully funded tiger teams to anyone who wanted it to integrate their own systems with a target that had been set in stone, cast in bronze, frozen in time and under lock and key so no one could alter it.
posted by fatbird at 3:07 PM on December 27, 2013 [1 favorite]

They could have made an end run around these "bureaucratic systems" that people here seem to think are both completely unavoidable and also absolutely ensured the failure of this project. At the very very least they could have asked for it, brought public attention to the crappy job that the government does at IT and say, "WE NEED THIS in order to make this project a success."

Almost every single one of the rules that goes into making the bidding process such a Byzantine mess has a constituency. Creating an exception for, say, minority ownership rules or veteran hiring practices and so on would have made enemies for a bill that passed by a knife edge as it was.

Crafting a reform bill for the federal contracting process would be a year long ordeal on its own.
posted by empath at 3:09 PM on December 27, 2013 [1 favorite]

The fault is that the government took responsibility for interfacing with insurance companies, instead of vice-versa.

The government should have specified the interface, and left it up to insurers to jump on board. You want the government money, you play by government rules (API, data structures, laws, QoS, regulations, etc.)
posted by five fresh fish at 3:10 PM on December 27, 2013 [13 favorites]

It seems to me that the Administration simply didn't give it much attention until it was far too late.

You seem to be unfamiliar with the rule making process and reading from bullet points compiled by Obama's political opponents.
posted by humanfont at 3:16 PM on December 27, 2013 [2 favorites]

You just cannot build a perfect system, all at once, and expect it to work for any meaningful value of "work."

I believe NASA has a system that, when followed, works flawlessly.
posted by five fresh fish at 3:17 PM on December 27, 2013

The biggest challenge faced by healthcare reform is massive fear of change.
posted by five fresh fish at 3:18 PM on December 27, 2013 [2 favorites]

It might be worth remembering that even the vaunted Big American Silicon Valley Tech Companies are not particularly good at putting up brand new web services that can handle tens of millions of new users on the first day. That's a tough problem, and the usual approach is to find a way to avoid it entirely.

Take Google, for example. Remember when gmail and calendar, etc were first coming out? They were always "limited beta" and you needed to get an invite to join, so you had to find a friend who knew a friend who had some extra invites. By controlling the supply of invites, Google limited the total number of users at first, so they could scale up the backend and fix early problems. Once everything was running smoothly, they increased the distribution rate of invities and paused to ensure the system met the new growth rate, and repeated as necessary until eventually overall demand was finally met and everyone had a dozen spare invites that they couldn't give away.

However, I don't think it would have been politically feasible to attempt this approach with healthcare.gov -- like, what, "the new healthcare site we promised is now available in select cities"? That would not have gone over well. So I don't know how it could have been avoided for this project.

Anyway, just wanted to point out that those kind of problems are par for the Big Bang Release model, no matter who did the work.
posted by ceribus peribus at 3:28 PM on December 27, 2013 [13 favorites]

I believe NASA has a system that, when followed, works flawlessly.

Not really. NASA has a low failure rate, but they do have their failures; and they're also a unique case. NASA has complex problems, but their problems are of a particularly constrained sort: they know the mission parameters going in, and it's actually possible to enumerate the entire set of needs up front. They're working within the laws of physics (rather than of society). Nobody is going to DDoS a Mars rover, and the ISS never has to stand up to an order-of-magnitude spike in web traffic.
posted by sonic meat machine at 3:32 PM on December 27, 2013

What's appalling is the shoddy job done and the lack of attention paid to it by the Administration.

I don't think it was lack of attention, it was lack of experience. The Obama administration clearly hired the wrong people to manage the development of healthcare.gov and perhaps other portions of ACA. I can recall an experienced engineer and manager discussing a hiring situation saying, "if you are going to hire someone to do something for you, you want to find someone who has done that thing before, successfully." In this case they needed someone with experience managing the development of a large IT project of this scope. If Obama had had more hands on experience putting together large organizations of this kind and putting in place huge pieces of legislation like this, he might not have mad the mistakes that he possibly has made. Perhaps, this was a big strength LBJ did have in implementing medicare and civil rights that Obama and perhaps even JFK were lacking.
posted by Golden Eternity at 3:40 PM on December 27, 2013

[T]he problem with big governmental infrastructure projects is that there aren't enough government engineers on permanent staff who understand the problems well enough and have enough authority to both decide what parts of a project will be outsourced and be able to manage the resulting contractors. [I]t's the same reason why private sector outsourcing contracts go bad. the US needs a Corps of IT Engineers, but you will never hear anyone advocate for it.

This. I quite like the idea of a Corps of IT Engineers, though I think we also need a Corps of Aerospace Engineers as well. Defense contracts are also a problem. The government needs the ability to competently oversee contracts and take work in-house quickly if the contractor can't handle it.

I really hope the next big citizen facing government IT project will be delivered more iteratively. I don't think healthcare.gov would have been anywhere near the fiasco it has been if a basic version had been functional a year before the mandates were scheduled to kick in. It would have given much more time to work out the kinks before the pressure really hit.
posted by cosmic.osmo at 3:44 PM on December 27, 2013 [3 favorites]

In this case they needed someone with experience managing the development of a large IT project of this scope

Blizzard entertainment had been running WoW for years before the Diablo 3 launch, which went pear-shaped the second it went live.

EA has had its hands in all sorts of enormous multiplayer online setups, same deal with the Simcity launch.

And these are videogames.

Personally, I would have been gobsmacked if the ACA site launched at all smoothly, regardless of who was at the helm. They tried to flip a switch from "off" to "on". Good luck with that.
posted by kavasa at 3:46 PM on December 27, 2013 [6 favorites]

"of the 80,000 information-technology workers in the federal government, for every one person under 30, there are 10 people over 50."

It is also very convenient and misleading to leave out the 30 to 50 cohort. You know....the bulk of the workers in just about any field other than snowboarding.
posted by srboisvert at 3:47 PM on December 27, 2013 [6 favorites]

Getting computer savvy young people clearances for govn't work is a real problem -- one that my govn't employed friends either have to deal with, or complain about in the form of a the lack of good people due to their inability to get clearances. Usually this is a result of the war on drugs. For example, if you smoke a joint you can't work for the FBI for seven years. I'm not sure why you'd want to work for the FBI and not all agencies are as strict, but it is a problem.

However, healthcare.gov shouldn't need cleared workers. They don't even take in HIPAA data, let along anything that should require a TS. I'm not sure why this would be a problem in this case.

As for all the people saying govn't doesn't do tech well, you're almost right. Once you get tech that is weaponized we are excellent at it. And interestingly, one of the largest differences (from my admittedly limited understanding of it) is how the defense department is allowed to choose it's contractors versus how the rest of the govn't is. I believe healthcare.gov had bids submitted and was bound by law to choose the cheapest option. On the other hand the dept. of defense, to the best of my knowledge, is allowed to choose any bid it likes regardless of price. I presume there are some oversights on this, but yeah.
posted by yeahwhatever at 3:47 PM on December 27, 2013

And putting this in context of other failures, the IRS, the FBI and the city of New York have all had big IT program failures where the costs ran into the hundreds of millions of dollars--and those failed. healthcare.gov is actually working half-assedly a couple months after it's public pratfall.
posted by fatbird at 3:48 PM on December 27, 2013 [1 favorite]

once you have a framework to represent the rules, you can hire 50 x ? contractors to work independently on each state's rules.

A framework that can handle every state's rules is not going to be simple, or suddenly make integration easy. It's going to be Turing-complete and complicated as hell.

I believe NASA has a system that, when followed, works flawlessly.

It's not flawless, but it does work very well. The only problem is that it costs 20x more: "In an industry where the average line of code cost the government (at the time of the report) approximately $50 (written, documented, and tested), the Primary Avionics System Software cost NASA slightly over $1,000 per line."
posted by jjwiseman at 3:49 PM on December 27, 2013 [3 favorites]

And your proposed way to resolve this is... what, exactly? Private-sector tech projects have greater freedom to choose when and how to resolve certain edge cases; public-sector ones don't. More importantly, those edge cases and "marginal groups" are often the ones for which government services simply have to work, because the private sector doesn't have much motivation to cater to them.

I didn't propose a way, but I would say that the required level of complexity and expense should be reduced to the point at which the most people can be served who should be served. As I said, the marginal groups ("the ones for which government services simply have to work") didn't get a working service -- and neither did a lot of other people.
posted by michaelh at 4:01 PM on December 27, 2013

There is a working service now. It's been working since late November. The grar it all out of proportion to reality here. It is like you've never seen a website crash from heavy volume before. If you want to piss and moan about an IT fuckup, look at Target. Malware infected everyone of their credit card terminals for 30 days and they still arnt confident it is fixed. Their holiday sales took a huge hit and the liability might put them into bankruptcy. That was the IT fuckup of the year.
posted by humanfont at 4:17 PM on December 27, 2013 [13 favorites]

If you want to piss and moan about an IT fuckup, look at Target.

Also the TJ Maxx break-in. This is something that gets left out of these discussions too frequently, how the private sector also fails at projects on this scale, and frequently where their continued existence is at stake: Hershey's failure to implement SAP nearly bankrupted them by screwing up the Christmas shipping season that accounts for a majority of their sales.
posted by fatbird at 4:28 PM on December 27, 2013 [3 favorites]

They could have made an end run around these "bureaucratic systems" that people here seem to think are both completely unavoidable

...because agencies outside of the black world commonly get to ignore all the anticorruption legislation, all the Civil Rights Act provisions, all the other fair-play legislation, and other actual no-shit for real laws because they're inconvenient.

At the very very least they could have asked for it, brought public attention to the crappy job that the government does at IT and say, "WE NEED THIS in order to make this project a success."

Yes, because at any point after 3 January 2011, the Republican-controlled House would have been happy, HAPPY, to pass such legislation and help ensure the success of legislation they hate beyond rationality.
posted by ROU_Xenophobe at 4:59 PM on December 27, 2013 [4 favorites]

Cost per line of code is not a very good metric.
posted by Nothing at 5:14 PM on December 27, 2013 [1 favorite]

fatbird: "Seniority in itself isn't the problem. The unspoken premise here is that you shouldn't get seniority if you're incompetent because a well run office either trains up or fires the incompetent. If you have a lot of know-nothings with seniority, then obvious question to ask is why they didn't get fired or corrected along the way. And nothing about having seniority should protect incompetence right now, either."

Mainly, I just find it bizarre that there's a 10:1 ratio when the population at large is closer to 1:1. And I expect a larger percentage of younger people to be qualified than the boomer generation. It shouldn't take 10 people to keep one fauxhawk sporting, horn-rimmed PHP programmer in check!

Moreover, there shouldn't necessarily be a kneejerk reaction to open source libraries as 'insecure.' But now we're derailing a derail.

humanfont: "There is a working service now. It's been working since late November. ... If you want to piss and moan about an IT fuckup, look at Target. ...Their holiday sales took a huge hit and the liability might put them into bankruptcy. That was the IT fuckup of the year."

Indeed. "Delayed software is eventually good; rushed software is bad forever."
posted by pwnguin at 5:24 PM on December 27, 2013 [1 favorite]

This whole story is framed wrong. The thing we should be comparing is the previous, much more screwed up in many ways, process of signing up for private insurance, using a private insurance company's website.

After I quit Google and sort of needed insurance, I could not make the Aetna California website take and keep my info and provide an insurance quote. I spent 8 hours over at least a week trying, and I still feel kind of like a sucker for wasting that much time. I'm glad I didn't REALLY need insurance.

Fast forward a year, and it took me an hour and 10 minutes, start to finish, to sign up for insurance using healthcare.gov. It wasn't perfect, but it's a huge improvement over the previous system.

And yeah, yeah, government inefficiency, blah blah blah.
posted by Hello Dad, I'm in Jail at 5:41 PM on December 27, 2013 [6 favorites]

Is it really known that the system is now working?

The front-end works, but that is kind of the easy part. Last I heard the payment plan was to basically give the insurance cos. a large pile of money and sort it all out later. I expect a whole further raft of problems in the beginning of the new year as the insurance is expected to actually kick in and start being used. Some chance to jeopardize the law's current form, probably with the Republicans doing something to further Republicanize the plan while getting it semi-working and then getting the credit for "healthcare."

As an engineer and programmer, this is just a straight up embarrassment. It's basically as if the government needed to build an ObamaCare office building, and maybe it was a difficult office building 'cause it needed to be really big and LEED Platinum or something, but they ended up having a real hard time with the damn thing just collapsing on them. Programming is hard, but humanity has kind of figured out how to make e-commerce websites and CRM backends, even if this is a hard e-commerce website.

There was that whole thing about the docs and specs being locked-down and compartmentalized so the Republicans wouldn't find anything to make fun of. That's basically trying very hard to fail IT.

The PR and propaganda pieces defending it are ridiculous, because even the defenses are embarrassing.

Dec. 1: HealthCare.gov is now working more than 90 percent of the time — a big improvement over October, when the site was operating only about 43 percent of the time and frequently crashed, said Jeffrey Zients, the administration official overseeing the improvements.

The availability of a service is often described in "nines," plural. Four nines is 99.99%, etc. "One nine" is a phrase you might hear in a joke.

Dec. 19: And with respect to missing pages, they constitute "fewer than 1 percent of all of the transactions that we have done since Dec. 1," one administration official said.

When they say "fewer than 1 percent" assume it basically rounds to 1 percent. Two orders of magnitude removed from the 0.01% "thrown under the bus" number. Just the idea that applications were being sent out with "missing" data is kind of absurd. Hint: have the computer make sure all required data is present before sending. It can do this.

The lack of imagination is stunning. It was so much harder because it had to be a hard full launch? Here's a bright idea, have a soft launch. I read that some difficulty is due to legal/regulatory restrictions on the means other agencies can use to provide data. These laws & regulations seem to be immutable.

I think Obama might actually have done better if he had for serious had someone put up git.healthcare.gov and just asked Open Source to take care of it.

It's one of my beliefs that people & organizations should not make the accusations of their enemies true without reason. So when Republicans accuse government of incompetence at carrying out various projects, I don't think actually making the government incompetent is a good response.
posted by save alive nothing that breatheth at 6:12 PM on December 27, 2013

saulgoodman: Just skimmed the article but looking forward to reading in-depth later…

I think saulgoodman may come to regret the notion of "looking forward" once he's actually sat down and gone through this article: large-scale IT projects are very…messy. (Though it sounds like he knows that already. :7)
posted by wenestvedt at 6:59 PM on December 27, 2013

I've survived a couple (but only barely).
posted by saulgoodman at 7:26 PM on December 27, 2013 [1 favorite]

As an engineer and programmer, this is just a straight up embarrassment. It's basically as if the government needed to build an ObamaCare office building, and maybe it was a difficult office building 'cause it needed to be really big and LEED Platinum or something, but they ended up having a real hard time with the damn thing just collapsing on them.

Except that the building didn't totally collapse on them. A more accurate description would be that the government built a 500 million dollar office building. Construction finished on time and on budget, but the occupancy permit was delayed for 30 days because of issues noted on final inspection. Also some items were found and added to the punch list during the walkthrough. The contractor resolved all the major issues in 30 days and move in was completed successfully for tenants. If that actually happened on a real construction project everyone with experience involved would popping champaign corks and celebrating because it is never that easy.

Programming is hard, but humanity has kind of figured out how to make e-commerce websites and CRM backends, even if this is a hard e-commerce website.

Peoplesoft and Salesforce are well known for their flawless implementations which never run over budget and deliver on time. Also they are never down and scale perfectly -- SAID NO ONE EVER.
posted by humanfont at 7:49 PM on December 27, 2013 [15 favorites]

The marketing around IT has confused even many techies, I fear, about just how effortless and fail-proof enterprise software development is and can be expected to be. Improvements in software development practices and tools don't necessarily solve the more fundamental engineering problems (the need for so many, technologically heterogeneous interfaces and critical dependencies on external systems over which the primary system's owners have little or no control, the need to deliver intuitive user interactione while enforcing data integrity constraints and asynchronously fetching data from multiple web services using different protocols, etc.)

I do think the approach should have been iterative had that been politically possible (and I've even gotten nods of agreement on that point from colleagues), and I think the Federal government should have published an API spec and required external systems to conform to it. The economic stimulus effect would have been shared more broadly through that approach, too, as insurance providers and state agencies rushed out to hire more developers of their own to build systems against that API. But of course, that approach would take longer, create critical dependencies for success on less accountable external entities, and create more potential failure points for fraud and that pesky bugaboo, waste.

You can have a complete, out of the box system with virtually all its features modeled and understood completely and you can implement it using the waterfall model, but you'd better be prepared to spend a decade at the job if that's what it takes. You can't really know how complex a development effort is going to be until you've completely analyzed the problem. And that usually doesn't happen until long after the ink on the contracts has dried, and the project scope and duration have been defined.
posted by saulgoodman at 8:19 PM on December 27, 2013 [3 favorites]

(And yes, I am kind of disappointed by the article. It's too focused on CGIs share prices and its shady accounting on acquisitions. I wanted more specifics about the heathcare.gov site launch.)
posted by saulgoodman at 8:22 PM on December 27, 2013

There is a working service now. It's been working since late November. The grar it all out of proportion to reality here....If you want to piss and moan about an IT fuckup, look at Target. That was the IT fuckup of the year.

No. Read this this:

The Centers for Medicare and Medicaid Services, or CMS, which oversaw the building of HealthCare.gov, only performed two weeks of end-to-end testing before the site went live on Oct. 1, representatives from contractors CGI Federal and QSSI told the House Energy and Commerce Committee.

That’s significantly less testing time than major Web applications typically endure in government or the private sector, the contractors said.

The private sector fails at large software projects like healthcare.gov all the time, but skipping integration testing until a few weeks before go-live is still inexcusable. That's the testing that answers the question, "does this thing really work?" Did they mistake this joke for a development methodology? A 10% failure rate is a very lax definition of "working."

The Target data breach is also not comparable to the healthcare.gov issues (unless Target decided to use "god" as the root password to all their servers). Getting security right is hard, much harder than deciding to spend a reasonable amount of time integration testing your multimillion dollar website/giant integration project before go-live. One subtle mistake by one programmer is all it takes.
posted by cosmic.osmo at 8:34 PM on December 27, 2013 [2 favorites]

Well, Oracle, a Silicon Valley company whose bread and butter is large enterprise systems, totally messed up the exchange for just one medium sized state, Oregon. The system is still so messed up that they are doing 90% of their enrollments on paper forms.

So you could say the government did a far better job than Silicon Valley. Perhaps the job is just a bit less trivial than many seem to think.
posted by JackFlash at 8:50 PM on December 27, 2013 [3 favorites]

Doesn't Oracle screw up essentially every project? It's how they make money.
posted by five fresh fish at 8:53 PM on December 27, 2013 [3 favorites]

Doesn't Oracle screw up essentially every project? It's how they make money.

This. Oracle makes trash (besides their database). It seems like they have a very good sales organization, though.
posted by cosmic.osmo at 9:04 PM on December 27, 2013 [2 favorites]

Executives being grilled by a hostile congress are not going to present a particularly objective view of the facts, nor are they likely to accept any responsibility for issues. Blaming QA and/or the requirements are classic excuses.
Failure rates have continued to decline they are now below 0.38%, while traffic rose to a record of 83,000 simultaneous users on Dec 23.
Scaling a website to support millions of users and extremely high concurrency is not an easy challenge. Many of the issues simply don't manifest until real users start doing real things with your data.
posted by humanfont at 9:56 PM on December 27, 2013 [5 favorites]

philip-random: “One thing that quickly comes clear is that the guys who broke the sound barrier, made it to the moon, pulled off all manner of other wild and wonderful stuff were NOT in it for the money. ”

rhizome: “The military has its own idiosyncratic set of motivations. Perhaps you could say that the free market has destroyed the nation's sense of civic duty, but I don't think that's particularly germane here: it's already gone, and the system is not (or no longer) set up to encourage it.”

My partner and I only take government and non-profit, public good clients precisely because we feel that's the way we can best contribute to civilization as a whole.

feckless fecal fear mongering: “4) Profit. Retire to an island somewhere.”

Little guys don't get these jobs. Only outfits run by people who essentially already have "personal jet things" are even considered for the kinds of jobs that pay enough money to retire on.

humanfont: “Peoplesoft and Salesforce are well known for their flawless implementations which never run over budget and deliver on time. Also they are never down and scale perfectly -- SAID NO ONE EVER.”

If you'll open your hymnals to page 596 let's all join in a rousing chorus of "Oracle Financials — or Whatever They're Calling It These Days — Is a Steaming Pile of Unimplemental Crapplications."

I know a local outfit that tried to re-implement their back-end in SAP. Spent $30M and never got the system stood up.
posted by ob1quixote at 1:21 AM on December 28, 2013

Moreover, there shouldn't necessarily be a kneejerk reaction to open source libraries as 'insecure.'

I don't think it's kneejerk. Hell, I'm pretty sure the DOD / CIA / NSA use open source libraries all of the time (like in those nifty clandestine malware creations that steal documents and send them off using SSL) , but there's an exhaustive vetting process for every update to the extent that you might as well just fork the project when it does what you need it to do and maintain it from then on yourself.

I suspect you're addressing a previous comment that didn't explicitly say
"Open Source is Insecure!" but did say that a bunch of "bullshit PHP" is "not gonna fly." To me that seems fair enough; I don't really want it written in PHP and there might be a kneejerk reaction to PHP in general that is unfair, but it's kludge piled on top of kludge that makes it way too easy to be horrendously sloppy in the WebDev world full of gotchas with relatively low stakes on the table if it comes crashing down...not exactly the easiest thing to fully vet and understand every time there's a major change.

I think Obama might actually have done better if he had for serious had someone put up git.healthcare.gov and just asked Open Source to take care of it.

Not to be snarky at all, this just struck me as strange. Can you name some other services that are built and deployed by a monolithic-like "Open Source" community (not the brainchild of a person with money and open source devs) that would lend any confidence to the possibility of the open source community just propping up the most bureaucratically complex public facing website of all time? Because that's what this is.

You talk about CRMs and ERPs being implemented elsewhere...well yeah. But I think this is the most complex customer-facing web-based thing, ever. Honestly. It's unprecedented. Expectations were too high, it was rolled out too abruptly, and it would've probably been better for the government to just create a new in-house bad-ass healthcare IT development department stacked with people who are excited to solve these problems, along with new NIST, HL7, whatever standards that address any shortcomings in existing interfaces. Knowing how sloppy all software is and particularly healthcare-related stuff, I wouldn't be surprised if many of these interfaces were just screen-scraping systems with hard-coded passwords in the back-end.

Sometimes a massive project succeeds, sometimes it fails, and it's shocking for me to read about companies dropping $30M on something and just not making it work. Deciding at $30M to cut their losses. What if they just need $200K of some hot-shot's time? But maybe not.

I'm a person in the role of "make it fucking work no matter what" and have built a decent career for myself being the guy who is aghast at stereotypically detached IT people (and occasionally devs, though their issues are difference) throwing up their arms at hard issues, handwaving them away as impossible or "not ready for prime time," and just likes to walk right past them and make shit happen and make them look like assholes for basically being unwilling to do the right thing. Healthcare IT and development is hard and a lot of typical schmucks don't really want to understand any of the healthcare aspect, and think they can get away with that mentality. They might be right most of the time, but they need to be a stone's through away from an SME at all times when making decisions that otherwise might take 30 seconds but causes weeks worth of headaches down the road.

It's not as hard as building 3D gaming engines or whatever, but hard in a different, certainly bureaucratic way that totally and seductively tempts developers to be sloppy and hard-code rigid systems far too often, with lots of hard decisions about how to abstract apart an absurdly complex rules-that-are-constantly-broken-based system.

I think the government should've created an in-house dev group full of the right people with the right stuff. To be persistent, to have "The Right Stuff" where you just beat the shit out of yourself really trying to solve a problem because you want to. And someone is in charge of making sure you don't kill yourself in the process and have a sense of perspective and are participating in a team effort when necessary, not acting as a rogue hotshot who "hates meetings" (key point: don't have too many fucking meetings), and going into crazy fugue you're-on-your-own mode when necessary. I have some stories where I had to reverse-engineer an EHR and hack it to pieces and my thought process at the time was "nobody in this company would have thought to do this or that it was possible or be able to do it." But the company wasn't paying anyone to do it either, they just knew that if I said I could do something, I probably could do it right now or figure it out with enough time. But when it's a $30M failure I do want to hear more...why? Do you just need to pay someone who gives a fuck and knows they have the skills to push through the problem even if they have no idea what the problem is yet or what specific skills will be required? Why did it take you $30M to find yourself here?
posted by lordaych at 5:00 AM on December 28, 2013 [9 favorites]

Earlier this month, I got an email from the insurance broker who sold me my health insurance saying that Blue Cross Blue Shield of Tennessee was cancelling my policy.

Thanks, Obamacare!

Except that the problem wasn't Obamacare. BCBST said I hadn't paid my premium in November. Except that I did pay my premium in November. I paid it online through the BCBST website. Or at least, I tried to. See, to pay your premium on the BCBST website, you can't just use a debit card. You have to enter your checking account and routing numbers every month. And I did that on the first day of November and got a confirmation number which I duly filed on my hard drive in the same folder as all of the rest of the confirmation documents I have gotten over the four years I have had this policy. But for some reason, BCBST never took the money out of my account, and the first I heard of there being a problem was the impending cancellation, which they didn't even tell me about, choosing instead to tell my insurance broker with whom I had no contact for at least a year, maybe longer. (And yes, there was more than enough money in my account to pay the premium that day in November when I attempted to pay and got my confirmation number.)

So I called BCBST and waited 30 minutes on hold until I could talk to a human being about this mixup. But once I got someone on the line, the connection was so bad that we ended up yelling numbers at each other trying to set up the transaction for what was at this point two months worth of premiums. And before we were able to complete the transaction, the call was terminated. So I called back, waited on hold for another 30 minutes or so, and talked to a second operator. The same thing happened—the connection was so bad that I was cut off before the transaction was complete. So I called back a third time, waited on hold for another 30 minutes, and again got a bad connection with an operator who was just as clueless and hard to understand as the other two. But the third time was the charm, I guess, and I was able, after several attempts, to make myself understood through the voice dropouts well enough so that the customer service representative was able to enter my checking account and routing number and give me yet another confirmation number.

Afterwards, I wondered if it was my phone or the BCBST's system that was scrambling calls so badly, so I called my mother, who lives in a small town about 350 miles from where I am and only about 70 miles from Chattanooga, where the BCBST offices are located. The connection was clear as a bell, so I surmised that it was the BCBST phone system that was messing up.

This, by the way, was the third time in a year I have had trouble paying through the BCBST website.

So the moral of my story is, apparently nobody can make a health insurance website or information system worth a damn, be they government or private insurance company, and if healthcare.gov had to try to interface with not only BCBST but dozens of other private health insurance companies from all over the nation whose IT infrastructres are in various states of fucked up, then it's no wonder that this entire enterprise has been such a clusterfuck. Single payer now!
posted by vibrotronica at 4:51 PM on December 28, 2013 [3 favorites]

I think BCBS especially might be a hot mess. In NY they gave different information about their exchange plans than on their site, like with real impact on costs. I tried to chat to their customer support about it. Everyone was super nice but couldn't tell me a thing and didn't seem to know a thing, including about relatively staple stuff like what is an HSA and why would anyone want them.
posted by Salamandrous at 5:41 PM on December 28, 2013 [1 favorite]

I have conservative friends and relatives that have said things to the effect of "If Amazon (or insert other major retailer here) launched a website that didn't work, everyone would be fired."

So I work for a major US and Canadian department store chain that re-launched their online store in September of last year, and it's still not working at 100%. The first 3-4 months were a complete and utter disaster, and every time they activate one of the new features, there are something in the neighborhood of 17,000 customer confusion/complaint emails.

Not one person has been fired over it.

The build of the site included both major tech companies (IBM), and an internal IT team. It has been painful every step of the way, and we're not re-inventing the wheel we're doing something that hundreds of other sites do.

I try to explain the complexity of multi-carrier, multi-state policies, and tell them life would be SOOOOOO much easier and if we had a single-payer healthcare system. It's just pushing a boulder up a hill.
posted by elvissa at 6:49 PM on December 28, 2013 [6 favorites]

« Older A Gift To You | A song for the formerly hearing impaired Newer »

This thread has been archived and is closed to new comments

MetaFilter

Inside the Company That Built Healthcare.gov
December 27, 2013 11:33 AM Subscribe

Tags

Share

Inside the Company That Built Healthcare.gov December 27, 2013 11:33 AM Subscribe

Tags

Share

Inside the Company That Built Healthcare.gov
December 27, 2013 11:33 AM Subscribe