Sinkhole of Bureaucracy
March 23, 2014 8:45 AM Subscribe
The US Office of Personnel Management's Retirement Operations Center is housed about 230 feet below the surface inside the caverns of an old limestone mine. The trucks full of paperwork come every day, turning off a country road north of Pittsburgh and descending through a gateway into the earth. Underground, they stop at a metal door decorated with an American flag.
Behind the door, a room opens up as big as a supermarket, full of five-drawer file cabinets and people in business casual. Here, inside the caverns of an old Pennsylvania limestone mine, there are 600 employees of the Office of Personnel Management. Their task is nothing top-secret. It is to process the retirement papers of the government’s own workers.
But that system has a spectacular flaw. It still must be done entirely by hand, and almost entirely on paper.
First in series examining the failures at the treat of troubled federal systems.
Behind the door, a room opens up as big as a supermarket, full of five-drawer file cabinets and people in business casual. Here, inside the caverns of an old Pennsylvania limestone mine, there are 600 employees of the Office of Personnel Management. Their task is nothing top-secret. It is to process the retirement papers of the government’s own workers.
But that system has a spectacular flaw. It still must be done entirely by hand, and almost entirely on paper.
First in series examining the failures at the treat of troubled federal systems.
Can't wait until Mulder and Scully find this place!
posted by valkane at 9:10 AM on March 23, 2014 [1 favorite]
posted by valkane at 9:10 AM on March 23, 2014 [1 favorite]
For just a moment when I was reading the FPP I thought the door was an American Flag with a [more inside].
posted by srboisvert at 9:11 AM on March 23, 2014 [1 favorite]
posted by srboisvert at 9:11 AM on March 23, 2014 [1 favorite]
The Ex- Files
posted by Gyan at 9:12 AM on March 23, 2014 [7 favorites]
posted by Gyan at 9:12 AM on March 23, 2014 [7 favorites]
“I used to chase people for months — literally — for one signature on one piece of paper. You want to talk about an egregious waste of taxpayer money?” recalled one worker who left the mine recently and declined to be named because of fears of retribution.It's not all that shocking that when you reduce staffing levels the remaining staff are slow to respond to requests outside the core objectives of the agency.
posted by Mitheral at 9:14 AM on March 23, 2014 [3 favorites]
I really wish they'd had spent more time talking about the absurdly complex laws that Congress creates and refuses to fix that effectively REQUIRE all this stupidity. Seems like ignoring the biggest problem. I could tell horror stories about absurdly specific parts of laws that basically required things be done manually.
posted by petrilli at 9:15 AM on March 23, 2014 [10 favorites]
posted by petrilli at 9:15 AM on March 23, 2014 [10 favorites]
I really wish they'd had spent more time talking about the absurdly complex laws that Congress creates and refuses to fix that effectively REQUIRE all this stupidity. Seems like ignoring the biggest problem.
Yes, but then no one could blame it all on lazy union workers.
posted by Thorzdad at 9:17 AM on March 23, 2014 [14 favorites]
Yes, but then no one could blame it all on lazy union workers.
posted by Thorzdad at 9:17 AM on March 23, 2014 [14 favorites]
Additional footage of a government worker entering the facility.
posted by kewb at 9:18 AM on March 23, 2014 [5 favorites]
posted by kewb at 9:18 AM on March 23, 2014 [5 favorites]
During the past 30 years, administrations have spent more than $100 million trying to automate the old-fashioned process in the mine and make it run at the speed of computers.
Every single one of the IT contractors hired should be sued and forced to return the money. Yeah OK, not possible and not a good idea for a lot of reasons. But the amount of incompetence and fraud in the world of US government contracting for IT is just astounding.
I've heard a few war stories over beers from folks working on fixing healthcare.gov. Nothing too specific (nor quotable), but the general picture of what was going on the past few years building heathcare.gov is a total fucking disaster of incompetence and stupidity. Nothing specific to the Obama administration, every previous presidential administration is just as bad if not worse.
Traditional government IT and procurements works in this entirely different world of wastage and multiyear development that turns out broken systems. I like to make fun of Silicon Valley software engineering practice as much as anyone, but it turns out there are software engineering cultures that do it a whole lot worse. Well not exactly "worse", in that those giant dinosaur companies charge $100M for a system that doesn't even function on delivery. In some twisted sense that's better.
posted by Nelson at 9:24 AM on March 23, 2014 [8 favorites]
Every single one of the IT contractors hired should be sued and forced to return the money. Yeah OK, not possible and not a good idea for a lot of reasons. But the amount of incompetence and fraud in the world of US government contracting for IT is just astounding.
I've heard a few war stories over beers from folks working on fixing healthcare.gov. Nothing too specific (nor quotable), but the general picture of what was going on the past few years building heathcare.gov is a total fucking disaster of incompetence and stupidity. Nothing specific to the Obama administration, every previous presidential administration is just as bad if not worse.
Traditional government IT and procurements works in this entirely different world of wastage and multiyear development that turns out broken systems. I like to make fun of Silicon Valley software engineering practice as much as anyone, but it turns out there are software engineering cultures that do it a whole lot worse. Well not exactly "worse", in that those giant dinosaur companies charge $100M for a system that doesn't even function on delivery. In some twisted sense that's better.
posted by Nelson at 9:24 AM on March 23, 2014 [8 favorites]
I had no idea anyone mined limestone underground. I always assumed it was done as they do in southern Indiana, in open-air quarries.
posted by Thorzdad at 9:25 AM on March 23, 2014 [1 favorite]
posted by Thorzdad at 9:25 AM on March 23, 2014 [1 favorite]
posted by Measured Out my Life in Coffeespoons
This is a great post, but it feels a little eponysterical. :-)
Back when the A in the GAO stood for "Accounting" rather than "Accountabilty" they built a massive squat office building to house all the paperwork, because the sheer weight of it all presented a structural challenge. Now they use computers, because duh, so after 9/11 it required very little upgrading to prepare for terrorism.
posted by anotherpanacea at 9:26 AM on March 23, 2014
This is a great post, but it feels a little eponysterical. :-)
Back when the A in the GAO stood for "Accounting" rather than "Accountabilty" they built a massive squat office building to house all the paperwork, because the sheer weight of it all presented a structural challenge. Now they use computers, because duh, so after 9/11 it required very little upgrading to prepare for terrorism.
posted by anotherpanacea at 9:26 AM on March 23, 2014
The dollar amounts they’re talking about don’t seem crazy: total spending $55.8 million, $25 million in 1987 on a failed upgrade, $106 million on another in 2008. The article has a LOLBureaucracy tone to it, but this seems like an obvious result of complex rules and the natural scale of a retirement system with 100,000 new entrants each year. If anything, I’m most bothered by the attempted IT upgrades that Nelson talks about, because they are often opportunities for one-time money burns. The normal costs, ~$108 per person, seem reasonable.
posted by migurski at 9:28 AM on March 23, 2014 [8 favorites]
posted by migurski at 9:28 AM on March 23, 2014 [8 favorites]
The dollar amounts they’re talking about don’t seem crazy...The normal costs, ~$108 per person, seem reasonable.
I agree. The average person on the street has no fathomable clue just how vast the work of the government really is, and how much it costs to do it all, even at skeletal budgeting. Because of this, it's always been dirt-simple to evoke outrage and anger among the masses by quoting these numbers without any frame of reference or comparison.
posted by Thorzdad at 9:39 AM on March 23, 2014 [11 favorites]
I agree. The average person on the street has no fathomable clue just how vast the work of the government really is, and how much it costs to do it all, even at skeletal budgeting. Because of this, it's always been dirt-simple to evoke outrage and anger among the masses by quoting these numbers without any frame of reference or comparison.
posted by Thorzdad at 9:39 AM on March 23, 2014 [11 favorites]
Their task is nothing top-secret. It is to process the retirement papers of the government’s own workers.
Can't wait until Mulder and Scully find this place!
They already did. I'm pretty sure that this is the place that stores vials of everyone's blood for nefarious alien purposes.
posted by ActingTheGoat at 9:39 AM on March 23, 2014 [1 favorite]
Can't wait until Mulder and Scully find this place!
They already did. I'm pretty sure that this is the place that stores vials of everyone's blood for nefarious alien purposes.
posted by ActingTheGoat at 9:39 AM on March 23, 2014 [1 favorite]
Yeah, the fact that you could write off $25 million-$100 million for a failed project as "not crazy" is itself absolutely insane. You could pay salary and benefits for 500 highly skilled Google/Apple/Facebook engineers for a year with that money, and I guarantee they would have delivered a system that worked.
Government IT is so deeply fucked in so many ways it's astounding.
posted by Itaxpica at 9:40 AM on March 23, 2014 [1 favorite]
Government IT is so deeply fucked in so many ways it's astounding.
posted by Itaxpica at 9:40 AM on March 23, 2014 [1 favorite]
To clarify, when I say highly skilled I don't mean that SV engineers are inherently highly skilled - I mean you can pay for senior-level engineers at that rate, not just interns/fresh out of college newbies.
posted by Itaxpica at 9:44 AM on March 23, 2014
posted by Itaxpica at 9:44 AM on March 23, 2014
Isn't it a problem that the final $106M system didn't work and was scrapped? Software systems shouldn't be an all-or-nothing roll of the dice, "well we gambled $106M and it didn't work this time. Oops."
Retirement processing is complicated but it's not that complicated. The WashPo's characterization of the complexity was "the new system would have to be capable of making up to 150 distinct calculations". Wow, 150? That's more than I can count with all my fingers and toes! And yet on delivery, "tests showed that the system could handle only five of the next 61 functions". How do you spend $108M building a system that then doesn't even do the basic, well defined function required of it?
The IT contractor responsible, Hewitt Associates, didn't seem to suffer too badly by their failure. They got bought out by AON in 2010, for nearly $5B. And now Aon Hewitt partners with leading employers and federal agencies worldwide to turn strategies into success. Curious definition of success.
posted by Nelson at 9:48 AM on March 23, 2014 [5 favorites]
Retirement processing is complicated but it's not that complicated. The WashPo's characterization of the complexity was "the new system would have to be capable of making up to 150 distinct calculations". Wow, 150? That's more than I can count with all my fingers and toes! And yet on delivery, "tests showed that the system could handle only five of the next 61 functions". How do you spend $108M building a system that then doesn't even do the basic, well defined function required of it?
The IT contractor responsible, Hewitt Associates, didn't seem to suffer too badly by their failure. They got bought out by AON in 2010, for nearly $5B. And now Aon Hewitt partners with leading employers and federal agencies worldwide to turn strategies into success. Curious definition of success.
posted by Nelson at 9:48 AM on March 23, 2014 [5 favorites]
and I guarantee they would have delivered a system that worked.
Well, if a random person the internet says it can be done, ok!
posted by Brandon Blatcher at 9:49 AM on March 23, 2014 [7 favorites]
Well, if a random person the internet says it can be done, ok!
posted by Brandon Blatcher at 9:49 AM on March 23, 2014 [7 favorites]
I want to work here so bad. Plastic ferns and strains of "Wichita Lineman" under the flurorescents. Probable secret vaults of expired crackers and bales of $2 bills for use after a Soviet attack.
Sane. Orderly. Safe.
posted by codswallop at 9:52 AM on March 23, 2014 [11 favorites]
Sane. Orderly. Safe.
posted by codswallop at 9:52 AM on March 23, 2014 [11 favorites]
Yeah, the fact that you could write off $25 million-$100 million for a failed project as "not crazy" is itself absolutely insane. You could pay salary and benefits for 500 highly skilled Google/Apple/Facebook engineers for a year with that money, and I guarantee they would have delivered a system that worked.
LOL. Maybe as long as you don't have to pay for a building, heat, electricity, cleaning, cafeteria, HR, management, lawyers, security, accounting, sales, lobbying, insurance and on and on.
Private contracting is running an entire business. It isn't a 100 engineer/programmer jamboree.
posted by srboisvert at 9:54 AM on March 23, 2014 [10 favorites]
LOL. Maybe as long as you don't have to pay for a building, heat, electricity, cleaning, cafeteria, HR, management, lawyers, security, accounting, sales, lobbying, insurance and on and on.
Private contracting is running an entire business. It isn't a 100 engineer/programmer jamboree.
posted by srboisvert at 9:54 AM on March 23, 2014 [10 favorites]
It’s certainly a problem when the systems are scrapped. I’m contrasting the normal operating costs of dealing with humans in one way for 40 years to the periodic digitization fever dreams of government IT (and WashPo readers). I imagine that retirement documents processing has got to be 80% exception handling, so it doesn’t shock me that these enormous money-wasters crash on the rocks.
posted by migurski at 9:56 AM on March 23, 2014 [1 favorite]
posted by migurski at 9:56 AM on March 23, 2014 [1 favorite]
Sane. Orderly. Safe.
Yeah, until Joey from HR comes in and clutch ovens the place.
posted by valkane at 9:58 AM on March 23, 2014
Yeah, until Joey from HR comes in and clutch ovens the place.
posted by valkane at 9:58 AM on March 23, 2014
The trouble isn't the concept of moving the process to IT, but rather that they're still doing the planning and procurement by the standards of the 1960s, which we've known doesn't work since the 1970s. That's why we've developed the various Agile methods, Extreme Programming, the Spiral Model, the Rational Unified Process, and so on. It seems to be just the government, the IEEE, and the PMI that still cling to the idea that you can design a system and somehow capture every requirement before a line of code has ever been written.
posted by sonic meat machine at 10:02 AM on March 23, 2014 [12 favorites]
posted by sonic meat machine at 10:02 AM on March 23, 2014 [12 favorites]
Sonic meat machine, that's about the nut of it. Dan Milstein is a great read on this topic.
posted by migurski at 10:07 AM on March 23, 2014 [1 favorite]
posted by migurski at 10:07 AM on March 23, 2014 [1 favorite]
Retirement processing is complicated but it's not that complicated. The WashPo's characterization of the complexity was "the new system would have to be capable of making up to 150 distinct calculations". Wow, 150? That's more than I can count with all my fingers and toes!
But the number is surely vastly more than that in the usual combinatorial explosion. The example they gave is that the formula for a law enforcement officer is different from that for a postal worker... but then so is the formula for someone who had been a law enforcement officer and then joined the postal service. And people in those situations who'd always been covered by FERS will probably be different from people in those situations who are under the older CSRS or who moved from CSRS to FERS.
And to make matters worse, many of these combinations probably can't be dealt with by simple algorithms like averaging by years of service or similar. Instead, many combinations were probably dealt with by direct legislation so there's little to do except have a vast lookup table -- if the retiree was a LEO and then postal worker, do this, but if the retiree was a postal worker and then a LEO. And there's probably nothing but a precedent somewhere about how to deal with someone who'd been a postal worker, worked as an LEO for five years, and then went back to being a postal worker, or someone who'd been a LEO then postal worker then worked for Interior.
posted by ROU_Xenophobe at 10:09 AM on March 23, 2014 [6 favorites]
But the number is surely vastly more than that in the usual combinatorial explosion. The example they gave is that the formula for a law enforcement officer is different from that for a postal worker... but then so is the formula for someone who had been a law enforcement officer and then joined the postal service. And people in those situations who'd always been covered by FERS will probably be different from people in those situations who are under the older CSRS or who moved from CSRS to FERS.
And to make matters worse, many of these combinations probably can't be dealt with by simple algorithms like averaging by years of service or similar. Instead, many combinations were probably dealt with by direct legislation so there's little to do except have a vast lookup table -- if the retiree was a LEO and then postal worker, do this, but if the retiree was a postal worker and then a LEO. And there's probably nothing but a precedent somewhere about how to deal with someone who'd been a postal worker, worked as an LEO for five years, and then went back to being a postal worker, or someone who'd been a LEO then postal worker then worked for Interior.
posted by ROU_Xenophobe at 10:09 AM on March 23, 2014 [6 favorites]
Bwithh in the bit I quoted. The retirement guys end up waiting for the retiree's former agencies to sign off on paper work and most of those agencies have been the victims of cost cutting measures. Let alone really stupid shit like sequester and government shut down.
posted by Mitheral at 10:12 AM on March 23, 2014
posted by Mitheral at 10:12 AM on March 23, 2014
> Yeah, the fact that you could write off $25 million-$100 million for a failed project as "not crazy" is itself absolutely insane. You could pay salary and benefits for 500 highly skilled Google/Apple/Facebook engineers for a year with that money, and I guarantee they would have delivered a system that worked.
Well, I worked as a Google engineer, and they put huge numbers of engineers into vast projects that never worked really well, limped along for up to a decade, and were eventually put out of their misery - so they aren't perfect my any means.
Let's also add to that that strong engineers have their pick of jobs, so the people working on the government retirement fix-up are going to be second-tier at best. And software development is a young field - many of the original pioneers are still alive, or have recently died - and there still isn't an established methodology like there is in, say, construction, where the pioneers have been dust for thousands of years.
As I tell people in lectures, if you look at software projects started by Fortune 1000 companies, over half of them never complete; over half of the remainder are completed but not actually adopted (probably because they're terrible); and over half the rest aren't "successful" (i.e. don't justify their initial investment). (These numbers are approximate and taken from various sources but are a realistic evaluation of actual results...)
I don't think these numbers for the government projects are good at all - this stuff isn't that hard, you should be able to complete it if you are disciplined even if you aren't Joe Super Programmer - but I don't think they're outrageous or insane numbers either.
(That said, from my limited experience in this matter, if you had an auditor going through the expenditures on that $100 million, there would be huge numbers of give-backs and perhaps criminal charges. People really take advantage of the government in these things...)
posted by lupus_yonderboy at 10:14 AM on March 23, 2014 [18 favorites]
Well, I worked as a Google engineer, and they put huge numbers of engineers into vast projects that never worked really well, limped along for up to a decade, and were eventually put out of their misery - so they aren't perfect my any means.
Let's also add to that that strong engineers have their pick of jobs, so the people working on the government retirement fix-up are going to be second-tier at best. And software development is a young field - many of the original pioneers are still alive, or have recently died - and there still isn't an established methodology like there is in, say, construction, where the pioneers have been dust for thousands of years.
As I tell people in lectures, if you look at software projects started by Fortune 1000 companies, over half of them never complete; over half of the remainder are completed but not actually adopted (probably because they're terrible); and over half the rest aren't "successful" (i.e. don't justify their initial investment). (These numbers are approximate and taken from various sources but are a realistic evaluation of actual results...)
I don't think these numbers for the government projects are good at all - this stuff isn't that hard, you should be able to complete it if you are disciplined even if you aren't Joe Super Programmer - but I don't think they're outrageous or insane numbers either.
(That said, from my limited experience in this matter, if you had an auditor going through the expenditures on that $100 million, there would be huge numbers of give-backs and perhaps criminal charges. People really take advantage of the government in these things...)
posted by lupus_yonderboy at 10:14 AM on March 23, 2014 [18 favorites]
> Instead, many combinations were probably dealt with by direct legislation so there's little to do except have a vast lookup table
And the problem is... what?
Vast lookup tables are actually a very practical way to do things. Maintenance is easy - look up the spot in the table and edit it. When a 1TB drive costs $100, it's extremely unlikely that the lookup table won't fit on one machine - heck, when 16 gigs of memory cost $200, it's likely that you can fit it all into RAM. Hash tables are well-understood and there are many fast, efficient, free implementations of them out there.
I'd honestly be shocked if they did anything so sensible. My theory is that all that logic is hard-coded into the program.
The big difficulty for mediocre programmers doing this is - what exactly do you store as the values in the table? When it comes down to it, it would have to be a rule. How do you represent a rule? When it comes down to it, it's a function - a piece of code! But I guarantee you that they won't be storing code in the database (it's a bad idea at any level of programming unless it's completely necessary and you have rigid rules around it). So you'd have to write a rule-representation language - an extensible, flexible one.
This is the sort of thing I love to do but, well, it's probably not going to happen for the government. Instead, lots of hard-coded logic...
posted by lupus_yonderboy at 10:22 AM on March 23, 2014 [3 favorites]
And the problem is... what?
Vast lookup tables are actually a very practical way to do things. Maintenance is easy - look up the spot in the table and edit it. When a 1TB drive costs $100, it's extremely unlikely that the lookup table won't fit on one machine - heck, when 16 gigs of memory cost $200, it's likely that you can fit it all into RAM. Hash tables are well-understood and there are many fast, efficient, free implementations of them out there.
I'd honestly be shocked if they did anything so sensible. My theory is that all that logic is hard-coded into the program.
The big difficulty for mediocre programmers doing this is - what exactly do you store as the values in the table? When it comes down to it, it would have to be a rule. How do you represent a rule? When it comes down to it, it's a function - a piece of code! But I guarantee you that they won't be storing code in the database (it's a bad idea at any level of programming unless it's completely necessary and you have rigid rules around it). So you'd have to write a rule-representation language - an extensible, flexible one.
This is the sort of thing I love to do but, well, it's probably not going to happen for the government. Instead, lots of hard-coded logic...
posted by lupus_yonderboy at 10:22 AM on March 23, 2014 [3 favorites]
Sonic meat machine, that's about the nut of it. Dan Milstein is a great read on this topic.
Thanks for that article; I hadn't read it. It jibes with my experience, and also enumerates some of the reasons I'm very big on short sprints. Getting management (and less, uh, motivated1) programmers to agree to short sprints has always been a struggle, but it really does force you to make every effort to break down a big task into smaller tasks that you can reliably estimate. Oddly, I also find that the small sprints force me to do more planning, because I am thinking about steps a couple of sprints out as opposed to making a single Grand Diagram and never revisiting it. Sometimes the tasks surprise me and accordion out into several sprints of abject pain ("But you said you'd have it by sprint 17!" "That's why it's an estimate."), but I've come to view that as paying my bills ahead of time so that they don't all come due at once.
1 Hilariously, I had a "colleague" who argued vehemently against 1-week sprints because "we'd have to be delivering features to QA by the second or third day of the sprint!"—yeah, man, that's the point. He now works for the government.
posted by sonic meat machine at 10:26 AM on March 23, 2014
Thanks for that article; I hadn't read it. It jibes with my experience, and also enumerates some of the reasons I'm very big on short sprints. Getting management (and less, uh, motivated1) programmers to agree to short sprints has always been a struggle, but it really does force you to make every effort to break down a big task into smaller tasks that you can reliably estimate. Oddly, I also find that the small sprints force me to do more planning, because I am thinking about steps a couple of sprints out as opposed to making a single Grand Diagram and never revisiting it. Sometimes the tasks surprise me and accordion out into several sprints of abject pain ("But you said you'd have it by sprint 17!" "That's why it's an estimate."), but I've come to view that as paying my bills ahead of time so that they don't all come due at once.
1 Hilariously, I had a "colleague" who argued vehemently against 1-week sprints because "we'd have to be delivering features to QA by the second or third day of the sprint!"—yeah, man, that's the point. He now works for the government.
posted by sonic meat machine at 10:26 AM on March 23, 2014
This is the sort of thing I love to do but, well, it's probably not going to happen for the government. Instead, lots of hard-coded logic...
It's the type of thing that functional languages are perfect for, but those break peoples' brains, so you get people writing fifty thousand lines of C++ instead.
posted by sonic meat machine at 10:28 AM on March 23, 2014 [2 favorites]
It's the type of thing that functional languages are perfect for, but those break peoples' brains, so you get people writing fifty thousand lines of C++ instead.
posted by sonic meat machine at 10:28 AM on March 23, 2014 [2 favorites]
As stated above: modern software development practices would work so much better here. Get a product team working onsite, developing specs and software in concert with the people already doing the work. Live with the old timers who know the quirks of the system. Run mockups and prototypes by people constantly.
Ultimately if the government itself can start thinking of itself as a tech company they will reap enormous rewards, as folks in so many other industries are discovering.
posted by wemayfreeze at 10:29 AM on March 23, 2014 [2 favorites]
Ultimately if the government itself can start thinking of itself as a tech company they will reap enormous rewards, as folks in so many other industries are discovering.
posted by wemayfreeze at 10:29 AM on March 23, 2014 [2 favorites]
I want to work here so bad. Plastic ferns and strains of "Wichita Lineman" under the flurorescents. Probable secret vaults of expired crackers and bales of $2 bills for use after a Soviet attack.
Springfield Underground has fat stacks of cheddar. There's also SubTropolis, the Marengo Warehouse, and many others.
posted by peeedro at 10:32 AM on March 23, 2014
Springfield Underground has fat stacks of cheddar. There's also SubTropolis, the Marengo Warehouse, and many others.
posted by peeedro at 10:32 AM on March 23, 2014
> I had a "colleague" who argued vehemently against 1-week sprints because "we'd have to be delivering features to QA by the second or third day of the sprint!"
He was right, though - one week is usually too short for a sprint. At one week per iteration, the mechanism of the sprint becomes a significant fraction of your engineer's work time, which is bad. And if any of your engineers is out, it becomes a sprint to the death.
Your poor QA department, having to run a complete set of tests every week, would be frazzled to the bone!
Sure, there are some times when it might be good - perhaps really early on before QA even gets involved ("spike development") or really late in the process, a series of quick bugfix sprints without full testing - but those are the exceptions, not the rules.
posted by lupus_yonderboy at 10:35 AM on March 23, 2014 [1 favorite]
He was right, though - one week is usually too short for a sprint. At one week per iteration, the mechanism of the sprint becomes a significant fraction of your engineer's work time, which is bad. And if any of your engineers is out, it becomes a sprint to the death.
Your poor QA department, having to run a complete set of tests every week, would be frazzled to the bone!
Sure, there are some times when it might be good - perhaps really early on before QA even gets involved ("spike development") or really late in the process, a series of quick bugfix sprints without full testing - but those are the exceptions, not the rules.
posted by lupus_yonderboy at 10:35 AM on March 23, 2014 [1 favorite]
I want to work here so bad. Plastic ferns and strains of "Wichita Lineman" under the flurorescents. Probable secret vaults of expired crackers and bales of $2 bills for use after a Soviet attack.
Great, now I have Wichita Lineman stuck in my head. Actually, now I have it stuck in my YouTube. And the Wichita lineman is still on the liiine!
posted by ActingTheGoat at 10:39 AM on March 23, 2014
Great, now I have Wichita Lineman stuck in my head. Actually, now I have it stuck in my YouTube. And the Wichita lineman is still on the liiine!
posted by ActingTheGoat at 10:39 AM on March 23, 2014
He was right, though - one week is usually too short for a sprint. At one week per iteration, the mechanism of the sprint becomes a significant fraction of your engineer's work time, which is bad. And if any of your engineers is out, it becomes a sprint to the death.
Planning such short sprints usually involved a half-day at that organization; an hour of product owner demo, an hour of retrospective, an hour of grooming stories, and an hour of "sprint planning." This is in contrast to the nearly week-long process usually used for the month-long sprint that was organizational standard. (Talk about brain-numbing.)
Granted, the project that I was working on with the most success with this cycle was a greenfield project, and so our legacy testing consisted of some interfacing with other systems within the organization, but we didn't run into many QA limitations.
posted by sonic meat machine at 10:42 AM on March 23, 2014
Planning such short sprints usually involved a half-day at that organization; an hour of product owner demo, an hour of retrospective, an hour of grooming stories, and an hour of "sprint planning." This is in contrast to the nearly week-long process usually used for the month-long sprint that was organizational standard. (Talk about brain-numbing.)
Granted, the project that I was working on with the most success with this cycle was a greenfield project, and so our legacy testing consisted of some interfacing with other systems within the organization, but we didn't run into many QA limitations.
posted by sonic meat machine at 10:42 AM on March 23, 2014
> Granted, the project that I was working on with the most success with this cycle was a greenfield project
Sure, as I said that's a good idea. Heck, I use a sort of agile methodology for some fairly large personal projects, and I have some sprints as short as one (very long) day - but me being all the parts of the system has a huge advantage.
The sad truth is that most programming is maintenance programming of some sort or another, and this demands significantly longer sprints. What's worked well in the past is a two-week cycle with significant testing each cycle, and a three or four month larger cycle with a "release" and significant regression testing.
posted by lupus_yonderboy at 10:46 AM on March 23, 2014
Sure, as I said that's a good idea. Heck, I use a sort of agile methodology for some fairly large personal projects, and I have some sprints as short as one (very long) day - but me being all the parts of the system has a huge advantage.
The sad truth is that most programming is maintenance programming of some sort or another, and this demands significantly longer sprints. What's worked well in the past is a two-week cycle with significant testing each cycle, and a three or four month larger cycle with a "release" and significant regression testing.
posted by lupus_yonderboy at 10:46 AM on March 23, 2014
And the problem is... what?
It means you have to have a definitive answer for something close to every possible combination of federal employers, lengths of service in each, years of service, pension schemes, and probably more. Coding that might not, as you note, be particularly difficult. Getting all the answers is going to mean lots of man-hours from the contractor, and lots of man-hours for OPM employees.
posted by ROU_Xenophobe at 10:48 AM on March 23, 2014 [1 favorite]
It means you have to have a definitive answer for something close to every possible combination of federal employers, lengths of service in each, years of service, pension schemes, and probably more. Coding that might not, as you note, be particularly difficult. Getting all the answers is going to mean lots of man-hours from the contractor, and lots of man-hours for OPM employees.
posted by ROU_Xenophobe at 10:48 AM on March 23, 2014 [1 favorite]
To build on what ROU_Xenophobe just wrote, I can also say that at my agency we lost a lot of senior HR people to retirement when we were in the middle of a years-long hiring freeze. That has resulted in huge problems for us because, yes, everything is written down somewhere, but now the rules are being interpreted by people with far less experience than 5 years ago. This inevitably results in slower processing of materials, and a much higher rate of errors and disagreement among analysts. Certainly part of the problem is poor IT practices, but at USDA you also have 150 years of accretion of rules and policies. People don't get promoted for eliminating old things, they get promoted for creating new ones. And then there's Congress... If there was a simple way out of this mess someone would be all over it because it would make them the Roger Daltrey of the federal service. I REALLY think that a lot of people dramatically underestimate how Byzantine government practices are, but it's usually because of the orders we receive from the Legislative branch, not some sort of willful thing.
posted by wintermind at 11:03 AM on March 23, 2014 [5 favorites]
posted by wintermind at 11:03 AM on March 23, 2014 [5 favorites]
And even if you have built the perfect lookup table, where every entry has been checked for correctness, Congress is going to adopt a new law the next day that will force you to correct 10% of the cells and check them for correctness. You do that, but then Congress adopts a new law...
Mining limestone in underground quarries is fairly common. Paris was built with limestone mined under it.
posted by Monday, stony Monday at 11:21 AM on March 23, 2014 [2 favorites]
Mining limestone in underground quarries is fairly common. Paris was built with limestone mined under it.
posted by Monday, stony Monday at 11:21 AM on March 23, 2014 [2 favorites]
My wife worked at Large Multinational Insurance Co, which spent $300M on a property insurance pricing tool intended to replace a legacy computerized system. Because it is highly regulated, in many ways insurance pricing is a giant lookup table with a few continuous functions; she can carefully hand check about 2 policies in a day, but it should be easy for a computer, right?
They rolled it out to a dozen states simultaneously with minimal to no field testing and turned the legacy system off with no setup to reactivate it. In a week they realized that the new system was consistently wrong, and immediately tried unrolling it, which was complete fiasco because they had literally not even considered how that could be done. In some states they were never able to get the legacy system to work again.
Now they have it limping along side by side with the legacy system, costing an additional $10M per year to operate (partly to fix its mistakes). Turns out these projects are complicated for a number of reasons. The data often come from countless sources and are poorly standardized and sanitized (detecting and correcting data errors is huge and complex). The number of states to consider does behave like permutations. There are numerous special cases, edge cases, and magic numbers. The system is required to work with other legacy systems, which have bizarre and idiosyncratic expectations and assumptions. Because finalized errors are extremely costly to fix, the accuracy requirement is very high.
posted by a robot made out of meat at 11:27 AM on March 23, 2014 [6 favorites]
They rolled it out to a dozen states simultaneously with minimal to no field testing and turned the legacy system off with no setup to reactivate it. In a week they realized that the new system was consistently wrong, and immediately tried unrolling it, which was complete fiasco because they had literally not even considered how that could be done. In some states they were never able to get the legacy system to work again.
Now they have it limping along side by side with the legacy system, costing an additional $10M per year to operate (partly to fix its mistakes). Turns out these projects are complicated for a number of reasons. The data often come from countless sources and are poorly standardized and sanitized (detecting and correcting data errors is huge and complex). The number of states to consider does behave like permutations. There are numerous special cases, edge cases, and magic numbers. The system is required to work with other legacy systems, which have bizarre and idiosyncratic expectations and assumptions. Because finalized errors are extremely costly to fix, the accuracy requirement is very high.
posted by a robot made out of meat at 11:27 AM on March 23, 2014 [6 favorites]
One thing which is often overlooked by casual observers critiquing the way software is built for the government is that in the private sector, if the requirements turn out to be insane, there's someone somewhere in the organization who is empowered to allow the developers to build some approximation of what was specified while leaving out the crazy parts. When you're working on something for the government, the responsible party is often Congress, and as we all know, they don't really give a fuck about doing things that make sense, so you're just stuck with it.
The outcome of these projects isn't necessarily tied to the process they used to develop them. It's not really fair to say "well, this is what you expect when you do waterfall." We know that it's as possible to build quality software with waterfall processes as it is with agile ones because while there are examples of both types of projects failing spectacularly, we also have examples of both methodologies succeeding.
Rather than basic methodology, I think what often prevents government IT projects from succeeding is that fundamentally the structure of the organizations involved prevents the people who are doing the work from doing a good job. Personally, I think this is because there is a tendency amongst the contractors to think of themselves as some kind of quasi-military organization where command and control approaches and rigid hierarchical management are necessary to prevent the untrustworthy laborers/developers from wasting the noble efforts of the beneficent management class.
When you don't empower your workers to make decisions - when they can't say "hey, this approach doesn't make sense because the formula would require that we perform the following steps in constant time, which would take 1 hour per person. What if we do it this way so that it'll only require some basic arithmetic while still arriving at the same answer...?" This sort of innovation is simply not possible when you're implementing something based on the letter of the law as passed by Congress.
I think the only way you can really solve this problem is by taking the work in house, although today this sort of notion is so radical that it's hardly worth going into detail on because it would never even make it out of committee.
posted by feloniousmonk at 11:38 AM on March 23, 2014 [12 favorites]
The outcome of these projects isn't necessarily tied to the process they used to develop them. It's not really fair to say "well, this is what you expect when you do waterfall." We know that it's as possible to build quality software with waterfall processes as it is with agile ones because while there are examples of both types of projects failing spectacularly, we also have examples of both methodologies succeeding.
Rather than basic methodology, I think what often prevents government IT projects from succeeding is that fundamentally the structure of the organizations involved prevents the people who are doing the work from doing a good job. Personally, I think this is because there is a tendency amongst the contractors to think of themselves as some kind of quasi-military organization where command and control approaches and rigid hierarchical management are necessary to prevent the untrustworthy laborers/developers from wasting the noble efforts of the beneficent management class.
When you don't empower your workers to make decisions - when they can't say "hey, this approach doesn't make sense because the formula would require that we perform the following steps in constant time, which would take 1 hour per person. What if we do it this way so that it'll only require some basic arithmetic while still arriving at the same answer...?" This sort of innovation is simply not possible when you're implementing something based on the letter of the law as passed by Congress.
I think the only way you can really solve this problem is by taking the work in house, although today this sort of notion is so radical that it's hardly worth going into detail on because it would never even make it out of committee.
posted by feloniousmonk at 11:38 AM on March 23, 2014 [12 favorites]
If anyone has ever tried to do vacation time accrual by hand, the know the idea of 'oh, it just needs to be a giant lookup table' has very little on the ground experience with HR or accounting, the two things at the core of pensions.
I found accounting attractive at one point because I was entranced with the idea that you could be exacting to the degree of zero error all the time (after studying design theory it was more compelling than it might seem). But it's amazing how soon it gets fuzzy. And when the numbers get larger and the time required to chase down errors the ability to simply abandon the effort as a matter of efficiency becomes an act of both trust/faith (someone isn't stealing) and budget discretion (the cost of time spent recovering money exceeds the value of recovery).
But the government doesn't have the budget discretion of Google and it has a ridiculous amount of oversight (imagine what sort of hell would result of a tea partier managed to shove through a requirement that you couldn't journal out a error during an audit). Google can burn well over $100 million on a project but if their profits continue to go up, no one complains -- in fact, they are praised for it at times. Imagine someone writing a scare piece about Google data centers the same way anti-bureaucracy articles are written ("and they have computing resources they only use 5% of the time!" "their employees are allow to take 10% of their workday and do nothing related to their job!")
Assume the value of these pension accounts is between $150-600K. It costs only $108 to process the validation and release of funds of perhaps over a half million dollars? Why would anyone even be talking about computerizing the system?
posted by 99_ at 11:52 AM on March 23, 2014 [8 favorites]
I found accounting attractive at one point because I was entranced with the idea that you could be exacting to the degree of zero error all the time (after studying design theory it was more compelling than it might seem). But it's amazing how soon it gets fuzzy. And when the numbers get larger and the time required to chase down errors the ability to simply abandon the effort as a matter of efficiency becomes an act of both trust/faith (someone isn't stealing) and budget discretion (the cost of time spent recovering money exceeds the value of recovery).
But the government doesn't have the budget discretion of Google and it has a ridiculous amount of oversight (imagine what sort of hell would result of a tea partier managed to shove through a requirement that you couldn't journal out a error during an audit). Google can burn well over $100 million on a project but if their profits continue to go up, no one complains -- in fact, they are praised for it at times. Imagine someone writing a scare piece about Google data centers the same way anti-bureaucracy articles are written ("and they have computing resources they only use 5% of the time!" "their employees are allow to take 10% of their workday and do nothing related to their job!")
Assume the value of these pension accounts is between $150-600K. It costs only $108 to process the validation and release of funds of perhaps over a half million dollars? Why would anyone even be talking about computerizing the system?
posted by 99_ at 11:52 AM on March 23, 2014 [8 favorites]
It seems to me like the first thing you do is forget about trying to capture any rules or logic, you just write a very basic and dumb document management system that can keep a bundle of files for each case. You want to go for the low hanging fruit, like eliminating the 50 emails per day from people trying to locate lost manilla folders, and all the time spent walking around and opening file drawers. And you certainly want to avoid all that bullshit about reentering data by hand and then having to double check it. The actual processing of each case file is still entirely manual, it's just that you have everything you need on the screen. Incoming files are scanned and then shredded. Archived paper files are scanned by automated machinery and a special team hired for that purpose. If you really need a paper copy of a file for some reason you can print it out, but the master data stays electronic at all times. Again, this is a completely dumb system -- it just manages a stack of scanned and OCRd files, it has no idea what they mean. Adding actual logic comes much later, only after you've got everything in the system. And maybe it never comes.
posted by Rhomboid at 12:02 PM on March 23, 2014 [2 favorites]
posted by Rhomboid at 12:02 PM on March 23, 2014 [2 favorites]
>Coding that might not, as you note, be particularly difficult. Getting all the answers is going to mean lots of man-hours from the contractor, and lots of man-hours for OPM employees.
Yes, but what sort of estimate is "a lot"?
Basically, you're getting people to fill out a lot of forms. How many forms are there? Well, you could certainly make a pretty good estimate by looking at the raw size of the legislation - perhaps take 100 random pages and see how many forms were needed.
And I'll bet if one thing the government is good at is knowing how long it takes forms to be filled.
feloniousmonk really hits the problem though - and this is, as is also mentioned above, hand-in-hand with the procurement issue. But that happens because it's a kleptocracy, so the details are deliberately obscure so that a small number of people can loot the public purse.
posted by lupus_yonderboy at 12:05 PM on March 23, 2014
Yes, but what sort of estimate is "a lot"?
Basically, you're getting people to fill out a lot of forms. How many forms are there? Well, you could certainly make a pretty good estimate by looking at the raw size of the legislation - perhaps take 100 random pages and see how many forms were needed.
And I'll bet if one thing the government is good at is knowing how long it takes forms to be filled.
feloniousmonk really hits the problem though - and this is, as is also mentioned above, hand-in-hand with the procurement issue. But that happens because it's a kleptocracy, so the details are deliberately obscure so that a small number of people can loot the public purse.
posted by lupus_yonderboy at 12:05 PM on March 23, 2014
To riff on feloniousmonk's comment, it's easy to blame government bureaucracy but the real problem is the combination of lack of empowerment and policy which discourages hiring qualified staff in-house. Most agencies have been facing budget cuts for awhile and there has been congressional pressure against hiring staff because hiring contractors allows you to campaign on having shrunk the government, you can receive donations from contractors, and few reporters will actually do the homework needed to note that "saved" positions tend to cost more once you factor in overhead.
Even if you do get permission to hire techies, the GS pay scale limits who you're going to be able to hire. There aren't all that many jobs at the top end of the scale and even fewer will be non-managerial spots which means that your sales-pitch for putting up with bureaucracy and political meddling starts with a significant pay cut for high demand professions. (This is the same reason why regulatory capture is such a problem – we'd be much better off giving the SEC staff 400% raises than having them make decisions about likely future employers when they get tired of below-market pay)
If you do manage to make it that far, you can do some interesting things. I know the CFPB runs a far more modern IT shop because they were able to start without years of legacy-blinkered culture and hire skilled staff directly rather than relying on contractors through a procurement process which basically legally binds you to the least functional waterfall process. It'd be really good for the entire government if that became the model in the future – hire good people, give them autonomy to get projects done and responsibility to deliver.
posted by adamsc at 12:16 PM on March 23, 2014 [2 favorites]
Even if you do get permission to hire techies, the GS pay scale limits who you're going to be able to hire. There aren't all that many jobs at the top end of the scale and even fewer will be non-managerial spots which means that your sales-pitch for putting up with bureaucracy and political meddling starts with a significant pay cut for high demand professions. (This is the same reason why regulatory capture is such a problem – we'd be much better off giving the SEC staff 400% raises than having them make decisions about likely future employers when they get tired of below-market pay)
If you do manage to make it that far, you can do some interesting things. I know the CFPB runs a far more modern IT shop because they were able to start without years of legacy-blinkered culture and hire skilled staff directly rather than relying on contractors through a procurement process which basically legally binds you to the least functional waterfall process. It'd be really good for the entire government if that became the model in the future – hire good people, give them autonomy to get projects done and responsibility to deliver.
posted by adamsc at 12:16 PM on March 23, 2014 [2 favorites]
CFPB’s an excellent example. The Presidential Innovation Fellowship program was also started on the basis of Todd Park’s commission of a new reading of tech-related employment law. Turns out that a lot of the pension & seniority questions associated with hiring full-timers can be circumvented with two year positions extendable to four years. There are a lot of techies completely satisfied with that length of tenure.
posted by migurski at 12:48 PM on March 23, 2014 [1 favorite]
posted by migurski at 12:48 PM on March 23, 2014 [1 favorite]
Everyone's defending the government here, explaining why things are such a mess and how the problem is too hard to automate, etc, etc. Fair enough, the WashPo article is pretty clearly critiquing bureaucracy. But I've reserved my contempt for the IT contractor, Hewitt Associates (now AON). They took the $106M, they signed the contract, they delivered nothing of value. It's disgusting.
posted by Nelson at 12:54 PM on March 23, 2014 [2 favorites]
posted by Nelson at 12:54 PM on March 23, 2014 [2 favorites]
Hewitt may also have been handed an undoable project. I’ve heard tales of current, newsworthy IT projects with line-items in the RFP for “XML databases” that seem to have been copy-pasted from experimental 90’s IT moon shots. CGI Federal, the company partially responsible for Healthcare.gov, simultaneously delivered a state-level healthcare exchange for Kentucky that by all accounts is a big success.
It’s not so much that the government or the contractors are to blame, but that the baseline framing of technology as something you might possibly buy or farm out is a problem. Or, as feloniousmonk says above, “the only way you can really solve this problem is by taking the work in house, although today this sort of notion is so radical that it's hardly worth going into detail on because it would never even make it out of committee.”
posted by migurski at 1:06 PM on March 23, 2014 [2 favorites]
It’s not so much that the government or the contractors are to blame, but that the baseline framing of technology as something you might possibly buy or farm out is a problem. Or, as feloniousmonk says above, “the only way you can really solve this problem is by taking the work in house, although today this sort of notion is so radical that it's hardly worth going into detail on because it would never even make it out of committee.”
posted by migurski at 1:06 PM on March 23, 2014 [2 favorites]
Nelson: I don't disagree with that sentiment – and would strongly support post-mortem audits + fines on failed projects – but would submit that a lot of the problems are often baked into the system. I've been involved in both public and private projects which went off the rails because everyone persisted in a broken waterfall cycle – entirely in good faith but without the introspection needed to realize that the entire dynamic was broken. Since the government purchasing process more or less dictates that kind of work, it's likely that they'd be able to say they did everything requested because the process was invented for the army to buy socks, not to develop something new and complicated.
posted by adamsc at 1:07 PM on March 23, 2014 [1 favorite]
posted by adamsc at 1:07 PM on March 23, 2014 [1 favorite]
On the subject of taking work in house: the GSA is trying hard to fix the way government IT works and is trying to set up better in-house capacity. If you're at all interested in trying to help that work, consider applying to 18F. They've made some great hires who are intimately familiar with these problems and are trying to fix the system.
posted by adamsc at 1:10 PM on March 23, 2014 [1 favorite]
posted by adamsc at 1:10 PM on March 23, 2014 [1 favorite]
I know the CFPB runs a far more modern IT shop because they were able to start without years of legacy-blinkered culture and hire skilled staff directly rather than relying on contractors through a procurement process which basically legally binds you to the least functional waterfall process.
I don't know anything about their IT department specifically, but the CFPB is also able to pay above the GS scale, which probably helps.
posted by dsfan at 1:14 PM on March 23, 2014 [1 favorite]
I don't know anything about their IT department specifically, but the CFPB is also able to pay above the GS scale, which probably helps.
posted by dsfan at 1:14 PM on March 23, 2014 [1 favorite]
Incidentally, the best thinkers in this area are in the UK right now. Mike Bracken, head of the Government Digital Service, delivered a barn-stormer closing keynote at last year’s Code for America Summit. I strongly recommend it.
posted by migurski at 1:17 PM on March 23, 2014 [1 favorite]
posted by migurski at 1:17 PM on March 23, 2014 [1 favorite]
I'm delighted to see comments in this thread from several folks working to improve US IT capabilities. You're making a true civic contribution, I'm just an area man complaining. There's a growing group of people trying to find better ways for our government to build software systems. Michael Byrne at FCC is another force for good; he tweeted a comment of his own on this article.
(And just to add a bit of factual stuff to the OPM / Hewitt fiasco... OPM said "$106.5 million has been spent on planning and acquisition" on a "10-year performance-based contract estimated to be worth approximately $290 million". Hewitt collected "$21M out of $27M set aside for the project". I'm not sure if Hewitt only ever stood to collect $27M, or if that was just one item in the whole $290M contract Hewitt was responsible for.)
posted by Nelson at 1:24 PM on March 23, 2014 [2 favorites]
(And just to add a bit of factual stuff to the OPM / Hewitt fiasco... OPM said "$106.5 million has been spent on planning and acquisition" on a "10-year performance-based contract estimated to be worth approximately $290 million". Hewitt collected "$21M out of $27M set aside for the project". I'm not sure if Hewitt only ever stood to collect $27M, or if that was just one item in the whole $290M contract Hewitt was responsible for.)
posted by Nelson at 1:24 PM on March 23, 2014 [2 favorites]
The system works! Treasury successfully looted by kleptocrats. Some hundreds receive mostly-pointless and pointlessly stressful jobs - this keeps them from rioting and reinforces their position in an authoritarian social structure. Maybe even some HR stuff happens, though that's just a nice side effect.
posted by save alive nothing that breatheth at 1:28 PM on March 23, 2014
posted by save alive nothing that breatheth at 1:28 PM on March 23, 2014
My experience is that much of the problem are the unrealistic qualification requirements of government RFPs, i.e. enough revenue and reserve cash, E&O insurance, and employees. The outfits that can meet those requirements are experts are gaming the system rather than delivering working software. Small outfits don't even get to submit a bid because they're knocked out in the qualification round. (Although I recognize that with millions of dollars at stake, maximizing the Truck Number is a worthwhile goal.)
The other major portion is that nobody, neither government nor private sector, is willing to spend what it actually takes to safely and reliably deliver systems. While IT is a mission-critical, Class IV, interconnected part of nearly every government and business of any size, it is very often still accounted for as an expense rather than as a capital expenditure. So they focus on minimizing the expense rather than ensuring that a system that works is what comes out of the other end of the development process.
posted by ob1quixote at 1:29 PM on March 23, 2014 [2 favorites]
The other major portion is that nobody, neither government nor private sector, is willing to spend what it actually takes to safely and reliably deliver systems. While IT is a mission-critical, Class IV, interconnected part of nearly every government and business of any size, it is very often still accounted for as an expense rather than as a capital expenditure. So they focus on minimizing the expense rather than ensuring that a system that works is what comes out of the other end of the development process.
posted by ob1quixote at 1:29 PM on March 23, 2014 [2 favorites]
Hewitt may also have been handed an undoable project.
Wait, by "handed" you mean "agreed to do in exchange for money", right?
posted by kiltedtaco at 1:34 PM on March 23, 2014 [2 favorites]
Wait, by "handed" you mean "agreed to do in exchange for money", right?
posted by kiltedtaco at 1:34 PM on March 23, 2014 [2 favorites]
The solution is not to contract this stuff out. Hire your own programmers and do it inhouse. It'll work, it'll be cheaper and you'll be able to maintain it!!
The whole government contracting business is the biggest boondoggle of all time.
posted by fshgrl at 2:01 PM on March 23, 2014 [1 favorite]
The whole government contracting business is the biggest boondoggle of all time.
posted by fshgrl at 2:01 PM on March 23, 2014 [1 favorite]
They got the defense department budget down here. And they got the negative for all your favorite movies. They got microfilm with tax return and newspaper stories. They got immigration records, and census reports, and they got official accounts of all the wars and plane crashes... And volcano eruptions and earthquakes and fires and floods -- and all the other disasters that interrupted the flow of things in the good old U.S. Of A. Now, what does it matter? All this filing and record keeping? We ever gonna give a shit? We even gonna get a chance to see it all? This is a great big 14 mile tombstone! With an epitaph on it that nobody gonna bother to read.posted by Redfield at 2:02 PM on March 23, 2014
fshgrl: "The solution is not to contract this stuff out. Hire your own programmers and do it inhouse. It'll work, it'll be cheaper and you'll be able to maintain it!!"
Great, now you go tell Congressional Republicans that the federal agency payrolls are going to increase. The involvement of the more expensive private sector is a feature for them, not a bug.
posted by tonycpsu at 2:33 PM on March 23, 2014 [2 favorites]
Great, now you go tell Congressional Republicans that the federal agency payrolls are going to increase. The involvement of the more expensive private sector is a feature for them, not a bug.
posted by tonycpsu at 2:33 PM on March 23, 2014 [2 favorites]
You don’t fix this at the level of Congress, you address it at the local/state level so Congress has examples to work from.
posted by migurski at 2:41 PM on March 23, 2014 [1 favorite]
posted by migurski at 2:41 PM on March 23, 2014 [1 favorite]
migurski: "You don’t fix this at the level of Congress, you address it at the local/state level so Congress has examples to work from."
I love the idea of "laboratories of democracy", but successes at the state level still have to conform to federal law. Members of congress really don't care one way or another what states are doing, and since roughly half of Congress as presently constituted isn't really interested in government being functional at all, I don't think it's realistic to expect them to want to take a successful approach to IT at the state/local levels and reproduce it at the federal level.
I'd love to be proven wrong, of course -- it'd be awesome to see if, say, the shop that did the Kentucky ACA website could succeed with some smaller federal projects, but a sample size of the couple dozen state ACA exchanges isn't enough to really say definitively that states are doing it better, or to make a case compelling enough to convince people hostile toward government itself to give it a try.
posted by tonycpsu at 3:00 PM on March 23, 2014 [1 favorite]
I love the idea of "laboratories of democracy", but successes at the state level still have to conform to federal law. Members of congress really don't care one way or another what states are doing, and since roughly half of Congress as presently constituted isn't really interested in government being functional at all, I don't think it's realistic to expect them to want to take a successful approach to IT at the state/local levels and reproduce it at the federal level.
I'd love to be proven wrong, of course -- it'd be awesome to see if, say, the shop that did the Kentucky ACA website could succeed with some smaller federal projects, but a sample size of the couple dozen state ACA exchanges isn't enough to really say definitively that states are doing it better, or to make a case compelling enough to convince people hostile toward government itself to give it a try.
posted by tonycpsu at 3:00 PM on March 23, 2014 [1 favorite]
To be honest, I'm placing much of my hopes on the growing adoption of open source by government agencies. http://gsa.github.io/federal-open-source-repos/ has plenty of work by contractors but also many contributions by civil servants — which is important culturally since our work is public domain and should be embraced as such — and, even more importantly, there's starting to be a small but growing list of projects which are being built cheaply using common open-source tools rather than turning into huge custom IT projects.
That doesn't help with massive projects – healthcare.gov was never going to be a simple site due to the inherent complexity of what it needs to do — but it'd be nice if that became the exception rather than the expected norm for a government IT project.
posted by adamsc at 4:13 PM on March 23, 2014 [2 favorites]
That doesn't help with massive projects – healthcare.gov was never going to be a simple site due to the inherent complexity of what it needs to do — but it'd be nice if that became the exception rather than the expected norm for a government IT project.
posted by adamsc at 4:13 PM on March 23, 2014 [2 favorites]
Ok, when your guys' computer system that works like magic gets developed, what happens to all those people working in the mine? </sarcasm>
posted by ArgentCorvid at 5:57 PM on March 23, 2014
posted by ArgentCorvid at 5:57 PM on March 23, 2014
This example of bureaucracy has a working computer system and catchy music.
posted by arzakh at 6:10 PM on March 23, 2014
posted by arzakh at 6:10 PM on March 23, 2014
This is a much more entertaining use for an old mine.
posted by homunculus at 9:14 PM on March 23, 2014
posted by homunculus at 9:14 PM on March 23, 2014
I love Metafilter software development discussions! Great work everybody.
posted by wemayfreeze at 10:19 PM on March 23, 2014
posted by wemayfreeze at 10:19 PM on March 23, 2014
(This is the same reason why regulatory capture is such a problem – we'd be much better off giving the SEC staff 400% raises than having them make decisions about likely future employers when they get tired of below-market pay)
Incidentally, the SEC doesn't use the GS pay scale for precisely that reason.
posted by atrazine at 3:43 AM on March 24, 2014
Incidentally, the SEC doesn't use the GS pay scale for precisely that reason.
posted by atrazine at 3:43 AM on March 24, 2014
Are there any countries that do software development well? I mean, part of the problem is that no one seems to have developed a pipeline for programmers that doesn't lead to Silicon Valley eventually. It's possible to imagine developing a computer science department at the federal government, but not so long as they can make ten times as much by moving across the country.
The SEC doesn't really seem like a good model; their folks often do get poached by the industry they regulate, even though the iron triangle is particularly dangerous there, but you see the same kinds of problems with Oracle and SAP wooing the project managers from big corporations, talking them into spending millions on failed IT projects, so I don't see how you prevent it in government.
posted by anotherpanacea at 7:02 AM on March 24, 2014
The SEC doesn't really seem like a good model; their folks often do get poached by the industry they regulate, even though the iron triangle is particularly dangerous there, but you see the same kinds of problems with Oracle and SAP wooing the project managers from big corporations, talking them into spending millions on failed IT projects, so I don't see how you prevent it in government.
posted by anotherpanacea at 7:02 AM on March 24, 2014
The outfits that can meet those requirements are experts are gaming the system rather than delivering working software. Small outfits don't even get to submit a bid because they're knocked out in the qualification round.
Well... having been on the government side of a small-business set-aside contract that fizzled out amidst a storm of miscommunicated objectives and intra-office recriminations without offering anything specifically useful at the end, I would offer that even if you get a couple of good candidates in the door there are a number of ways things can go off the rails. (Maybe a member of the selection committee will railroad everyone else into choosing the suboptimal bid for reasons of personal prejudice. Maybe the RFP steams through the approval process based on one person's glorious yet not-fully-articulated vision, leaving the rest of the office unsure what the goal is. Maybe the project sponsor will check out for a while and leave the staff guessing at ambiguities in the proposal. Maybe the contractor technically has capabilities that in practice are oversubscribed and not available for the project. Maybe an executive gets impatient and demands results early, disregarding the timetable laid out originally. Maybe all of these things and more will happen in quick succession, not that that was my experience but maybe it was.)
Let's also add to that that strong engineers have their pick of jobs, so the people working on the government retirement fix-up are going to be second-tier at best.
Also on the public side: a friend of mine works for a large government agency that is attempting to modernize/digitize its paperwork system. There is an office (named something like a Michael Bay vehicle) solely dedicated to implementing this changeover, its goal date for putting everything online is years and years away (and apparently recedes constantly), and in her words "it's where the dead weight gets shipped off to from other offices."
posted by psoas at 7:59 AM on March 24, 2014
Well... having been on the government side of a small-business set-aside contract that fizzled out amidst a storm of miscommunicated objectives and intra-office recriminations without offering anything specifically useful at the end, I would offer that even if you get a couple of good candidates in the door there are a number of ways things can go off the rails. (Maybe a member of the selection committee will railroad everyone else into choosing the suboptimal bid for reasons of personal prejudice. Maybe the RFP steams through the approval process based on one person's glorious yet not-fully-articulated vision, leaving the rest of the office unsure what the goal is. Maybe the project sponsor will check out for a while and leave the staff guessing at ambiguities in the proposal. Maybe the contractor technically has capabilities that in practice are oversubscribed and not available for the project. Maybe an executive gets impatient and demands results early, disregarding the timetable laid out originally. Maybe all of these things and more will happen in quick succession, not that that was my experience but maybe it was.)
Let's also add to that that strong engineers have their pick of jobs, so the people working on the government retirement fix-up are going to be second-tier at best.
Also on the public side: a friend of mine works for a large government agency that is attempting to modernize/digitize its paperwork system. There is an office (named something like a Michael Bay vehicle) solely dedicated to implementing this changeover, its goal date for putting everything online is years and years away (and apparently recedes constantly), and in her words "it's where the dead weight gets shipped off to from other offices."
posted by psoas at 7:59 AM on March 24, 2014
Are there any countries that do software development well?
Migurski mentioned the UK, particularly on the strength of Mike Bracken and his team's work reinventing UK IT process. data.gov.uk is one output you can see from their work, in this case making government-produced data more accessible.
Estonia is another interesting example, a country that's intensely modernized around mobile devices and a national ID infrastructure. Here's a glowing summary written by a US VC firm.
posted by Nelson at 8:30 AM on March 24, 2014
Migurski mentioned the UK, particularly on the strength of Mike Bracken and his team's work reinventing UK IT process. data.gov.uk is one output you can see from their work, in this case making government-produced data more accessible.
Estonia is another interesting example, a country that's intensely modernized around mobile devices and a national ID infrastructure. Here's a glowing summary written by a US VC firm.
posted by Nelson at 8:30 AM on March 24, 2014
Itaxpica: "Yeah, the fact that you could write off $25 million-$100 million for a failed project as "not crazy" is itself absolutely insane. You could pay salary and benefits for 500 highly skilled Google/Apple/Facebook engineers for a year with that money, and I guarantee they would have delivered a system that worked.
Government IT is so deeply fucked in so many ways it's astounding."
It's clear you don't actually have any idea what it takes to automate a process on this scale. $100M is less than 6 months budget for this branch. If I could automate a 100% manual process for that kind of money, I'd retire a bazillionaire.
This it the problem with discussing "what's wrong with the government". People begin with wild-assed assumptions that they mistake for knowledge, and... well, at that point they jump right to the Answer.
posted by IAmBroom at 7:24 PM on March 24, 2014 [3 favorites]
Government IT is so deeply fucked in so many ways it's astounding."
It's clear you don't actually have any idea what it takes to automate a process on this scale. $100M is less than 6 months budget for this branch. If I could automate a 100% manual process for that kind of money, I'd retire a bazillionaire.
This it the problem with discussing "what's wrong with the government". People begin with wild-assed assumptions that they mistake for knowledge, and... well, at that point they jump right to the Answer.
posted by IAmBroom at 7:24 PM on March 24, 2014 [3 favorites]
« Older The Vatican archives are being digitized. | Formula One Racing documentaries, from the first... Newer »
This thread has been archived and is closed to new comments
posted by jenfullmoon at 8:56 AM on March 23, 2014 [1 favorite]