The government expected to save about $70 million a year
June 2, 2018 5:16 AM   Subscribe

"A lot of us were middle-aged women so, it's like, who was going to listen to us?" An auditor general's report has concluded that the billion-dollar Canadian federal Phoenix payroll fiasco (previously), which has underpaid or overpaid more than half of Canadian civil servants, arose from a "culture that makes people believe they can't bring forward those problems."

In the words of the report:
We found that Public Services and Procurement Canada did not fully engage departments and agencies during the development of Phoenix. The Department did not seek their extensive experience and knowledge in processing complex pay requests, which would have helped the Department to develop Phoenix to meet their needs.
In addition:
To assess whether Phoenix was ready government-wide, Public Services and Procurement Canada had planned to conduct a pilot implementation with one department. A pilot would have allowed the Department to determine if the system would work in a real setting without affecting pay that was still being processed by the old pay system.

However, we found that in June 2015, Phoenix executives cancelled the pilot because of major defects that affected critical functions and outstanding problems with system stability, and they did not have enough time to reschedule the pilot without delaying Phoenix implementation. They decided that rather than delaying Phoenix, there would be no pilot.
posted by clawsoon (40 comments total) 21 users marked this as a favorite
 
This is exactly how civil service/public employee processes get done.

The people at the top disrespect the people who do the work - I've seen it again and again. They think that the actual payroll staff, or accounting staff, or admin staff are just useless slackers who could be replaced by trained animals and that therefore their work must be trivial, lightweight and easy to automate, as well as requiring no experience or depth of knowledge.

So they institute these systems on the cheap with the basic goal of cutting staff, and then they're astonished when it turns out that actually you do need the time and the experience to do the work, and that employees with experience aren't just trash.

We need to take some lessons from the teachers and start striking occasionally, TBH.
posted by Frowner at 5:45 AM on June 2, 2018 [49 favorites]


Oh god yes, this is how my last big tech project went, actually, this is how the last 5 years of my career went, and led to me making a career change. Mediocre men, promoted over excellent women, being incredibly stupid in the name of "decisive" or worse, "transformative" leadership.

I am loving the culture shift in how the reporting on these things is going.
posted by warriorqueen at 6:05 AM on June 2, 2018 [19 favorites]


Hundreds of millions of dollars wasted implementing a system that did an imaginably worse job than whatever it was replacing.

The "obedient" culture of the public service also contributed to the failure of Phoenix, Ferguson said, as public servants reduced budgets in an effort to please political masters who like to see projects completed on time and on budget.

"Public servants" in this case almost certainly means middle managers, because reducing budgets to please political masters is basically all public sector middle managers do these days.
posted by The Card Cheat at 6:20 AM on June 2, 2018 [1 favorite]


Preferring project deadlines and budget over product functionality and reliability? Yet more 'improve the numbers not the product' management.
posted by Jessica Savitch's Coke Spoon at 6:21 AM on June 2, 2018 [3 favorites]


Okay, after that kneejerk reaction I just want to add:

- blaming "an obedient culture" is still blaming the underlings! The executives were stupid and whether or not people were being obedient, it doesn't really take that much insight to understand if you eliminate a chunk of the work to be done for budget reasons, and then you eliminate the testing phase, you might be creating a huge problem.

- Canadian organizations are really prone to this because they look at US implementations and they think we can accomplish the same thing...but with way, way smaller budgets and smaller teams. I saw this over and over in media moving things online. How come we can't have multimedia like The New York Times! Maybe because they have 25 people working on it and we have like...3.5, half of whom have other duties.

- executives seem to believe that technology should always cost less, and do more. I think people who are late Boomers expect, on some level, that everything is going to work the way it did when spreadsheets and word processing (probably Lotus 1-2-3 and WordPerfect) entered the office and suddenly a lot of things were way easier to do, and especially, to modify. Suddenly your memos didn't have to be perfect before they were typed, etc.

- and just for snark, “In order to prevent an incomprehensible failure like Phoenix from happening in the future, the government has to understand which parts of its culture are causing that type of action.” - I don't think incomprehensible means what you think it means here. My edit: "In order to prevent incomprehensible arrogance and stupidity leading to failure"
posted by warriorqueen at 6:23 AM on June 2, 2018 [31 favorites]


As a former IBMer and a current federal government employee, I have so very, very many thoughts about this. But I am changing contract status in one week and I need Phoenix, the God of All Things Holy, Like Eating and Paying Rent to smile on me, so I will keep them to myself.
posted by jacquilynne at 6:34 AM on June 2, 2018 [15 favorites]


warriorqueen: blaming "an obedient culture" is still blaming the underlings!

Yep. "If only these middle-aged women who the government had already announced it was going to let go had done..." ...what, exactly?

I've been involved off and on for the past couple of decades in automating processes on much smaller scales - more like 20-200 people, not 100,000+ people. I've had the chance to see a number of automation projects fail either a little bit or a lot. Where I'm at right now: Ask the people who know how the process works what would make it work better, then implement what they ask for. (Then test with a small group.) For whatever reason - and I'm sure that there are interesting cultural and sociological reasons for this - it seems that there's a preponderance of middle-aged women who know how the processes actually work.
posted by clawsoon at 6:36 AM on June 2, 2018 [17 favorites]


jacquilynne: so I will keep them to myself

:-(
posted by clawsoon at 6:37 AM on June 2, 2018


Another of the driving factors in automation projects (as it should be) is the demographic timebomb - more and more of the highly skilled and experienced civil servants are nearing retirement, and decades of hiring cutbacks mean that there are relatively few younger staff to learn those skills from them. In the next 3-5 years, half of our province's civil servants are eligible for retirement, which I personally think a good fraction will take. The stress of doing more with less for so long makes a nice quiet retirement look mighty good.

Added to this is the competition - citizens can shop 24x7 from any IP address in Canada, so why do they have to physically go stand in line at a government service center? Why do they have to wait until 10 AM Eastern time to telephone someone and wait on hold for twenty minutes? We have to automate, we have no choice - but we also have to push back hard on unrealistic demands from above, and not many can do that. Yeah, this is blaming us for our master's failings, but what choice do we have? Wait for the electorate to put tech-savvy politicians into office?

Btw, I can find plenty of examples of shitty private sector projects too - many have sunk their companies, and for much the same reasons. But that's a subject for another FPP. :)
posted by Mogur at 6:40 AM on June 2, 2018 [3 favorites]


We found that Public Services and Procurement Canada did not fully engage departments and agencies during the development of Phoenix.

HA HA I AM NOT AFFECTED BY THIS RELATABLE CONTENT AT ALL.

We're doing a massive project where I work--it's a complete replacement of our student records system. I say "we", but it's really "they", because no one has consulted me or any of the other three or four people I work with.

Oh, us? What do we work on? Only the main student-facing system that pretty much everything else hangs off of. Student enrollments. Course materials. Assigment submissions and grades. Timetables. The sort of thing that gets and puts a lot of its data from and into a student records system, and about which you might reasonably expect to be asked somewhere around the requirements phase of a project like this.

NOPE. They're bringing the very first phase online in a month or two. Total number of meetings with us, the people who develop for and support this pretty vital piece of what the university does: three. It is going to be a massive pile-up of Final Destination 2 proportions, and the only thing I'm really glad of in all this is the more or less iron-clad guarantee of job security that is the looming certainty that it will take years to clean it all up.

Okay, just one example. Just one. Completely and utterly without talking to us, they've changed the internal IDs of university courses from the sensible system that we already use--e.g., ENG1080.12345 for a particular course and a specific instance of that course--to a completely meaningless and arbitrary number like 8675309. We key everything off of that course ID prefix. Everything. Things are going to break like a suspension bridge made of Twizzlers, and I'm not going to be paid nearly enough for helping to fix it.

Sorry. Had to get that off my chest, mainly because the other guy in my office is tired of hearing it.
posted by Mr. Bad Example at 6:43 AM on June 2, 2018 [43 favorites]


But using an auto-incrementing integer for your primary key is pretty standard. What happens when departments get split, combined, or renamed, but you still want to track the course, including history of the course?
posted by Mogur at 7:02 AM on June 2, 2018 [3 favorites]


"culture that makes people believe they can't bring forward those problems."

This is a super weird spin on the linked article under this text, where the quoted text is just one among many issues raised, and clearly not the primary one; for some reason what the article (And the report) identifies as the primary cause seems to not even mentioned in the OP? (Actually, the quoted text doesn't seem to be in the article it links at all, so maybe it's just a linking mistake. But framing things like this in the OP while ignoring the fact that the report does actually blame the people at the top is rather odd; this "culture" stuff doesn't really appear in the report in the way this quote suggests.)

From the linked article:
Earlier Tuesday, Canada's Auditor General Michael Ferguson described Phoenix as "an incomprehensible failure" which he laid mainly at the feet of three so-called "Phoenix executives," high level bureaucrats in charge of implementing the new pay system.
[...]
Ferguson noted there was plenty of blame to go around, from the previous Conservative government that approved the Phoenix project to the current Liberal government that launched the system.

However, he said a great deal of power rested with three top bureaucrats who oversaw Phoenix, who in some cases kept key information from the department's deputy minister, and who also ignored dire warnings from outside departments and consultants.

"The project executives were to blame for the project failures. The deputy minister who was in place when the system was launched is accountable for the failure that happened on his watch," Ferguson said.

"The former government is accountable for not having built an appropriate oversight mechanism…and the current government is also responsible for fixing the problem, a fix that will have an incremental cost of more than $1 billion and will take years."
Conclusion (1.105) from the auditor general's report, which doesn't even remotely mention the "culture" stuff:
We concluded that the Phoenix project was an incomprehensible failure of project management and oversight. Phoenix executives prioritized certain aspects, such as schedule and budget, over other critical ones, such as functionality and security. Phoenix executives did not understand the importance of warnings that the Miramichi Pay Centre, departments and agencies, and the new system were not ready. They did not provide complete and accurate information to deputy ministers and associate deputy ministers of departments and agencies, including the Deputy Minister of Public Services and Procurement, when briefing them on Phoenix readiness for implementation. In our opinion, the decision by Phoenix executives to implement Phoenix was unreasonable according to the information available at the time. As a result, Phoenix has not met user needs, has cost the federal government hundreds of millions of dollars, and has financially affected tens of thousands of its employees.
posted by advil at 7:21 AM on June 2, 2018 [6 favorites]


My apologies, advil. The quote is from a statement by the auditor general about the report that's quoted in this article. I mixed up which articles I was linking.
posted by clawsoon at 7:25 AM on June 2, 2018


In a very small way I got stuck in the Phoenix debacle a few years ago! I worked as a poll clerk in the last federal election and never got paid. It took about 6 months of weekly phone calls to the fed's payroll call centre to finally get a cheque. At one point, after numerous calls where the agents were friendly and polite but had no idea where my money was, I got a call back from a fraud investigator who told me I had been reported to the RCMP and to expect a visit from them! Of course that never happened and when I called the following week, the supervisor I spoke to apologized by saying there was no record of that call but it wouldn't surprise her that someone would do such a thing.

And this was all for $300! Thank fucking god I wasn't relying on this to pay my mortgage every month.
posted by monkeymike at 7:45 AM on June 2, 2018 [4 favorites]


However, he said a great deal of power rested with three top bureaucrats who oversaw Phoenix, who in some cases kept key information from the department's deputy minister

Wow- if true, that alone is pretty shocking. It's the public-sector equivalent of lying to your CFO or COO.



We concluded that the Phoenix project was an incomprehensible failure of project management and oversight.

Ironically, this is probably one instance where "inconceivable" would have been a more appropriate word choice, and would mean what you think it means.
posted by TheWhiteSkull at 9:15 AM on June 2, 2018 [2 favorites]


I thought 'comprehensive failure' would have been closer to what seemed to be being described -- and has the advantage, or disadvantage depending on your point of view, of being a familiar phrase.
posted by jamjam at 9:43 AM on June 2, 2018


"Yeah, we're not going to listen to your objections or advice, and we're going to sideline anyone who speaks out."
...
"The problem is a culture of obedience."
posted by evidenceofabsence at 10:10 AM on June 2, 2018 [18 favorites]


Ferguson noted there was plenty of blame to go around, from the previous Conservative government that approved the Phoenix project to the current Liberal government that launched the system.

here I was hoping this would all get laid in the laps of Conservatives, so yeah, call my biases unconfirmed.

But that said, the paranoid part of my political mind can still see this as a deliberate Conservative plot to fuck things up for the Liberals. They knew they'd probably lose the next election. They knew this payroll system would be a fiasco*. They gambled that the incoming Libs would be stupid enough to just go ahead with it because, well, (and speaking of confirmation bias), look at the mess they've chosen to make of the whole Kinder Morgan thing.

anyway, I do love a good disaster story. Thanks for this, clawsoon, and advil for digging into things a little deeper.

* I don't actually think the Conservatives were that cunning. Rather, I think they were like various clients of my experience (and/or heard of over the years), who don't get hi-tech, who make software demands that contradict each other (ensure incompatibility), but the software company needs/wants the gig, so they say, "Sure, we can do that," often without even consulting with the geeks who'll have to accomplish the task. I believe FUBAR applies.
posted by philip-random at 10:25 AM on June 2, 2018


This all seems like a totally predictable consequence of trying to do a $275 million project on a $155 million budget.

The part that is really amazing to me is that yes, 51% of paychecks had errors under Phoenix— but even in the old system, 30% of paychecks still had errors!
posted by phoenixy at 10:38 AM on June 2, 2018 [3 favorites]


but even in the old system, 30% of paychecks still had errors!

back in the late 1990s when I did some contract work for CBC radio, I literally never saw my boss. He was completely consumed with the ultimately impossible task of reconciling two independently developed software systems that were never intended to "talk to" one another. One had been developed in Toronto, the other in Vancouver. And then some genius(es) in upper management decreed that the two be integrated ...
posted by philip-random at 10:52 AM on June 2, 2018 [1 favorite]


What shocks me most about Phoenix is that when things went wrong there was no alternative way of paying people. Cash advances, cutting cheques, those all used to be a bit burdensome but definitely in the realm of the possible (I can remember a time where I’d get a stack of crisp red $50s and then have to salute the Queen) but in the land of Phoenix there was no way to temporarily redress an obvious payroll error. How people aren’t able to sue/shutdown things is beyond me.
posted by furtive at 11:34 AM on June 2, 2018 [3 favorites]


It's only implied, but they moved ALL payroll to Miramichi -- obviously experienced payroll personnel do not want to up and move there, because no one wants to move there, but especially not when you have a family and a spouse who probably works in a different department in the government and can't just move to NB, and this was how they got rid of everyone with any experience for "what do we do when Mary is on half time secondment from department A to department B and is off on vacation/stress leave/maternity leave" (stress leave comes up a lot -- for instance, in all the new employees working in Miramichi doing payroll in a terrible system they were untrained on with excessive files to process).

Everything I have heard from people I know in Ottawa places equal blame on the Conservatives and Liberals for this.
posted by jeather at 11:49 AM on June 2, 2018 [4 favorites]


It's also important to note that in the old system, payroll was also incorrect a lot. Less incorrect, but there was a real need to improve.
posted by jeather at 11:57 AM on June 2, 2018 [1 favorite]


blaming "an obedient culture" is still blaming the underlings!

I'm sure it sounds like criticism to them, but yeah, no it isn't. As a manager, that's a a second stab (in addition to "that was a stupid thing to do") right to the heart. It's explicitly absolving underlings of the "but why didn't you speak up, it's your responsibility, you know" deflection.

Managers leaders are responsible for the culture of their organization; it doesn't just magically happen to you. The culture you have isn't a matter of luck. People (attempt to) make mistakes and dumb decisions all the time, but because of the culture of caring, engagement, and empowerment, the org is resilient against the occasional dumbass attack. "Culture of..." directly calls management bad. Period.
posted by ctmf at 12:03 PM on June 2, 2018 [4 favorites]


I assume O Mighty Phoenix will not be displeased if I comment in a way that does not implicate him. One of the things I am only starting to understand about working in government is the mind-boggling complexity of government payroll. Among my limited circle of acquaintance in government, there is almost no one who actually knows how much they are supposed to be getting paid. Everyone has these complicated webs of acting pay, assignment adjustments, catch-up payments, leave averaging, different benefit plans, high-low pensions, pension buy back payments, etc, etc -- and I work in a department where everyone just pretty much comes to the office every day, so relatively speaking our pay is simple. All of that is currently being complicated by Phoenix adding on a couple of extra layers of past pay mistakes being caught up with adjustments and current pay mistakes making everything weird again but it seems like it was already kind of a mystery to most public servants even before Phoenix.

The 30% errors pre-Phoenix is surprising, not because the number is high but because anyone could determine how many paycheques were wrong.
posted by jacquilynne at 12:14 PM on June 2, 2018 [5 favorites]


As one of Canada's happy band of civil servants, I am currently awaiting a pay fix from the Phoenix system, with no hope of redress likely for months, if not years. I am constantly amazed at the lack of concern among my colleagues in a similar position, which I put down to the Canadian culture of politeness. In my country of origin (I am newish Canadian), no-one would tolerate being treated this way by an employer, and would be taking to the streets to protest. What is needed is a 1-day general strike to remind our managers, and their masters, the great Canadian public, that we need a faster fix. My solution: get everyone to estimate how much they are owed, cut them a cheque and fix the details later. I know colleagues who are unwilling to take parental leave to avoid having their pay cut off as a reward for returning to work, others have not been paid properly (or, in one case, at all) for months. It is an atrocious situation, which is corroding morale on a daily basis.
posted by aeshnid at 1:05 PM on June 2, 2018 [1 favorite]


pre-Phoenix everyone had a departmental compensation advisor assigned to their file who could fix problems right away. Now there is just a call centre in Miramachi or wherever. Good luck getting through to them, and good luck finding someone familiar enough with your file to fix it.
posted by fimbulvetr at 1:44 PM on June 2, 2018 [2 favorites]


Has any of these ambitious national level IT project ever worked out anywhere?
posted by srboisvert at 2:35 PM on June 2, 2018 [1 favorite]


What is needed is a 1-day general strike to remind our managers, and their masters, the great Canadian public, that we need a faster fix.

Management will look and say "They wildcated for a day and the world didn't fall apart. Let's give them 1 day off a month with no pay going forward."

For nostalgia's sake they could name them after Bob Rae.
posted by srboisvert at 2:37 PM on June 2, 2018


In what I can only assume is a deliberate act of self-parody, the government recently announced it would be moving to a "Pod Model" for dealing with compensation issues. In this "Pod Model" or "Pay Pod" system:
"...groups will work with client departments to address all outstanding transactions in an employee's pay file, in contrast to the current approach of addressing pay issues by transaction type,"
Yes. Ladies and gentlemen, after having done away with the archaic model where government departments employ payroll advisors who work for that department, to solve payroll issues specific to that department; having dealt with the massive issues that causes; the people running the day to day machinery of government have hit on this brilliant solution:
Government departments are ASSIGNED payroll advisors working for PWSGC to solve payroll issues specific to a particular department.
I need to go drink and re-watch "Yes, Minister", until I can laugh at this without screaming.
posted by Grimgrin at 2:45 PM on June 2, 2018 [2 favorites]


Yes. Ladies and gentlemen, after having done away with the archaic model where government departments employ payroll advisors who work for that department, to solve payroll issues specific to that department; having dealt with the massive issues that causes; the people running the day to day machinery of government have hit on this brilliant solution:

This whole thing frosts me so much. It has happened in various ways where I work - "oh, let's centralize all the instances of [complicated transaction] and fire a bunch of people" and then it becomes apparent that different areas tend to have different needs and it's a lot easier to resolve a problem when Joe in your department can make a note "fix Tom's appointment, issue check" and keep that on his desk until it's done than when Someone Somewhere Off Campus In A Building has a queue of forty people whose appointments need fixing, none of whose roles or pay types are particularly familiar to Someone.

Expertise matters. Not reinventing the wheel matters. Getting someone in a job and training them and letting them build up a few years of experience makes things so much better than having a steady churn of people in and out of a boring queue-based gig where they push buttons all day.
posted by Frowner at 3:03 PM on June 2, 2018 [3 favorites]


But using an auto-incrementing integer for your primary key is pretty standard.

It may be pretty standard, but that does not mean it's a good idea. You can only do that if all your data will fit on one machine. If you have more data than that, obviously you need to store it in some sort of multi-machine cluster... and there is no reasonable distributed way of implementing an auto-inc integer.

(The most common way of doing lockless IDs in a distributed system is by using UUIDs, but when the underlying data has a natural unique "name", like the course number example, it is madness to not use it as your primary key.)
posted by phliar at 3:07 PM on June 2, 2018 [4 favorites]


A friend who works for gov says a co-worker turned down a promotion because they were currently getting paid correctly and didn't want to risk changing anything.
posted by RobotHero at 3:32 PM on June 2, 2018 [6 favorites]


The government expected to save about $70 million a year

And that was probably possible, in the modern world, except for the universal incompetence of the leadership we've been sold. Imagine if we'd got what we pay for.

As Johnny Rotten put it: 'Ever get the feeling you've been cheated?'
posted by Twang at 4:36 PM on June 2, 2018 [1 favorite]


Mediocre men, promoted over excellent women, being incredibly stupid in the name of "decisive" or worse, "transformative" leadership.

Huh. So the software consulting company I work for has the slightly unfortunate claim-to-fame of being the only contractor to deliver a successful project on time and on budget for a large government entity several years back. I'm now thinking about the fact that the head developer on the project was a woman, our area director is a woman, and the two people directly in charge of recruitment and hiring are women.

Interesting.
posted by Nonsteroidal Anti-Inflammatory Drug at 8:09 PM on June 2, 2018 [5 favorites]


I say "we", but it's really "they", because no one has consulted me or any of the other three or four people I work with.
Oh god, I thought this was just my hopelessly siloed institution! Separating (permanently and with nearly zero communication) the people who work daily with the system from the people who make all the high-level decisions about the system seems like an obviously terrible idea and yet, institutions and governments keep doing it!
posted by soren_lorensen at 6:50 AM on June 3, 2018 [2 favorites]


All three of the unnamed executives are retired (and 2/3 of them were women if that pokes a hole in your mental image).
posted by quaking fajita at 7:56 AM on June 3, 2018 [1 favorite]


It definitely does, thanks for sharing that info.
posted by warriorqueen at 9:01 AM on June 3, 2018


While it's a bit of a side issue and summary statistics like this don't tell the whole story, the Canadian PS has done a lot towards increasing the diversity of the PS, even among the executives*.

That's not to say that the cultural biases that lead to statements like "A lot of us were middle-aged women so, it's like, who was going to listen to us?" are solved just like that, but I think it's worth giving credit where credit is due. I think a lot of organizations could learn from the way the PS does that particular aspect of its hiring: managers are not required by policy to hire the "best qualified" candidate, and can hire based membership in employment equity groups from among the candidates that meet the job's designated essential criteria. (Although PS hiring as a whole is a gong show ASK ME HOW I KNOW.)

* The "workforce availability" number of 10% for "members of a visible minority group" doesn't match what I know about the demographics of the Canadian population, but I don't have time right now to figure out what's going on with that discrepancy.
posted by quaking fajita at 9:48 AM on June 3, 2018


A brief continuation of the aside on diversity:

The "workforce availability" number of 10% for "members of a visible minority group" doesn't match what I know about the demographics of the Canadian population,

So, I work in Employment Equity for the government, and am looking at our workforce availability right now, and for Visible Minorities in my department the WFA is 12%, still low though.

In context, the workforce availability is still being derived from the 2011 census (and probably will be until 2020 or so) and it is calculated based on Canadian citizenship. So, visible minorities (or anyone even) who are not citizens are not counted towards the workforce availability for the Government of Canada.

/aside
posted by aclevername at 7:10 AM on June 4, 2018 [2 favorites]


« Older Ello, ello, wot's all this then?   |   The Train of Tommorow Newer »


This thread has been archived and is closed to new comments