DETECTING AGILE BS
January 24, 2019 10:55 AM   Subscribe

The purpose of this document (pdf) is to provide guidance to DoD program executives and acquisition professionals on how to detect software projects that are really using agile development versus those that are simply waterfall or spiral development in agile clothing (“agile-scrum-fall”).
posted by jenkinsEar (137 comments total) 69 users marked this as a favorite
 
As a software engineer who does contract work for the DoD, this document is definitely on the right track. But this problem is by no means exclusive to the DoD...I feel like most organizations that say they are "Agile" actually have no real understanding of the term.
posted by Edgewise at 10:59 AM on January 24 [13 favorites]


Don't forget that the traditional waterfall model is sometimes more like a wacky water ride waterfall with random jets of water shooting out from any direction and whirlpools of endless meetings swirling into a void.
posted by sammyo at 11:12 AM on January 24 [40 favorites]


As someone who has only ever worked at places that don't have any particular development philosophy other than, "Let's work as a team and get shit done," is there actually any obvious net value to Agile or any of these other systems?
posted by Anticipation Of A New Lover's Arrival, The at 11:12 AM on January 24 [10 favorites]


The best, most well-designed, on-time projects I've ever done were waterfall. Every Agile project I've been on turns into a scope creep festival with a moving target of a launch date.
posted by grumpybear69 at 11:12 AM on January 24 [39 favorites]


But as a caveat, those waterfall projects were at a very high-functioning org with every single one of their ducks in a row, so that may have had more to do with the success than the particular methodology we used.
posted by grumpybear69 at 11:15 AM on January 24 [15 favorites]


(“agile-scrum-fall”)

Yeah in the software dev industry we refer to this as the much more smoothly-off-the-tongue "wagile."
posted by allkindsoftime at 11:16 AM on January 24 [12 favorites]


Also I just read that table of Agile Values and it made my eyes pop out of my head with horror. Contracts don't matter? Documentation doesn't matter? Process doesn't matter? Like, how do you set and manage expectations or ever do anything in a repeatable way or stop your organization from turning into a battling scrum (see what I did there) of egomaniacs who believe rules don't apply to them because they are Good At Their Jobs?
posted by Anticipation Of A New Lover's Arrival, The at 11:18 AM on January 24 [13 favorites]


Honestly, the more of this I read the more Agile sounds like basically a cult.
posted by Anticipation Of A New Lover's Arrival, The at 11:20 AM on January 24 [43 favorites]


In the end it’s all code and fix....
posted by rivets at 11:22 AM on January 24 [6 favorites]


Oh it's multiple (consulting for a fee) warring cults.
posted by sammyo at 11:22 AM on January 24 [18 favorites]


My scrum-master says you're all wrong

(as someone who regularly hires technical people, I've found that people who list 'scrum' or 'agile' skills on their resume have fewer useful skills than people who can, you know, actually develop things)
posted by AzraelBrown at 11:26 AM on January 24 [19 favorites]


Let's work as a team and get shit done

This is a pretty accurate summation of the original Agile Manifesto, TBH.
posted by tobascodagama at 11:26 AM on January 24 [10 favorites]


Isn't that all jobs that involve more than one person though?
posted by Anticipation Of A New Lover's Arrival, The at 11:30 AM on January 24


I've been on acquisitions programs where someone got it in to their head that Agile needed to be in the contracting structure and the contractor claimed they were going to do "Agile" hardware development. I asked how this was going to be accomplished (each delivered unit costing, oh, 8 figures or so). I'm not sure I ever received a satisfactory answer.
posted by backseatpilot at 11:32 AM on January 24 [1 favorite]


> Honestly, the more of this I read the more Agile sounds like basically a cult.

It can definitely become that way, especially among those particular sorts of managers who find the unending march of studying for certifications a far more comforting life than that of working towards delivering a product. At its heart Agile is pretty simple and straightforward and can be loosely summarized as: "The work necessary to make a thing is best determined by the people involved in the making of it building a consensus regarding the steps needed to do it, and then adjusting as needed until the thing is done." Everything beyond that is rulesbuilding and elaboration, sometimes circumstantially necessary and sometimes spurious.

The number one reason why Agile projects have failed, in my experience, have been because stakeholders outside the development and design teams refuse to allow themselves to be beholden to the same processes they hold their teams to. So features are added or changed without moving the deadline (or by moving the deadline a small amount by fiat, rather than by discussion with the team). Or management tries to collapse the schedule so that development has to start before the product's design and requirements are determined, or they wall off the discrete teams working on fragments of a product so that added time is necessary to get the pieces to work together properly.
posted by at by at 11:33 AM on January 24 [31 favorites]


This brings back memories of an Agile contract (I am a technical writer/editor) where I was routinely told "Oh, we've got this" and wouldn't let me document anything except the UI.

I was mad as hell for two weeks until I realized they'd given me an office with a door and absolutely nothing to do since a) I was not an Agile acolyte, and b) I checked in with the project manager twice a day. My actual boss called the client and asked if they wanted to stop the contract, and they said "Oh, no, catlet needs to be here but we don't actually need her to do anything. We just need a writer to be on staff, but one of our guys will do it all."

So, I drank a lot of expensive coffee and wrote a book.
posted by catlet at 11:34 AM on January 24 [71 favorites]


On reading the document, it actually makes quite a few good points about project management in general; we're a small shop and don't have a very rigid project planning method, but I'm still going to forward it to my developers. It makes some good points about making sure what you're doing is useful, and not just crossing off the steps to being done. I don't think it specifically rules out waterfall or other planning methods (waterfall is perfectly fine for the small projects we do), it sounds more about consciously staying on track and that simply declaring it Agile doesn't automatically mean things are on track.
posted by AzraelBrown at 11:36 AM on January 24 [1 favorite]


I feel bad that I kinda phoned it in on a recent job application. I had to explain how I was familiar with Agile methodology and I'm like, "well I worked at that company one time, but mostly I build software for scientists so the goal posts are always moving and I have to release multiple times a day sometimes." I'm pretty sure they'll take that as some kind of grave insult.
posted by klanawa at 11:37 AM on January 24 [3 favorites]


The best, most well-designed, on-time projects I've ever done were waterfall.

This is 100% the reverse of my experience, in the ~18 years I've been making software professionally. The idea that anything of real complexity can be planned out in excruciating detail (design, team schedules, etc) 18-24 months in advance is insanity, and has failed literally every project I've ever seen it attempted on.

Oh, I've had people try to claim success on those projects, of course -- but those people always ignore that we dropped 50% of the original scoped requirements, added 25% more we'd never heard of till 3 months before the end date, and also the thing that was built is a shambling horror that falls over in the slightest wind.
posted by tocts at 11:38 AM on January 24 [14 favorites]



But if we don't have agile then we wouldn't need an MBA grad, who really wanted to work in a bank, to run daily meetings where everyone explains what they're working on?! The team who all sit together would just somehow know. Is this the sort of dystopia we want our technology developed in?
posted by Damienmce at 11:39 AM on January 24 [4 favorites]


Also I just read that table of Agile Values and it made my eyes pop out of my head with horror.

I think it's easy to misunderstand agile values, because as paraphrased they don't really get the intent across.

For example, the point is not that documentation is bad, it's that over-documenting things that are obvious or spending so much time documenting (particularly writing super-detailed design documents) that you don't actually have time to deliver value is bad. I've never worked on an agile team where we didn't document things as much as was truly needed, but we always strive to keep documentation simple and to-the-point. Nobody needs a 20 page write-up of the exact structure of your tiny component that will be out-of-date the first time someone makes a code change. Document the external interfaces, document any important design decisions, and move on.

Similarly, the point is not that contracts are bad, but that you should be more focused on meeting customer needs than defining up front precisely what you will build. A focus on contracts over value means you're fundamentally taking an adversarial stance with your users -- you're basically saying you'd rather deliver them shit if they didn't do a good enough job (over a long enough time, costing way too much money) to precisely define the not-shit they want, versus taking a collaborative approach where the goal is to actually build things the customer needs.

Not focusing on contracts is really about not ever being in a position to tell your customers, "I know, now that we built it, that we realize this thing doesn't work, but you agreed 6 months ago this is how we'd build it, so as far as I'm concerned my job is done!".
posted by tocts at 11:44 AM on January 24 [17 favorites]


> Like, how do you set and manage expectations or ever do anything in a repeatable way

Agile isn't repeatable, exactly. Repeatable software process means 'can build software to specification.' The problem is there is nobody out there who can write that specification; users are too non-technical and developers don't know the problem domain. You know those multimillion dollar projects where someone comes up with The Plan? It's a rather seductive myth, that someone can construct The Plan -- a specification, design and billing estimate -- from whole cloth. But then a year later when they try to put it all together nothing fucking works, or when you put it all together and it miraculously works, but customer won't pay because they interpreted the software spec and contract differently! That's what Agile sets out to fix.

The Agile prescription is basically 'put it together now, find out the problems in the specification, the design, and the software posthaste.' Do not trust The Plan. If The Plan gets in the way of doing that, change The Plan. The expectations your customers should have are basically 'it will get better every 2 weeks, in the ways you want.'
posted by pwnguin at 11:45 AM on January 24 [7 favorites]


This, exactly:

The idea that anything of real complexity can be planned out in excruciating detail (design, team schedules, etc) 18-24 months in advance is insanity, and has failed literally every project I've ever seen it attempted on.

Oh, I've had people try to claim success on those projects, of course -- but those people always ignore that we dropped 50% of the original scoped requirements, added 25% more we'd never heard of till 3 months before the end date, and also the thing that was built is a shambling horror that falls over in the slightest wind.


You can maintain your strategic aim, but tactically, along the way, you will always, always learn something new and change your mind to a non-zero degree. At least Agile/Scrum allows frequent resets, baked right into the process.

In my experience, the people that don't like Agile/Scrum really just want someone to tell them what to do. Or they want to tell you how they want to be told what to do. They've failed the "yes, and..." test of teamwork.

"Let's groom the backlog."
"No. Wanna know what you're doing wrong?"
/sigh
posted by Cool Papa Bell at 11:53 AM on January 24 [8 favorites]


My only problem with agile is that not every application has a front end that can be iterated, or has an easy to demo interface that can be reviewed with clients, or even if it does, it might have 50 clients, not 1. It seems to fly over so many heads. Sometimes waterfall is the best. It really depends on the project and the application. Of these apps, you can perhaps redesign them to be able to become agile-friendly, but that's not really all that financially productive.
posted by The_Vegetables at 11:54 AM on January 24 [7 favorites]


I've seen waterfall projects go off the rails because the stakeholders have no discipline when it came to scope.

I've seen agile projects go off the rails because the stakeholders have no discipline when it came to scope.

I've rarely seen stakeholders have discipline when it came to scope, but when they did, it didn't much matter WHAT trendy or un-trendy philosophy we worked within, things went OK.

Agile/Scrum/etc are often touted as specifically and uniquely capable of dealing with scope creep, but they're not.

The failure mode I have seen 100% of the time is when the Big Boss can't commit.
posted by tclark at 11:55 AM on January 24 [38 favorites]


Similarly, the point is not that contracts are bad, but that you should be more focused on meeting customer needs than defining up front precisely what you will build.

Wait, does software development not do change orders? In construction, if the client wants to change the design someone scopes it out, puts a number on it, and then it gets written up as a change order for the customer to sign (or not, if they don't like the price) and it becomes part of the contract. Is that not a thing in software?
posted by Anticipation Of A New Lover's Arrival, The at 11:55 AM on January 24 [9 favorites]


You know those multimillion dollar projects where someone comes up with The Plan? It's a rather seductive myth, that someone can construct The Plan -- a specification, design and billing estimate -- from whole cloth. But then a year later when they try to put it all together nothing fucking works, or when you put it all together and it miraculously works, but customer won't pay because they interpreted the software spec and contract differently!

Also, yeah, good lord: this. The Plan is the fucking worst. I don't care about your plan.

Tell me the business problem you want solved, and I and my team(s) will find a way to solve it. The way we solve it probably won't be how you envision it. It probably won't be how I envision it either. Nonetheless, we will solve it, by examining options, trying things quickly, throwing away things that don't work, and getting feedback constantly.

Tell me The Plan, though ... ugh. Once someone has defined The Plan, instead of the point being to solve problems, the point becomes to follow The Plan, even when it becomes clear that it's leading you off a cliff.

Fuck The Plan.
posted by tocts at 11:58 AM on January 24 [10 favorites]


Is that not a thing in software?
Buildings are built under the waterfall methodology, so yes, change orders under waterfall are common. The waterfall methodology is not the best way to build software, because like buildings, the upfront design can be awful and user-unfriendly, like a building where they put the main entrance in the wrong place. They ain't going to rip the front of the building off and reorient it due to client dissatisfaction, they are going to put in a lousy sidewalk or just let them complain.
posted by The_Vegetables at 12:07 PM on January 24 [1 favorite]


I am thinking back to that recent thread now where people were like, "No really, nobody knows how to build software," and I didn't believe them, and now I'm starting to believe them.

It's like if someone said, "Build me a house," and then you had to ask, "OK, do you want it to keep water out?" and the client said, "Well, no, some water getting in would be OK," and then people actually built that, and then the client said, "Wait, this is awful! I do want to keep water out of my house!" and then people tore down the entire house and built another one, only they weren't sure if the shingles should lap facing up or down, and they went with up, but that didn't work, so they turned them over, but by this time the house was full of mold, so they made a new house entirely out of concrete (but left the old one, because there was one person who still preferred it even with the mold) only nobody asked the client if they needed windows, so then later they had to add windows by cutting big holes in the walls, and then the client was mad that their house-building project was behind schedule, and then nobody learned anything and they just went through it all again the next time.
posted by Anticipation Of A New Lover's Arrival, The at 12:08 PM on January 24 [38 favorites]


Wait, does software development not do change orders? In construction ...

So ... yes, kind of, but the analogy isn't really apt. A huge amount of software development is about building things nobody has ever built before; if it already existed and met the needs of your potential users, they could just go buy an exact copy right now and wouldn't have to wait (because unlike houses, a new instance of an existing piece of software can be created instantaneously for basically zero cost).

Additionally, most people making software aren't being paid directly by the person who wants to use it (e.g. it's not like buying a house, where the person paying for it plus the change orders is also the person who wants to live there). They're being paid by people whose intent is to eventually sell the thing they're building. So, they don't necessarily have a precise need, they have instead an idea of what they think potential buyers will need. So, the ability to react to real feedback becomes important, lest you spend 3 years making a thing and then nobody wants to buy it.

Software development is way closer to invention than construction. We don't often know where we're going when we start, because if we did then the thing we're about to build probably would already exist, obviating any need for us to build it.
posted by tocts at 12:11 PM on January 24 [16 favorites]


It sounds like a lot of the issue is that nobody knows how to accurately describe what they want out of a piece of software, so people end up just saying "Build me a thing," and then you make a thing, and they say, "No, it needs to be different," and then iterate as necessary until it is almost what you wanted.
posted by Anticipation Of A New Lover's Arrival, The at 12:15 PM on January 24 [15 favorites]


people end up just saying "Build me a thing," and then you make a thing, and they say, "No, it needs to be different,"

I'm a systems engineer and basically my job is sitting in between there trying to hold everything together. You know, I take the requirements from the customer and give them to the developers. Well, not literally, physically carry the requirements there. My secretary does that.
posted by backseatpilot at 12:19 PM on January 24 [14 favorites]


When I did consultancy for large companies, our process was very simple and always successful.

We would not develop a requirements doc until we'd interviewed stakeholders.

We did not begin development until the requirements doc had been signed off. Our requirements docs were very thorough.

We did not deliver anything that was not in the requirements doc; conversely if it was in the doc, it was in the deliverable.

We fixed bugs that showed the software was at variance from the requirements.

Anything else was a change request and brought us more money and time.

All of the above stipulations were ironclad, and in the contract.

We made it super clear that changes very early on were much cheaper than those when we were doing integration testing.

And that's how you make happy customers and lots of money in software consultancy. N.B. if you can't get timely sign-off on your requirements, you ditch the client. A very critical aspect of having a happy customer is picking customers you can make happy. (Oh and it absolutely doesn't matter whether it's Waterfall or Agile or whatever.)
posted by seanmpuckett at 12:21 PM on January 24 [53 favorites]


...over-documenting things that are obvious ...
Yeah, I'm gonna stop you right there. What you find obvious, I probably find necessary, since I am not you. The same goes for my end users, who probably think they understand your base assumptions, but are wildly out in lala-land thinking that you meant a unicorn when you said something even slightly jargonish.

I cannot tell you how frustrating I find it when a developer says "just install this patch in _path_" without giving me a root path to start from. It's obvious to them, I'm sure, but those base assumptions really don't fly when I work in the full OSI stack and you only work in the application layer. The number of times I've had to literally corner someone and ask them to specifically name the full path for a simple patch install is ridiculous.

My rule has become "if you make any assumptions, you are making an ass out of u and mption."

Please fucking document _everything_.
posted by daq at 12:22 PM on January 24 [18 favorites]


Software development is way closer to invention than construction. We don't often know where we're going when we start, because if we did then the thing we're about to build probably would already exist, obviating any need for us to build it.

If only! Enterprise ecommerce development is almost entirely "build me this thing that people have built a thousand times before but customize it to my very specific needs." In fact the project I'm working on now is the result of them trying to use a hosted off-prem managed solution that turned out to be not flexible enough. So that statement, while idealistic, is sadly untrue.
posted by grumpybear69 at 12:23 PM on January 24 [8 favorites]


I have been a government contract "inventor", but not in software. I think agile makes sense for gov. contracts in that hopefully it convinces whoever holds the purse strings to approve smaller, iterative contracts. If we have a system that supports small projects that build on each other, I don't think it actually matters how the project is actually run (every sprint of agile development looks like a tiny waterfall to me). Unfortunately IME government contracting as a whole is designed for big-three mega-million multi-year contracting, not "give this company 30,000 for one engineer to work on this problem for one month, then we'll see where we are then." When I started in 2006 these kinds of tiny contracts were my company's bread and butter but for a variety of reasons it really started to dry up and suddenly every project had to go through a 1-year-long procurement process which government PMs aren't going to pursue for every little thing. The purpose of these involved bid processes were to weed out bad actors while getting the "best price" but it never seemed to work that way.
posted by muddgirl at 12:25 PM on January 24 [4 favorites]


Or management tries to collapse the schedule so that development has to start before the product's design and requirements are determined, or they wall off the discrete teams working on fragments of a product so that added time is necessary to get the pieces to work together properly.

It me.

I got tired of this shit and said so and refused to do work on things that didn't have clear requirements. Its one of the reasons I got fired.

Also I just want to point out that the homepage of The Agile Manifesto is a hazy stipple effected picture of a bunch of white dudes in biz-caz hanging out in front of a whiteboard amazed by their own brains.
posted by fluttering hellfire at 12:25 PM on January 24 [25 favorites]


Yeah, I'm gonna stop you right there.

Yeah, I'm gonna stop you right back. I can barely begin to reckon a count of the thousands of person-hours of effort that have been wasted writing design docs that could practically have been auto-generated from the source code, because someone believed that somehow having this information in a Word document made our process better.

My ability to document code and processes is exemplary, and I do not mind being egotistical in this one moment in saying so. Documenting the right amount, as opposed to drowning people in documentation, is a huge part of that.
posted by tocts at 12:26 PM on January 24 [8 favorites]


Enterprise ecommerce development is almost entirely "build me this thing that people have built a thousand times before but customize it to my very specific needs."

I've seen this before as well, and am currently of the opinion that more stakeholders should look at the available software, see what ALMOST fits their needs, and rather than customizing it, putting up with something that's good enough. At my current job we're about to re-commit a dumb mistake by breaking off a company's main software update branch, when last time it happened we ended up locked out of bug fixes and revisions for over a decade and are now..... around 9 releases obsolete. We're about to ask another vendor to change a bunch of stuff for us and put us in the same boat for all THEIR future bug fixes and revisions.

It would save a hell of a lot of time and money to revise internal processes than break a vendor's off-the-shelf product so it fits.
posted by tclark at 12:29 PM on January 24 [6 favorites]


klanawa you're not alone. I also build software for scientists. One thing I run into a lot is the users (stakeholders?) (scientists / researchers in my case) having a habit of SCREAMING when ever anything happens that might change their process or workflow. Understandable to an extent, but it's kind of disheartening when your users become REALLY attached to some really buggy FORTRAN library that some grad student wrote 16 years ago.

Also, the references to "The Plan" made me remember this thing i read in some music mag when I was a teenager. I thought it was funny then, but then I did Warped Tour for 2 months.
In the beginning was The Plan...

And then came the assumptions.
And the assumptions were without form.
And suddenly the The Plan was without substance.
And darkness was upon the face of the roadies.

And they spake amongst themselves, saying,
"The Plan is a crock of shit, and it stinketh."

And the crew chief went unto the stage manager and said,
"It is a pail of dung, and none may abide the odor thereof".

And the stage manager went unto the production manager, saying,
"It is a container of excrement, and its fragrance is very strong,
such that none may abide it."

And the production manager went unto the tour manager, saying,
"It is a vessel of fertilizer, and all who draw near to it
are overcome by its strength."

And the tour manager spake amongst the management
company staff, and they said one unto the other,
"It contains that which aids plant growth, and it is very strong."

And the management company staff then went to the
management company executive, saying unto him,
"It promotes growth, and it is very powerful."

And the management company executive
went unto the band and said unto them,
"This new Plan will promote the growth and vigor of the tour,
with powerful effects to be felt by the audience!"

And the band looked upon The Plan, and saw that it was good.

And The Plan became Policy.



And this is how shit happens.
Applicable to a big tour, software development, gosh, pretty much anything I guess. Not sure of the origins of this text - does anyone know?
posted by capnsue at 12:31 PM on January 24 [16 favorites]


What seanmpuckett describes above is pretty much exactly how the construction industry works, in my experience. Including the part about learning to not work with clients who are going to give you a bad time. It's extremely rare to actually fire a client mid-build (at least, it's literally never happened at the companies where I've worked, though I've heard of it happening at other companies) but avoiding selling them a contract in the first place is fairly common, and occasionally you'll have a client where you end up cancelling at some point in the design phase. There's also the soft no, where you quote a slightly absurd price on the basis that they'll probably tell you to go pound sand, but if they do bite the ensuing headache will at least be worth it.
posted by Anticipation Of A New Lover's Arrival, The at 12:36 PM on January 24 [4 favorites]


“Adopt a DevSecOps culture for software systems”

doubleplusgood!
posted by thelonius at 12:36 PM on January 24 [7 favorites]


The thing about that buggy 16-year-old FORTRAN library is that they've probably successfully published a whole bunch of papers that used it and so they already know their peer reviewers will give it a pass. Heck, they may have built an entire career out of being the person who made (or took credit for making, anyway) that crappy little bit of software which has since become the industry standard for their little corner of the research world, and scrapping it would mean basically torpedoing a big chunk of their professional reputation.

It's still pretty dumb, though.
posted by Anticipation Of A New Lover's Arrival, The at 12:50 PM on January 24 [1 favorite]


As someone who is going out of business because my "niche" is apparently "clients who want the world but will only pay less than they spend on the executive's coffee budget for their entire public-facing website" and is now "too old to get a real job" and who just spent three months of my own dime trying to implement a new package which was hideously "documented" and designed by idealists who never, ever had to deal with the way their software was used in the real world...bah. Do none of you have actual budgets you have to work towards? Is it just Teslas and McMansions and a new iPhone every six weeks all the way down?

(Sure, I'm bitter. Wish I knew what the secret was in hooking up with these sort of clients. Maybe I'll add agile buzz words to my pitches.)
posted by maxwelton at 12:55 PM on January 24 [1 favorite]


My ability to document code and processes is exemplary
tocts, I'm sure it is. However, please read again what I said about assumptions. Some Word document is not documentation. A proper README is. A proper dependency tree is the base minimum. And if I have to fire you, or you get hit by a bus, I need to be able to hand off the project to another dev, and if they have to spend 6 months reading your code line by line to figure out what you were trying to do, it's useless to me and I might as well have them start over from scratch (which I have had to do).
Also, actual documentation in code is great, provided it's consistent.
posted by daq at 1:01 PM on January 24 [6 favorites]


McMansions are normally built on spec. The developer buys the land, licenses a handful of plansets, builds a dozen or so houses according to the three designs that were licensed, and then tries to sell them. Custom building is where you have to make sure you're picking a client who will not be impossible to work with and who actually, like, has the money to pay you for the work (not a given). It's actually easier with McMansions, in some ways—you're taking a bit of a gamble that they will actually sell, but at least in my region that never seems to be a problem.
posted by Anticipation Of A New Lover's Arrival, The at 1:02 PM on January 24 [1 favorite]


@tclark Forking vendor software so that it is no longer eligible for upgrades or patches sounds awful. Mostly my clients want stuff implemented on an (admittedly very expensive) eCommerce platform that is built for customization. It is basically a combination of an API spec and core functionality. So it goes: client wants application to do X, please build it on LONG_RUNNING_PLAFTORM. Sometimes we introduce NEW_UNTESTED_TECHNOLOGY because one of the other vendors likes SHINY_NEW_THINGS.

Back when we had great waterfall projects I had been basically building a framework (based on Hibernate, Spring and JBoss) in iterations across lots of projects, so with each successive project the unknown quantities kept decreasing and could increase overall functionality and scope without introducing too much risk. We also had an architecture council, BAs, project managers, program managers, stakeholders who cared about providing quality requirements, a proper budgeting process where we estimated resource costs and utilization over time, actual project plans, change control processes, etc. In retrospect it was almost utopian.
posted by grumpybear69 at 1:22 PM on January 24 [2 favorites]


Agile will not solve all your problems, and if you do it wrong, then it won't do anything for you, and can in fact become a distraction. Agile will not get your work done any faster. However, if performed properly by people who understand it, Agile really gives you two valuable things:

- The stakeholders will have a much better understanding of the current status of the project, and can plan accordingly.
- The developers will have a much better idea of what the customer actually wants, and deliver that instead of something they don't really want.

Both of these come from the short iteration cycle with plenty of communication, especially demos. Management doesn't waste its time trying to "plan" every second of the day with MS Project. Instead, once a sprint is done, they get to see what the product can do right now. Incidentally, if you're doing Agile right, then the product should be theoretically shippable at the end of every sprint; that doesn't mean it will have all the features that the customer requested, but it shouldn't have any half-completed features, and it should be able to do something useful.

Anyway, back to the demo...doing a demo every sprint (2 weeks is a good sprint length), the development team can (will) get immediately feedback, and can course-correct the next sprint if they have gone off-track. Right off the bat, this beats waterfall, where there is no guarantee that your stakeholders will truly experience the product until the whole thing is done, at which point it is far more expensive to make changes.

Agile doesn't really do much more than that. But that's enough. There are some other ingredients that are important to making things run smoothly, but the quick turnaround of sprint/demo is what makes it work. If you're not demoing at the end of every sprint to someone who represents the customer, then you're completely wasting your time. And if you're skipping your team retrospective after the demo, then you're skipping a great opportunity to learn and improve.

Many so-called Agile teams skip the retro and demo to themselves. Under those circumstances, it's unsurprising that a lot of organizations don't reap the advertised benefits of the approach.

By contrast, if you're doing waterfall, but you are taking steps to get customer-based feedback and honest soul-searching, you may not need Agile. Agile is really just a formal way to do just that.
posted by Edgewise at 1:23 PM on January 24 [9 favorites]


I get the idea that a lot of "Agile" shops jettison the core idea of frequent releases with a small, well-defined unit of functionality. Without that, it seems to me, it's just a culture of daily meetings and new jargon.
posted by thelonius at 1:28 PM on January 24 [12 favorites]


The concern in my soon to be former office was all about the appearance of performing Agile. We have teams at each others throats, releases are a train wreck, but every couple of weeks we game shit in Jira so reports we don't care about, viewed by people who don't care about us look good.
posted by fluttering hellfire at 1:30 PM on January 24 [4 favorites]


grumpybear69's waterfall process also looks recognizably familiar to me. It kind of sounds like some people know how to build software, anyway.
posted by Anticipation Of A New Lover's Arrival, The at 1:32 PM on January 24 [1 favorite]


We aren't doing it right, but management spent millions on embedding an Agile consulting company here for going on two years.
posted by fluttering hellfire at 1:34 PM on January 24


Anticipation: that water/house analogy? Yeah, that's basically it. I think the problem is that we have a language that can define the construction of a house completely (construction plans), but the only languages that define a software product completely are software languages. Everything else (diagrams, English) is too woolly. And so by the time you've defined it, you've built it.

That's why the many-small-steps of agile (every two weeks saying "did you mean this? how are we doing? what's next-highest priority?") is better than the giant leap of "first we'll design it, then we'll build it" of big-design-up-front - you have the same amount of risk, but you've chopped it up into smaller pieces, so if things start to go off the rails you know sooner.

Poor analogy: shouting instructions to a blindfolded man. "Go one step forward, go one step forward, go one step forward" sounds inefficient, but it's a lot less risky than "Go 1000 steps forward" because you can correct your instructions as you go.
posted by Leon at 1:37 PM on January 24 [4 favorites]


Wait, does software development not do change orders? In construction...

If you wanted to build a restaurant, you can sit down and plan this out very, very well. Real estate, permits, kitchen size, seats, bathrooms, etc. These are well known requirements and development steps. You could sit down and make a list of all the things you need, down to the number of forks and knives. You have much better odds of planning that construction project and having the desired outcome than a comparable software project.

Plus, construction is literal. Halfway through a project, if you tell a builder to move the kitchen to the other side of the building, they'll quickly point out all the ways that's more or less impossible. "But we already laid the gas line and it takes three weeks to get a permit..." But halfway through a software project, you're still sorta-kinda talking about the theoretical and ideas and methods.
posted by Cool Papa Bell at 1:39 PM on January 24 [1 favorite]


I have issues with agile but that doesn't mean that I'd ever want to go back to waterfall. I work as a tester/test automator and working in an agile/Scrum environment for the last five years has been so much better than my previous fifteen years slogging through waterfalls. I love that I test a feature and if it doesn't work like the acceptance criteria says it should, it either gets fixed or it doesn't get merged.

In my previous job which had 18 month release cycles, my job was basically just cataloging bugs that would never get fixed. There was never actually time in the cycle to fix bugs that we found during the testing phase so we just posted them to Jira and hoped that they might get fixed in the next 18 month cycle.
posted by octothorpe at 1:49 PM on January 24 [6 favorites]


I think there are some misconceptions about construction and how easy it is to plan everything. The change order often goes like this: Closely read plans etc. Bid low. Make it up in change orders. "Why did you put the toilet flange in the dining area of my new restaurant?" "That's what's on the plans, what am I a mind reader? Do you want a change?"

Its not as sleazy as it sounds.
posted by Pembquist at 2:13 PM on January 24 [2 favorites]


Agile is the worst possible development methodology with the sole exception of every other one.

I've been doing software development for about 20 years, and a competent team trumps whatever methodology you're using. Most companies are optimizing for the lowest-common-denominator staff, and end up with equivalent results.
posted by blue_beetle at 2:14 PM on January 24 [6 favorites]


Plus, construction is literal. Halfway through a project, if you tell a builder to move the kitchen to the other side of the building, they'll quickly point out all the ways that's more or less impossible.

THIS THIS THIS!!!!!!
posted by MikeKD at 2:20 PM on January 24


Ehhhhh, in my experience change orders are not where construction companies make their money. Possibly in commercial—I've mostly worked resi, so my experience may not generalize to something like a restaurant. But really, no construction company I've ever worked at has actually liked change orders. They disrupt the schedule, and they're not good for the customer's experience. Certainly one marks them up to a normal profit margin (usually anyway, sometimes there's wiggle room to mark them back down if it's something the company could reasonably have been expected to have caught earlier, or if it's a large unexpected cost that risks torpedoing the whole project) but there are usually negative externalities that are not fully captured in the quote price. Occasionally they can even scuttle an entire build, such as for instance when a hazardous situation is uncovered that the customer cannot afford to rectify. Really, we'd prefer to just have everything go according to the original plan.

But yeah, if we get halfway through building something and then you decide you want the toilet over here when the plans you signed off on clearly show it as going over there, that's totally on you and you're going to pay full freight for the change or else it's not happening. Not reading your contract or looking at your plans is no excuse.
posted by Anticipation Of A New Lover's Arrival, The at 2:25 PM on January 24 [3 favorites]


Not reading your contract or looking at your plans is no excuse.

Lol, it is for all of our clients.
posted by fluttering hellfire at 2:26 PM on January 24 [1 favorite]


Well then your management sucks, I'd say.
posted by Anticipation Of A New Lover's Arrival, The at 2:28 PM on January 24


Sorry, that was a little glib. Seriously though, having a plague of shitty managers is not the same as literally nobody knowing how to manage projects. At least two people in this thread have described software developers who have managed to find ways to spec a project, come up with a scope of work, get a contract signed, and then stick to the plan with only limited and well-controlled changes. Clearly it can be done, because it is done.
posted by Anticipation Of A New Lover's Arrival, The at 2:30 PM on January 24 [1 favorite]


I just want to point out here that if I popped into a thread about construction to tell a bunch of practicing construction workers that they're doing it wrong and obviously they need to do things the way software engineers do, I wouldn't be suffered as gladly as this.
posted by tobascodagama at 2:31 PM on January 24 [10 favorites]


OK, OK. Point taken. Have at it, everyone.
posted by Anticipation Of A New Lover's Arrival, The at 2:35 PM on January 24 [2 favorites]


"I've been doing software development for about 20 years, and a competent team trumps whatever methodology you're using"

Only 20 years? Jeez.... 43 years and counting ;-)

But I so agree. Competent people left to Get Things Done will make their own processes and ways of working, and modify them to suit the project, business constraints, and competencies and preferences of the team. In other words, agility.

Agile methods are useless if blindly imposed. I remember with amusement being a contractor in a team, ending up guiding them through their first sprint... and being the only one on the team who wasn't a newly qualified Scrum Master. (I had been there for a couple of weeks doing some C# training, and stayed on to help)

And for anyone who doesn't know where waterfall came from, let me leave this here: https://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970.pdf

It's an amazingly good paper, all the more so because the guy figured so much out from first principles.
posted by 43rdAnd9th at 2:40 PM on January 24 [1 favorite]


"Why did you put the toilet flange in the dining area of my new restaurant?"
"That's what's on the plans, what am I a mind reader? Do you want a change?"


Emphasis mine. On the plans. You knew you needed a toilet flange somewhere. In theory, it's on a blueprint before someone starts swinging a hammer. In theory, I get it.

In software, there are many, many more instances of, "Well, before we started, we didn't even know we needed XYZ because it wasn't until end-user testing that we realized nobody could understand ABC."
posted by Cool Papa Bell at 2:44 PM on January 24 [3 favorites]


One more thing ... in construction, you're generally constructing for humans, and humans haven't changed all that much in 400,000 years. We breathe, we eat, we shit.

But every two years or so, the computer hardware changes.

You know, they've been making movies about the same way for 100 years. Actors, costumes, props.

Now imagine that for every movie you want to make, first you have to invent a new camera.
posted by Cool Papa Bell at 2:47 PM on January 24 [2 favorites]


Agile is about as close as we're getting to solving the problems laid out in the mythical man month, I think that's pretty damn cool.
posted by nikaspark at 2:53 PM on January 24 [1 favorite]


things that are obvious

I spent 20 minutes explaining the modulo operator to someone today. I didn't realize until 19 minutes in that they didn't know it was an operator. They were thinking that 9%2 worked something like 9e2, instead of like 9/2. It would not have occurred to me to document that - nor did it occur to the author of Learning Python the Hard Way, which is how we got into the conversation.
posted by clawsoon at 3:04 PM on January 24 [2 favorites]


nor did it occur to the author of Learning Python the Hard Way

I have not read that book, but the C community seems to think that guy is an idiot, based on his C book
posted by thelonius at 3:08 PM on January 24 [2 favorites]


Back to the linked document: I'm pretty sure I know a foolproof way to detect who's still doing the BS version of Agile after reading the document. It'll be the people who create a checklist of all the specific software listed on page 3 and insist that they're agile because they use all of those pieces of software.
posted by clawsoon at 3:09 PM on January 24 [5 favorites]


I did a house for a software engineer where he tried to get us to manage his design project the way he did software and it was a disaster. We spent so much time preparing to talk to him at what were supposed to be quick scrums twice a week, in a way that he didn’t either tell us was too vague (eg without updated renderings) or too similar to last week (because we had the right idea last week), that we didn’t have enough time to do the actual design.

Given how poor a fit that was I doubt trying to use design and construction methodology to manage software development would be much better.
posted by q*ben at 3:16 PM on January 24 [3 favorites]


> Seriously though, having a plague of shitty managers is not the same as literally nobody knowing how to manage projects

In 2014, Oregon sued Oracle (and vice versa) after they paid them $248 million to build a health insurance portal that did not work and was thrown away in the end. Among the postmortems:
Oregon's projects were also hampered by disagreements among state officials, as revealed by two independent reviews of the debacle—one of which was commissioned by Kitzhaber. Cover Oregon and state officials repeatedly changed their mind about the scope of the work they wanted, Oracle said
And Healthcare software in general is a great place to mine for billion dollar fuckups.The NHS fiasco also comes to mind, but I'm not sure there's any positive lessons. A more positive example is the healthcare.gov portal, which was rescued by the adhocteam, became the template for the US Digital Service's Playbook. That playbook includes a direct reference to agile and iterative development:
Build the service using agile and iterative practices

We should use an incremental, fast-paced style of software development to reduce the risk of failure. We want to get working software into users’ hands as early as possible to give the design and development team opportunities to adjust based on user feedback about the service. A critical capability is being able to automatically test and deploy the service so that new features can be added often and be put into production easily.
And now we have the DoD publications decrying the proliferation of agile management B.S., because people want to check off the agile box without the hard work of changing or improving.
posted by pwnguin at 3:16 PM on January 24 [6 favorites]


Anyone who treats Agile as a system rather than a philosophy is doing it wrong.

Where I work we talk about outcomes, never requirements. We don't start unless we can have a user to consult with from the very first day of the project. We keep our teams to a maximum of three people. We aim to have working software in production within the first day of starting. We aim to have software that achieves the outcome halfway through the project, and we spend the next half enhancing it. Any feature request that doesn't get us to the outcome is added to the backlog, and then we work with the customer to 80/20 the backlog in the second half of the project.
posted by awfurby at 3:20 PM on January 24 [2 favorites]


At its heart Agile is pretty simple and straightforward and can be loosely summarized as: "The work necessary to make a thing is best determined by the people involved in the making of it building a consensus regarding the steps needed to do it, and then adjusting as needed until the thing is done."

Well, when you put it like that it sounds entirely reasonable.

In my experience, the people that don't like Agile/Scrum really just want someone to tell them what to do. Or they want to tell you how they want to be told what to do.

I don't like "scrum" in quotation marks, applied by fiat to software developers to solve problems that actually exist outside the software development organization. I kind of prefer Kanban to Scrum on a personal level, but I'm totally on board with short, iterative cycles with plenty of opportunity for feedback and course correction, no matter what you call them.

At my last job, I'd say that the loose summary is a very accurate description of how the developers on my team worked. We were on a six week release schedule not because the developers didn't work quickly or iterate their work, but because regulatory compliance meant that developers were separated from QA, and QA was separated from Acceptance, and six weeks is how long the QA/Acceptance/Release cycle took. On the development team we were always working on the release after the one currently in QA. Our manager and a couple PMs groomed the backlog a bit, and eventually we had a working flow in Jira where tickets got tagged for their expected release and we could see a bit more of a roadmap than just "these are current; don't bother with anything else." But you'd work on a ticket and file it as complete, and six weeks later you'd be working on the next release when QA would email you a question about one of the tickets you'd worked on. Maybe you'd have to switch branches and fix a bug, and then switch back to your current work. But we mostly kept busy, because we could see the backlog and start getting two releases ahead of schedule if that wasn't just going to cause political problems (spoiler: it was probably going to cause political problems, so near the end of many releases we did kind of run out of stuff to do until the next release could be opened for development, but it wasn't because we weren't trying).

But then the company moved us to scrum because a different development team (not subject to the same regulatory compliance for … reasons) had been using it, and their releases were faster than ours, so it was assumed that we had to be the problem (we were not; QA capacity was). There were several problems with this plan, from the Scrum Master™ the company brought in as a contractor who spent months (about six, IIRC) failing to understand that regulatory compliance meant we couldn't do our own QA (seriously, we went around in circles with her on this for months), to the fact they shoved all of us along with devops and QA into one "agile" team with 17 people in it, to the fact that they insisted that dev and QA all needed to be on the same release every sprint. For the first half of every sprint, QA had nothing to do. For the second half, the developers had nothing to do. This is not how you do agile well. Eventually they finally realized how many hours were being wasted and QA moved a sprint behind dev, but they were STILL PART OF THE SAME 17-PERSON TEAM for some reason. And it turned out QA really struggled with two-week cycles, so what really happened was that half of QA was one sprint behind and part of QA was actually two sprints behind, but we were all in all the same standups.

Also it turned out that most of the "releases" of the team that had been on scrum before us were paper releases only. Most of what they'd completed didn't work. None of their code had actually shipped to production (even what little of it "did" work). And they probably should have been subject to the same compliance rules we were, but I guess they were getting away without it because of the whole "not in production" thing.

So, that. I hate that. You'd probably agree with me that it's not really a functional agile model, but down in the trenches I've found that sort of disfunction to be depressingly common.
posted by fedward at 3:31 PM on January 24 [8 favorites]


the traditional waterfall model is sometimes more like a wacky water ride waterfall with random jets of water shooting out from any direction and whirlpools of endless meetings swirling into a void.

And the urine content is just so, so high.
posted by rokusan at 3:44 PM on January 24 [4 favorites]


Seriously, this entire thread reminds me why I retired. I can feel my blood pressure rising, comment by comment.

I'm going back to the liquid kittens.
posted by rokusan at 3:51 PM on January 24 [12 favorites]


In my experience, the people that don't like Agile/Scrum really just want someone to tell them what to do.

Or they've had some terrible experiences with something that management insisted was Agile, and they are unwilling to go through the same frustrations again.

Documenting the right amount, as opposed to drowning people in documentation, is a huge part of that.

It is, it is - but most software leans strongly toward "not enough documentation" rather than "too much;" devs are prone to thinking that "obvious" to them is also obvious to:
1) Other devs on their team,
2) Devs on other teams, who have different software proficiencies,
3) Project managers,
4) Administrative managers (the ones who set budgets), and
5) End users.

Many devs are somewhat aware that "end users" need lots of things documented - but they may believe that any feature an end user doesn't see, like the raw code, doesn't need more than a couple of keywords because categories 1-4 will all understand it when they see it.
posted by ErisLordFreedom at 4:11 PM on January 24 [3 favorites]


I wasn't intending to compare software development to construction contracting, I was just chiming in on the nascent construction contracting thread submerged below the torrent of words about the topic at hand.

With regard to the Oracle Oregon debacle. As an outsider it resembles the more often than not arc of large public construction projects. There is usually a real deficiency in the ability of the school district or whatever to supervise the project due to a lack of relevant experience. There are not just a few holes down which money can disappear, there is a whole colander of holes. It is easy to blame the naivete of the client for all the cost over runs/lousy outcomes but I find it a little hard to do that since the contractor knows exactly what they are doing and is taking advantage. This is not down to the people who actually build the thing who, of course, just want the project to go well and smoothly both from a natural sense of the aesthetics of efficiency and because of pride.
posted by Pembquist at 5:05 PM on January 24 [1 favorite]


Thanks for posting the PDF; it makes a really good point about agile BS. Here’s a few tests I would add:
What’s the story of how your company became agile? (The idea should originate within the working teams rather than from management whenever possible. It’s fine that it needs management support but “the CTO said we were agile now” is a bad scene.)
How do you handle building technical runway for front end features? (Something that might not get prioritized but must be done to support the prioritized FE feature. They need to have a reasonable plan for this.)
What’s a dysfunction or issue that using agile has exposed in your org?
posted by CMcG at 5:33 PM on January 24 [4 favorites]


How timely, someone at work posted this video to me this morning, about working on IT projects in large orgs. It starts out amusing, then becomes hilarious, then becomes terrifyingly accurate. Masterful subtitles.

One thing I'm not crazy about with regards to the Agile approach is that it's essentially unfalsiable. Any time it doesn't work, people just say they weren't doing "true agile"; there's even an agile term for it, though damned if I can remember it.

Whilst this is often true on one level, I do have to wonder how many projects are "true agile", successful or no. Not many in my enterprise environment, that's for sure.
posted by smoke at 5:34 PM on January 24 [8 favorites]


I tend to think that anyone that really needs a whole workflow methodology spelled out for them is probably insufficiently thoughtful so as to be able to create a good end product in the first place, and that's probably why you see some of the most dogmatic Agile boosters producing some of the worst crap you've ever laid eyes on. The basic principle, that specifications are extremely unlikely to be correct up front due to the nature of software, and so smaller delivery milestones and the resulting ability to adapt to the changing spec makes for better results, is a valuable notion that probably all of those productive teams mentioned above have hit on intuitively.

The one actual practical recommendation from scrum or whatever that I think is valuable is the notion of velocity estimation. I've been on teams that have really worked at making good time estimates, and although the beginning period is always hard, in all of those cases we've actually arrived at a pretty good shared understanding of how long things take and a surprisingly consistent ability to estimate the team's ability to finish a certain amount of work in a certain amount of time. Of course, you have to trust that the manager of such a team understands that the only metric that matters is total team velocity and isn't the sort of overbearing taskmaster that thinks it's a good idea to ride an individual developer's ass for not getting their 20 units of work done in a sprint, and that's unfortunately kind of rare, but when it works it works well, IMO. You also have to trust that the people above your manager understand that good effort estimation is a process of developing shared expectations based on data, and that they can't just conjure deadlines out of thin air and expect them to have any bearing on reality. Sadly, that's even more rare.
posted by invitapriore at 6:04 PM on January 24


> The one actual practical recommendation from scrum or whatever that I think is valuable is the notion of velocity estimation

You can actually get away without it and just count cards, if you're good enough at writing cards.
posted by Leon at 6:09 PM on January 24


Ugh. Agile. try getting two contracts on one gov't contract onto the same 'release train'.
posted by 922257033c4a0f3cecdbd819a46d626999d1af4a at 6:37 PM on January 24


I did a house for a software engineer where he tried to get us to manage his design project the way he did software and it was a disaster.

As a software engineer who appreciates Agile, I have to say that this guy's hubris is astonishing. I would have expected a disaster, so imagine my shock.

Given how poor a fit that was I doubt trying to use design and construction methodology to manage software development would be much better.

I wouldn't use this incident as a window into software development. I suspect that your customer also wrote poor software, because he clearly doesn't understand his own process. It unavoidably takes time to get a team up to speed in Agile.
posted by Edgewise at 6:46 PM on January 24 [5 favorites]


One thing I'm not crazy about with regards to the Agile approach is that it's essentially unfalsiable. Any time it doesn't work, people just say they weren't doing "true agile"; there's even an agile term for it, though damned if I can remember it.

That's a very understandable suspicion, and even as someone who appreciates Agile, I've seen it done wrong more often than right. There are usually two reasons for a failed adoption of Agile: the development team doesn't really get the point so they just go through the motions, or they fail to get management buy-in.

One possible issue with Agile is that it increases the communication between team members, as well as with stakeholders and customers. If your company's communication is dysfunctional, more of it is not necessarily going to help anything. Trust is an absolute requirement. If management doesn't trust development, which is often the case, then Agile won't help you.

In short, there really are a lot of ways that Agile can go wrong. So maybe your suspicion is justified. All I can really say to dispute it is that I have seen situations where Agile makes things better. But you can't just sprinkle a little Agile on your company and expect much to change. It's a good idea to actually train critical team members, like the product owner and the scrum master, and not just hire a consultant. It's a big commitment to make it work, and if your process isn't broken, it might not be worth it.
posted by Edgewise at 7:04 PM on January 24 [6 favorites]


One thing I'm not crazy about with regards to the Agile approach is that it's essentially unfalsiable. Any time it doesn't work, people just say they weren't doing "true agile"; there's even an agile term for it, though damned if I can remember it.

A term I've heard used is "scrumbut" -- e.g. "we use scrum, but ... [exceptions]", where the exceptions often make as much sense as if someone said, "I'm a vegetarian, but I eat chicken, beef, and pork".

That said, I get your frustration. It can seem like any failure is waved away with "they did it wrong". All I know is, I've basically never seen waterfall work despite working for years at companies using it, while the only times I've seen agile fail is when someone (usually management, the higher up the more likely) sabotages it with "... but we still need to [insert a bunch of waterfall stuff that will 100% prevent agile from functioning]". It's never been a mystery what the problem was, and it didn't require a complex explanation from an evangelist. Everyone could see how it was going to fuck things up from the start, and lo and behold, it did.
posted by tocts at 7:10 PM on January 24 [3 favorites]


It seems to me like a company with good trust between management and engineers, and great communication among all stakeholders, is going to do well no matter what their design philosophy.
posted by muddgirl at 7:15 PM on January 24 [4 favorites]


why over-documentation is better than under-documentation: if I, a new programmer coming into your project trying to learn everything from step zero, encounter huge swathes of documentation that is extraneous and useless, I, a new programmer coming in to learn from step zero, can fairly rapidly assess what things can safely be pruned. because if it's that obviously unnecessary, it will be obvious to me, the newbie ("why is he telling me this"). (if it turns out i was wrong and erased something essential, well, luckily there's a copy saved in git, right? recovering documentation that was written once is a lot easier than recovering documentation that existed in the programmer's brain before he forgot it)

Whereas if I encounter giant gaping holes missing, great, I get to spend four weeks painstakingly reading every line of code and trying things to make it work, in an effort to plug some of the holes. That doesn't mean that obviously asinine things like "export it all to a word doc" aren't asinine. I agree that documentation external to the code itself should be external interfaces and examples. But at some point someone is going to need to go into your actual code to change some thing or to figure out why something that should be working isn't.

(I also really agree with the philosophy that code should whenever possible document itself, especially with helpful descriptive variable names (ah, glorious "temp", a vector that was actually a crucial linchpin of the entire calculation across multiple steps AND part of the final results))

But honestly I've never in my life encountered something over-documented to the extent that it was even a fraction of the frustration and difficulty that the constant under-documentation I encounter causes me. Skipping/skimming over documentation that is there, is a hell of a lot easier than extrapolating documentation that isn't.

Plus of course documenting is a favor to yourself, two years later (if you're staying at the same project that long, and if not, please document as a mercy to whoever your replacement will be). So many times past me has saved my own bacon in that fashion. Contrary to how I feel right now when everything is so glaringly obvious that writing it down seems painfully redundant, two years is enough to erase my memory of everything but the barest, vaguest sensation I once knew what this code did and why. When I find that past me took care of present me in such a considerate way by documenting everything anyway I feel a warm glow.

(I suspect the best way to do documentation is write extensive documentation 1-2 weeks after writing the code you're documenting, and then review it six months later when you've had plenty of time to forget but your brain hasn't done a total reset)
posted by Cozybee at 8:30 PM on January 24 [6 favorites]


sure, I can email you, the original developer, to ask for help filling the holes, assuming I can track you down and you answer emails.

sadly you, the original developer, will have already forgotten.

it will only take you two weeks rather than four to re-familiarize yourself enough to answer the question, (if you're even willing to do that now that you've moved on to shinier things).

but we'd all be happier if you'd just documented what the damn function did to begin with. and not vaguely. explicitly. and if your function actually did five things, not one, ok, that's bad code design, but I need to know in the documentation that it does 5 things, not just the "main" thing it does. and why did you do that inscrutable thing over here? etc.
posted by Cozybee at 8:41 PM on January 24 [2 favorites]


I’m gonna start telling people that where I work our agile method is called “the aristocrats”
posted by nikaspark at 9:43 PM on January 24 [5 favorites]


I especially love to start a project doing something with COTS/ERP only to find out after starting that the client has more than 1000 undocumented/poorly documented customizations so that the software does not behave at all the way I expect and therefore won't play well with the new customization the client wants.
posted by Altomentis at 11:37 PM on January 24 [1 favorite]


A huge amount of software development is about building things nobody has ever built before

Hah!

A huge amount of software development is building things everybody has done before, and googling stackoverflow when you get stuck.
posted by MartinWisse at 2:13 AM on January 25 [15 favorites]


And then smashing your head against the desk when you do exactly the thing in the docs for the open source library you're using, which is also exactly what the Stackoverflow posts all say to do, and yet it's still not working. So you go over to GitHub and find out that there's an outstanding issue for the thing that's not working for you that hasn't been touched in six months despite the fact that comments from people also experiencing the issue keep piling up and there are at least two competing pull requests that claim to fix the issue.

It's a good idea to actually train critical team members, like the product owner and the scrum master, and not just hire a consultant.

Anecdotally, I've never once heard a story involving an external scrum consultant with a happy ending. When I've seen agile work (and I have), it's been because we had tech leads with prior agile experience and/or training driving the implementation while doing work on the product itself. Your managers should be doing agile training as well, but they shouldn't be driving how the processes are implemented.
posted by tobascodagama at 5:34 AM on January 25 [2 favorites]


It'll be the people who create a checklist of all the specific software listed on page 3 and insist that they're agile because they use all of those pieces of software.

So much this. It wouldn't be a bad list, if they just omitted the names of specific software and kept the categories.

Sigh - that is one of my biggest beefs - a good process has to be software agnostic. It needs to be nimble enough to handle changes in available tooling.
posted by jkaczor at 5:45 AM on January 25 [2 favorites]


I've been amazed at how good a team can be at pointing stories. We don't nail the velocity for every sprint but averaged out against half a dozen two-week sprints and we come pretty close.
posted by octothorpe at 6:22 AM on January 25


I did a house for a software engineer where he tried to get us to manage his design project the way he did software and it was a disaster.

I did a house remodel using agile techniques. Agile doesn't mean that *everything* is variable and no plans are ever made, it's divides work into 'deliverable chunks' and that's easy to do in a construction project. If you are doing an Agile remodel, you have to understand and prioritize (Ie: removing the wall is a priority for a kitchen, not stone countertops). As the project moves along and a 1970s electrical box can't pass code for a kitchen, the funding that wasn't pre-allocated on those stone countertops can be allocated on updating the electrical box instead, which means the budget stays relatively fixed. Agile fixates on the delivery date and meeting the budget.

In a 'waterfall' remodel project, the stone countertops would be purchased early because everything is moving towards a fixed delivery date with fixed functionality. So finding late that the electrical box needs upgrades would be a change order CR for the budget increasing total project cost.
posted by The_Vegetables at 6:59 AM on January 25 [1 favorite]


no plans are ever made, it's divides work into 'deliverable chunks' and that's easy to do in a construction project.

Just riffing off the cuff here - but I *think*, a key difference between construction and software development is that in construction the actual work being done by tradespeople is not micromanaged by PM's/leads who are not doing the work. They are told... The framing has to be done by "X" date, because electrical needs to be done for "Y".

The tradespeople are treated like professionals who have earned their expertise - and oversight is provided independently, via inspections by third-party agencies. That is another key difference, I rarely see any organizations who do software inspections post-mortem, at stage-gates - sure automated unit testing/test driven development helps, but it only proves that code written is functioning, not that it is fit for purpose or complies to any level of quality or standards.

While I haven't been involved in home construction (I have tradespeople in my extended family though) - I am guessing plans and blueprints do not describe every last detail of how/where electrical outlets are placed (regional codes would apply), nor standard things like distance between framing studs, nor how to put up drywall/sheetrock, nor how to paint or apply wallpaper.

So yes - while there is a master blueprint, every last detail is not specified. Meanwhile, over in software development la-la land PM's want microscopic levels of control, traditionally the waterfall method does try and do that.

In my experience, you need something "in-between" when working on bespoke projects where the requirements are mandated by "users" - high-level blueprints and designs (so everyone knows the overarching vision), but... at the ground level - things change, platforms differ, technologies "rev" faster than in other industries (or do not work the way documented by their authors) - so, techniques used to build something two years ago, won't be the ones you use now. (One example is how quickly the JavaScript ecosystem goes through frameworks, tooling, paradigms, etc. Personally, I think that has only truly stabilized in the last 1.5 years to a common set.)

However, there are best practices, "bodies of knowledge" and governance guidance that let your strategic team members implement without being dictated to the "n'th degree". OTOH, when you are building products/offerings, you are implementing your distinct vision - the marketplace will ultimately decide if you picked the right user stories and scenarios - and popularity will determine if your overall blueprint (architecture) can scale over the course of your products viable lifetime.
posted by jkaczor at 9:24 AM on January 25 [1 favorite]


And of course as soon as I call somebody out for relying too heavily on construction analogies that don't directly apply, the whole damned thread pivots to house construction.
posted by tobascodagama at 9:53 AM on January 25 [3 favorites]


I've been in software development for 22 years and went through the whole change from waterfall to Agile. I've been a programmer and manager on both sides, an Agile coach and trainer, a SAFE Release Train Engineer, and currently manage the DevOps tooling group for a very large company.

My observations:

* Agile as stated in the Agile Manifesto is pretty unarguably good. It emphasizes human interaction, "show don't tell", and autonomy and responsibility for people on teams.
* Agile is just a general umbrella. A company can't just "do Agile" so if you hear that, it's often a red flag. Do you do Scrum? Kanban? Do you use XP methods? Modern DevOps tools and processes? Whaddaya mean, "agile"?
* The number of people doing Agile training and especially acting in pure Agile roles is much larger than the number of people who know what they are doing. The Agile Manifesto was not a money-making play *at all*. It was a bunch of experienced professionals getting together and saying "The way we do things now sucks and it's stupid. Teams hate it, customers hate it ... what are the elements of a good way of writing software?"
* A local web development firm with 5 people in an office may think that a lot of Agile methods are obvious - they're right. But saying no one needs help in this area is like saying "well, I go to a bunch of triathlons and I'm super healthy - why do people need dieticians and physical trainers? It's obvious!"
* Agile done well is really, really good at exposing problems that are under the surface, and then getting blamed for them. It can't magically fix command and control project managers, team members who shouldn't have been hired, stupid contracts, impossible schedules, etc. But the whole idea is that you THINK AND TALK ABOUT the problems with your process and interactions and work on making things better bit by bit.
* People hopping on the Agile bandwagon and trying to coopt it usually destroy the simplicity. See: SAFe and other scaling frameworks, the whole PMP Institute agile mess, etc.
* There's about a zero percent chance that, if I sign a contract for a 2 year project with you and build it perfectly to spec, you will be happy with the outcome and not bitch about it. Things change. Needs change. Far better to use development methods that preserve options as long as possible and just assume things will change. Example: "The application shall make use of DES encryption for all communications." Whoops! DES got cracked. But the contract!

I'm glad at least a few people on the thread have had good experiences, because that's what I care about. It really is the best expect for all the others. The corporate world is such a mess that it's a miracle that good ideas like Agile survive at all. In closing: Last year I was in a conference session with one of the most respected and experienced agile coaches in the world and she had very astute answers to a couple of questions. Q: How long do you thing Agile will be around? A: Honestly I didn't expect it to last this long. It's a testament to how sound the basic ideas are. Q: How can I make Agile work in my large company when I can't get executive buy-in? A (paraphrasing): Maybe it's time we spent less effort trying to help big companies that are trying to die.
posted by freecellwizard at 10:14 AM on January 25 [6 favorites]


heh... well, admittedly - I use construction analogies quite heavily when I am giving presentations - but I typically caveat them with similarities to different techniques... You don't build skyscrapers the same way you slap up tract housing. When you slap up tract housing, you use as many pre-built components as possible (think pre-built roof trusses). Yet planning a subdivision takes infrastructure awareness.

They are analogies - should be a quick, common reference to help people outside of software development get to a common understanding.

Back-in-the-day, when I was implementing a commercial "Enterprise" product within a reasonably large organization (130,000 people), I had to come up with a decent analogy for why our implementation team was NOT going to customize every single "site" that we were creating for each new client (team/group/department)...

The best I could come up with, was...

"Think of our staff as citizens of a former Soviet-bloc country, who have just come out of the darkness. We are tasked with making their lives measurably better, and kick starting the economy. The soviets had built us some great road infrastructure, but our people have been left so poor, they have no cars. Now - we are bootstrapping the population - every family gets one free car - there is no choice, we only make one model and we do not tailor it. Our cars have only 2-doors, a shit engine and max out at 60kph, but ... they are free"...

..."Oh, and they only come in grey - we don't care whatever color or how you want to paint it - get a can of paint and even use a brush if don't have a roller - go nuts"...

Yeah, I offended more than a few executive stakeholders with that one... But it was apt...
posted by jkaczor at 10:16 AM on January 25


One possible issue with Agile is that it increases the communication between team members

I was reading an old Joel on Software post the other day. High levels of communication between team members made me think of this bit:
The trouble is, getting into “the zone” is not easy. When you try to measure it, it looks like it takes an average of 15 minutes to start working at maximum productivity. Sometimes, if you’re tired or have already done a lot of creative work that day, you just can’t get into the zone and you spend the rest of your work day fiddling around, reading the web, playing Tetris.

The other trouble is that it’s so easy to get knocked out of the zone. Noise, phone calls, going out for lunch, having to drive 5 minutes to Starbucks for coffee, and interruptions by coworkers — especially interruptions by coworkers — all knock you out of the zone. If a coworker asks you a question, causing a 1 minute interruption, but this knocks you out of the zone badly enough that it takes you half an hour to get productive again, your overall productivity is in serious trouble.
I wonder if people who are productive in high-communication environments find it easier than the rest of us to switch back into the zone when they're interrupted.
posted by clawsoon at 12:03 PM on January 25 [5 favorites]


The real question is whether being "in the zone" but building the wrong thing is better or worse than building the right thing while frequently being knocked out of the zone by communication needs.

Although specific agile implementations like XP (and, in theory if not always in practice, Scrum) put heavy emphasis on including a "definition of done" with your user stories to minimise the amount of design-related communication that is required outside of backlog grooming and daily standup meetings.
posted by tobascodagama at 12:11 PM on January 25 [1 favorite]


R.e. "the zone", another bad practice I see that often goes along with "agile" is the adoption of large open floor plan workspaces with little or no privacy. Pretty much no one likes these, but they are much cheaper than offices or team rooms. If you are a team of 7 people in a physical area, you can develop team habits about quiet time/question time/fun time etc. But if you're in a huge space with 200 people on different teams with different needs and schedules, it's a mess. Ideal for me would be a walled-off area with a couple of team rooms, a shared "quiet room" for people to work when deep thinking, and a public-facing area for standups, meetings with people outside the team, etc. Sadly I don't see open spaces going away due to cost. This is why I'm wearing a $300 set of noise canceling headphones right now.

Also, YMMV but I've done a lot of agile stuff with distributed teams and I have this to say: Distributed teams are pretty much bullshit. There are some hacks, but to actually work as a team, nothing beats sharing a physical space for at least part of the time.
posted by freecellwizard at 12:14 PM on January 25 [3 favorites]


Not gonna try and drag the whole thread back into construction but I'd like to respond a little to The_Vegetables' comment about using Agile to do a renovation. If anything, I think this actually supports people's point that maybe construction and software development are just fundamentally different in terms of management best practices, but maybe some of this will resonate with you developers, who knows.

If I had a customer who wanted to take something away from their project every time something new got added in, I would scream. What you are doing there is at least doubling the administrative load of every change order, probably more, because not only do I now need to process two unrelated change orders—e.g. electrical upgrades to solve pre-existing code violations, and a credit for downgrading the countertop finish—I also need to help you find a place in the project where there is fat that can be cut, and then if we're talking something like countertops I also need to walk you through the process of re-selecting the new finish. And because you want to make it a net no-cost change, I am not even making any money from it—I am doing all that administrative work for free.

Maybe that's not going to mess with your timeline, but probably it is because for finish selections in particular I can tell you that it is critical to get them out of the way as early as possible. Customers routinely procrastinate on that stuff up to or even beyond the last minute, and I have certainly seen projects come to a dead stop because we were waiting for the counters to be templated, fabricated, and installed. As the customer, you are not expected to know in advance how long that process takes. As the builder, it is my responsibility to know that and make sure it gets done in a timely fashion—but not too early because that can be disruptive as well.

Basically, by the time your cabinets go in you better have those counter selections locked down because I'm going to get the countertop guys out there the next day (I've probably already had them scheduled for a week, since they aren't necessarily available on short notice) to template them, and then there's a two week fabrication period before they can go in, during which time we cannot do finish plumbing, finish electrical, paint, or trim. Finding things to do to keep the project rolling during that time is already a challenge, so a delay in countertop selection probably means a delay in the whole project, and that's not good for anyone because I'm not producing (which is how I make money—project delays cost me money) and the customer isn't getting their kitchen up and running in time for Thanksgiving.

Basically, if you are trying to do a renovation project where you only have exactly enough money for the project as proposed and not a dollar more, you shouldn't be doing it. You should have a 10% contingency fund to cover unknown factors like an electrical compliance issue that's discovered during the rough electrical phase. No honest contractor will ever tell you that a renovation is going to go 100% according to plan and that you will definitely not end up having to pay for things that nobody foresaw—that's the nature of renovation, and of all construction for that matter. If you are buying a renovation, you should always have some money in reserve for contingencies, because there are always going to be contingencies.

If you start nickel-and-diming me and picking apart your project over having to fix things like code compliance issues, our relationship will quickly sour. If you tell me you want to do a kitchen reno but you don't want to pick your finishes until the last minute because you're not sure if you'll have enough money for them, I'm going to tell you to come back and talk to me when you can afford to do the project. I'm not saying that one never has to cut something out—sometimes really big unforseen issues crop up, or sometimes the customer wants to do a significant design downgrade partway in (generally doable, although you'd have saved more money if you hadn't waited until the project was already underway to tell me that) and you just have to cut some fat and issue a credit. It happens. But if you go in with the attitude that any unforeseen costs will be paid for by cutting things from the design, well, that's just not a recipe for a smooth, efficient, cost-effective project. That's a recipe for pissing off your contractor.

I will leave you all to draw your own analogies now, if you wish to do so.
posted by Anticipation Of A New Lover's Arrival, The at 12:48 PM on January 25 [5 favorites]


You should have a 10% contingency fund to cover unknown factors like an electrical compliance issue that's discovered during the rough electrical phase.

You are completely describing waterfall delivery, where time to market and pre-determined features are the most important metric, more important than total budget. Software projects have really tight ROIs managed by financial divisions, not housewives who watch a lot of HGTV (just kidding), so staying on budget is more important than pre-determined features.
posted by The_Vegetables at 1:40 PM on January 25


HGTV-watching customers (not just housewives, what the hell kind of joke is that) are actually the literal worst. They want the world, and they want it yesterday, for free, with no mess. Because that's how it works on TV.

Anyway, what I was attempting to describe was why Agile is a bad methodology for construction, not why Waterfall is a good one for software development.
posted by Anticipation Of A New Lover's Arrival, The at 1:54 PM on January 25 [1 favorite]


You are completely describing waterfall delivery, where time to market and pre-determined features are the most important metric

Agile does make a lot more sense if you define it as the admission that completing all the functionality the customer wanted in any sort of predictable time frame is beyond any known development practice, so looking two weeks ahead is the best we can do.
posted by grahamparks at 2:14 PM on January 25 [4 favorites]


But I still want to know where you guys get these customers? Every customer I've ever had--on admittedly pretty small projects, would never sign off on a "well, just start working and bill me as we go, and maybe it will eventually be done" project. They have budgets, period, of a fixed amount of $ and time, and an end-goal of what they want the software to do. You guys all seem to be talking about projects where there is unlimited money until things are "done". Where do I get on that train?
posted by maxwelton at 2:29 PM on January 25 [4 favorites]


It does kind of sound like what we in construction refer to as T&M, i.e. time & materials. In construction, T&M is really best avoided for anything but the smallest projects and change orders (like two days' work or less) because it gets out of hand super easily and leads to some seriously negative customer experiences.
posted by Anticipation Of A New Lover's Arrival, The at 3:36 PM on January 25


Every customer I've ever had--on admittedly pretty small projects, would never sign off on a "well, just start working and bill me as we go, and maybe it will eventually be done" project.

The pushback against that is, "if you demand we stick to the original plan, you get only the original requested features. If Google changes their API in the meantime and integrating with it would now break your app - well, that's what's in the contract so that's what we're delivering. If, however, you want software that does what you want it to do rather than matches a pre-existing list of traits, we can be flexible. Flexibility costs, but you get more than just software that works - you get software that you understand, software that will be easy to change later."

Agile was developed for software services that are dealing with customers who had time and budget to allow for flexibility, but thought that software should work like ordering a meal from a "check one from column A, 2 from column B" menu. For smaller projects, where the goal is understood by everyone at the beginning and the design specs actually cover the final goal, it's not needed.

For creating new enterprise software, or a video game with a two-year dev plan, you want something a lot more adjustable than "here's the initial list of specs and a here's the dollar cap... go." Agile works in two-week cycles and can be adjusted to a bit slower than that; it wasn't designed for projects that last less than a few months. (It can work for them but it wasn't invented because of a large collection of botched 4-12 week projects.)
posted by ErisLordFreedom at 4:11 PM on January 25 [3 favorites]


To bring in an analogy other than construction: In computer animation, costs and timelines tend to go out of control when:
  • The client doesn't know what they do want until they see what they don't want.
  • The contract is set up in a way that allows the client to push for more and more revisions and improvements without it costing the client any extra money.
  • The company takes on work that they haven't done before, and discover along the way that it's more complicated and expensive than they expected.
In other words... uh... sounds similar to both construction and software development, at least from a high level.
posted by clawsoon at 4:11 PM on January 25 [4 favorites]


The company takes on work that they haven't done before, and discover along the way that it's more complicated and expensive than they expected.

Considering that technology is not static, this is very typically the case anyways/always.

Hmmm - actually, the ones that I find really stick to their (almost always "waterfall") estimates/timelines and workplans are those working in a moribund platform or ecosystem, where nothing has changed for 5 years or more. (And, well ... while those people may know the problem domain very well, they are generally not the "leading lights of the technology sector" - which is not necessarily bad - for larger projects, there are many roles to fill - not everyone can be a "rockstar developer")
posted by jkaczor at 6:09 PM on January 25 [1 favorite]


Maybe it would be better to stick with a single toolset long enough to actually get good at it, rather than having to re-learn basically all the mechanics of your job every eighteen months? That doesn't sound very efficient. Why does the software industry have such rapid cycling?

My own industry can sometimes be too conservative about adopting new tools and techniques, but there are real benefits in terms of predictability and efficiency to sticking with something you already know how to do and just trying to get incrementally better at that thing over time, rather than adopting every new technology that promises to revolutionize your productivity. Most of the time they are fads at best, gimmicks at worst.

I realize that all of computing moves super fast and that you have to keep up, but what would it look like if software devs as a whole industry all decided to focus more on adapting the skills they already have rather than adopting new skillsets wholesale?
posted by Anticipation Of A New Lover's Arrival, The at 6:26 PM on January 25 [2 favorites]


Maybe it would be better to stick with a single toolset long enough to actually get good at it, rather than having to re-learn basically all the mechanics of your job every eighteen months?

The software and hardware changes every 18 months. Every three years, you'd be working with horrifically out-of-date tech. There's occasionally a concept of "hey, if everyone just slowed down, we'd be much better with the tech we have, and have fewer corporate implosions and drastically horrible live products." But of course, that involves everyone slowing down, and new startups have no interest in that.

There's currently no shortage at all of coders who say, "here's a tiny niche that nobody's covering yet - let's specialize in it, get VC money to get started, and become the industry powerhouse in Tiny Niche X." And then everyone's scrambling to either set up their own Niche X team, or fears being squeezed out of the market entirely if Niche X turns out to be the next Salesforce.com.

(I work in doc imaging. I sometimes think I'm the only one on the planet who misses Acrobat Capture Reviewer.)
posted by ErisLordFreedom at 6:36 PM on January 25 [3 favorites]


So what I'm hearing is that yes, slower cycling might be a net improvement, but that because of market forces everyone is trapped in a perpetual Red Queen scenario, where you all have to run as fast as you can to stay in the same place?
posted by Anticipation Of A New Lover's Arrival, The at 6:39 PM on January 25 [4 favorites]


In software development, yes. And it's not just corporate greed doing that - part of it is the huge number of amateur coders making apps and macros for fun, and posting them where anyone can use them, and more coders grabbing bits of code and making new apps, and so on.

The options and available pool of information increase exponentially. A company can skip them by focusing on its one expert niche, but it can't branch into new areas without diving into the seething mass of Random New Software Stuff. And if they want to be ready to branch into new areas, they have to keep a connection to what's going on, so they don't miss out on "yeah, that was trendy a year ago, but then someone found these flaws that make it nonviable in the long run."

And if a software company wants to build programs, apps, games, whatever for new clients - they need to keep on top of what's current, what works with today's hardware and OS options (okay, that one you can blame on corporate greed; take it up with Microsoft and Apple), and what's currently getting media attention that might boost their visibility without having to pay extra for advertising.
posted by ErisLordFreedom at 6:49 PM on January 25 [2 favorites]


But I still want to know where you guys get these customers?

It depends on what you're making and the scale of your company, but for most development projects you don't have one single customer who contracted you for a single piece of work. Rather, you're building a product that you hope to sell to multiple customers, many of whom don't even necessarily know who you are before you put it on the market.

So most of the time you're working with somebody serving as a proxy for your customers. In Scrum and XP terminology, this is the "product owner" role. (In practice, this is often someone with an official "Product Manager" title, though I'm pretty sure the originators of Scrum and XP envisioned it being done by "staff"-level engineers.) Their job is to interact with your actual customers, if you have any, or to do research on what your customers will want when you have some, and then turn the results of that into user stories that represent discrete pieces of functionality that developers can deliver on.

So ultimately what I'm saying is that I think there might be the important missing piece of context here for those who haven't worked in a (functional) agile environment before. You're usually interacting on a daily/sprint-ly basis with someone who represents your customers as a class rather than directly interacting with one individual customer who contracted you to make a bespoke piece of software. Because you don't have one individual customer, you have hundreds or thousands or even millions of them. That's the context that agile arose from.

In that context, the importance of the "user stories" aspect of breaking things into the smallest possible discrete pieces of deliverable functionality becomes more obvious. Because if you have something that you can work on in two weeks and then do a demo of instead of something that you need to wait six weeks to integrate into the next milestone, your product owner can go and take that demo and show it to your actual customers in some form or fashion, whether that's directly reproducing/replaying the demo for them (if you only have a few) or running a small-scale A/B test with a focus group. Or, if you're a DevOps shop, you just push it live as see what happens. Whatever the exact method, you can get feedback to test the product owner's understanding of the user story almost as soon as the work is complete, even though it may not show up in an official release for months.

Now, you certainly can use agile processes in a contract-work environment (and that probably represents a lot of what the DoD is doing that inspired). But understanding how agile came to be and why it's seen as such an improvement over what came before is easier when you understand the environment it came out of, which is a product-driven world rather than a contract-driven world.

(And if you're working in an environment where the idea of delivering discrete pieces of functionality makes no sense? Yup, most of the popular agile methodologies are really bad at that! There are very few software projects where that's actually true, though, and most of those involve hardware integration.)
posted by tobascodagama at 6:50 PM on January 25 [4 favorites]


(Although here's where I point out that kanban as a formal process originated with Toyota's physical manufacturing, which you'd think wouldn't be compatible with agile software development based on all that crap I just said. But it makes sense when you focus on the "discrete pieces of functionality". In a physical process, you'd have "mount the drive train to the chassis" rather than "as a user, I want my comment to appear on the site when I click 'post'", but either one of those represents a discrete piece of work with a defined end point.)
posted by tobascodagama at 6:58 PM on January 25


Every customer I've ever had--on admittedly pretty small projects, would never sign off on a "well, just start working and bill me as we go, and maybe it will eventually be done" project.

There are always unknown unknowns. So you can have cheap and risky, or expensive and less risky. I prefer the second, but if you want the first, that's cool too. Maybe you'll get something useful, maybe I'll make a pile of cash on change requests.
posted by Leon at 7:10 PM on January 25


> this is often someone with an official "Product Manager" title, though I'm pretty sure the originators of Scrum and XP envisioned it being done by "staff"-level engineers.

My understanding is that the ideal was that the customer would be a team member. It rarely worked out like that in real life though, hence the Product Owner role.
posted by Leon at 7:13 PM on January 25


> Maybe it would be better to stick with a single toolset long enough to actually get good at it, rather than having to re-learn basically all the mechanics of your job every eighteen months?

Serverless is next. That's gonna be fun. I'm hoping I can skip the docker generation entirely.
posted by Leon at 7:16 PM on January 25 [1 favorite]


"user stories"

I hate this vocabulary so much - it's somehow both corny and infantilizing
posted by thelonius at 8:08 PM on January 25 [4 favorites]


So what I'm hearing is that yes, slower cycling might be a net improvement, but that because of market forces everyone is trapped in a perpetual Red Queen scenario, where you all have to run as fast as you can to stay in the same place?

I don't think it's as simple as this. No doubt, there's a fair bit of churn because developers love few things more than inventing a new way to do something (please, spare me your frameworks!). That said, the rapid cycling isn't primarily because people just can't not use the latest thing. Generally, the newer stuff is better. It's faster to build stuff with, or cheaper to maintain, or can do things you couldn't do before (or couldn't do easily).

There are certainly software shops that take a slower approach, but it often leads to stagnation and eventual death as new competitors who don't have to deal with the bullshit the old tech required come along and out-perform them.
posted by tocts at 8:15 PM on January 25 [1 favorite]


"Software projects have really tight ROIs managed by financial divisions"

But this varies, the "fast cheap good pick any two" priority varies and critically the "what is still good" varies, and I'd be interested where software engineering theorists have gamed this out.

In my part of the world, you're roiling in ROI if the thing works, the problem is most things don't work and there is alway the question "how do you justify not just chasing the diminishing marginal return on that working thing since that diminishment starts pretty high up?" Other business absolutely have wafer-thin margins. Some developers update their web service every week, others bake their code into EEPROM in hardware to be sold for Christmas. Some developers ship a product you can change freely (surprise, new app UI!), others are locked in for a multi-year deprecation period.

Software development is change management: 1) what can you change, cost / time / product but it's critical to know which product changes are okay and which are not (from contracts, expectations, public stored values to your schema), and 2) what is going to push you to change how much and how fast -- incomplete understanding, change in the state of the world.

Theoretical pure waterfall development:
0) *** a miracle occurs ***
1) user provides a sufficient requirements doc to proceed with waterfall
posted by away for regrooving at 12:37 AM on January 26 [1 favorite]


Surely it is at least partly the contractor's responsibility, during the sales phase of the project,
to help the customer come up with a "sufficient requirements doc?"

I guess what bewilders me is that most of the insurmountable problems people identify with Waterfall are problems that would totally be issues over here in construction as well, except over in construction we've come up with time-tested solutions for them whereas in software dev people say that it would take a miracle.

For instance, we have sales people who are either technically adept or work with people who are technically adept, and who help our prospective customers come up with a workable plan that will meet their needs, cost it out, and set reasonable expectations for how the process is likely to go. We don't just throw our hands in the air and say that it's impossible. It would be impossible, if we let the customers try to spec their jobs all on their own. That's why we assist them.
posted by Anticipation Of A New Lover's Arrival, The at 8:42 AM on January 26 [2 favorites]


I think the fundamental misunderstanding remains that construction is not a good analogy.

For one thing, humans have been building structures for tens of thousands of years. They've been building software for less than 100 years. It is an incredibly new discipline, and is evolving at a rapid rate.

For another, software is not a physical artifact. If you build me some shelves, they will function in expected ways without any extra work on your part (weight limits and size notwithstanding). I don't have to worry that they will hold a book but not a vase. Software doesn't work like that. Every aspect of how it reacts to interaction is possible only because someone thought it through, and made it possible. How hard would gathering requirements for building a kitchen be if you had to know ahead of time every dish the client wants to cook in it?

This is not to say nobody has ever used waterfall; IBM and other big names did so from the beginning of computing. But, it's an approach that takes years of time and masses of cash and big, pre-agreed contracts, all of which are in short supply in most shops these days. A 3-5 year lead time to first customer use of a project is a suicide mission. The market will have moved on, and someone will have filled that niche.

Customers won't pay enough for software, in most cases, to allow waterfall to work (for certain definitions of "work"), so in most cases it's a total shit show.
posted by tocts at 9:19 AM on January 26 [2 favorites]


Our customers are really "customers". The role of the customer is mostly played by the product analyst who acts as a surrogate for the current and/or potential users of the system. I work for a billion dollar non-profit so they have money to burn for internal project development that may or may not someday be spun-off into an external project.
posted by octothorpe at 9:29 AM on January 26


It's just weird that you have people here saying, "Agile works best when customers are willing to spend an open-ended amount of money to work on something until it's done," and then also people saying, "The reason Waterfall doesn't work is that it's too expensive, nobody wants to pay what things cost." You have people saying, "Customers never know what they want and expecting them to would be like expecting a miracle," and then you have people saying, "What works best is to come up with a comprehensive design spec and then stick to it."

And everybody seems to be asserting all of these things with an equal amount of confidence, which is what really blows my mind. Like, it just seems really dysfunctional.
posted by Anticipation Of A New Lover's Arrival, The at 9:52 AM on January 26 [2 favorites]


Actually you know what, I'm just being annoying again. I should just avoid these kinds of threads in the future.
posted by Anticipation Of A New Lover's Arrival, The at 10:03 AM on January 26


Development is.... well, yeah, dysfunctional's a good word. You're not being annoying, you're pointing out the Emperor's wonderful birthday suit. And developers love engaging with these questions, 'cos it's all opinion.

> It's just weird that you have people here saying, "Agile works best when customers are willing to spend an open-ended amount of money to work on something until it's done," and then also people saying, "The reason Waterfall doesn't work is that it's too expensive, nobody wants to pay what things cost."

In Waterfall, the product is delivered at the end of the project. In Scrum, the product is delivered every two weeks (more often in some variations). If the client pulls the plug half way through, they should still get something of value. Theoretically. Again, it reduces risk in exchange for slightly more cost.
posted by Leon at 1:04 PM on January 26 [2 favorites]


Also worth mentioning: construction has it’s own management cult. It’s called Lean.
posted by q*ben at 4:09 PM on January 26 [1 favorite]


Maybe it would be better to stick with a single toolset long enough to actually get good at it, rather than having to re-learn basically all the mechanics of your job every eighteen months? That doesn't sound very efficient. Why does the software industry have such rapid cycling?

IMO, the actual conceptual magnitude of this tool churn is overstated in most contexts (except for front-end web development, probably). If you have a decently broad understanding of the basic principles of how software and hardware work, there's not much you'll find surprising in a new programming language or a new set of concurrency primitives or what have you. There's always some learning overhead, but in experience it's comparable with the work you have to do just to get up to speed with a new team.
posted by invitapriore at 4:25 PM on January 26 [4 favorites]


Anticipation Of A New Lover's Arrival, The: And everybody seems to be asserting all of these things with an equal amount of confidence

I see the same when reading parenting books.
posted by clawsoon at 4:19 AM on January 27 [1 favorite]


Lean has also infected healthcare management, although most healthcare leadership doesn't actually understand the concept.
posted by catlet at 12:50 PM on January 27


I wonder if there's a parallel with education reform movements: A new idea is tried (or, more often, an old idea is resurrected), and it works fantastically well in pilot programs. Then, when it's disseminated and turned into policy and millions of average teachers and administrators implement it, it doesn't work any better than the old way of doing things.

When highly-motivated, highly-skilled programmers and managers organize their work in a new way, it works great. This is how everybody should work!, they think. They publish a manifesto. The manifesto is turned into policy. Millions of average programmers and managers start working in the new way, and for some reason they still get average results.
posted by clawsoon at 10:44 AM on January 29 [3 favorites]


And everybody seems to be asserting all of these things with an equal amount of confidence, which is what really blows my mind. Like, it just seems really dysfunctional.

Yeah, that's pretty much software development in a nutshell. All of these contradictory things are usually true: the customer will not originally contribute well enough to the requirements phase (or often even know what they want, except in the cases they know exactly what they want, but even then they're often wrong); the customer wants a Cadillac on a used Honda sort of budget; projects without rigorous budgeting of time and resources will go well over both; the best way to scope out time and resources is to start with a good spec (but see how the customer isn't really helpful in this, at least to start). In this regard what waterfall does well is define a really rigid scope (and thus budget), even if it's not perfect; what agile does well is allow you to refine the spec as you go with less disruption to the ongoing project, even if it runs greater risk of scope creep.

And the "building a house" analogy is useful in the negative: a client commissioning a house at least has a mental picture of all the parts that go into a house, even if they may not quite understand what those individual parts would cost. Between building codes and a well understood risk of fire, no customer is going to argue with a builder about which electrical wire to use. Most of them aren't going to try to leave out the kitchen sink to save some money. Even the ones who "never cook" have an idea of a house as a tangible thing with market value, and the lack of a kitchen sink really messes with the market value. And they may even understand that if they want to change the scope after the project has already started (adding a room, say, or upgrading or downgrading finishes) that they may incur new costs associated with that beyond just time and materials.

Software customers don't, however, have a similar base understanding of software components, so they will indeed try to push you to use things they've heard about or which they think might be cheaper. It's really wearying to go through requirements gathering with customers who want to spec out, say, specific Drupal versions for clean sheet implementations, or ask you why you're using React when you could be using Angular (or vice versa). Nobody hires an architect on the basis of which CAD package he or she uses, but they will seek out software developers based on limited understanding of the tools they're requiring. And scope changes happen all the time.

On the bright side, agile is a pretty good way to work when you have more of a services business than a fixed product you sell once, because when clients pay you for services they reasonably expect that more services will come with more payments. Long running clients may negotiate prices down or price new features into contract renewals instead of paying as they go, but when you already have a relationship and a billing cycle the add-ons do scope themselves more naturally. Software subscription models also support this sort of continuous development better than buy-once-every-few-years boxed software, which is why Office 365 and Adobe CC and the like are pretty standard now.
posted by fedward at 10:28 AM on January 31 [2 favorites]


« Older In doing so set a course for adventure, and later...   |   Copa de la Diversión: Es Divertido Ser Un... Newer »


This thread has been archived and is closed to new comments