Agile Programming: How to doom your software project

This morning, Kasia pointed me to "Agile Programming", and her take was that it was yet another buzzword. Looking into it, I find that it's more than that - it looks like a roadmap for running a project into the ground. And nobody would know why.

Reading the Manifesto for Agile Software Development we see a list of lovely platitudes that sound so nice, but hide a sinister reality.

• [We value] Individuals and interactions over processes and tools

Obviously we want to treat team members with respect and bring out the best in everybody, but those interactions have to be minimized as much as possible. Brooks' Law says that adding manpower to a late software project makes it later, and this is because of the exponentially increasing number of communications channels.

Tools and processes are fundamental sources of quality, and though of course it's possible to saddle a team with lousy tools, good tools offload much of the tedium and allow the team to expend its energies on things that matter.

Of course, no programmer is an island, and an awful lot of hard problems are solved not by a good tool, but by two developers chatting over coffee. But I think that a lot more projects have been sunk by not-enough-tools than by not-enough-interaction. I doubt many have been sunk by not-enough-people.

• [We value] Working software over comprehensive documentation

I can certainly see where this comes from, and it's hard to argue that if the software doesn't work, it doesn't matter how good your docs are. But professional developers revolve around the specification: without a spec, the developers are just hacking at code without a common purpose. This works great at the beginning (lots of progress) but goes downhill very rapidly.

The best way to appreciate the value of a good spec is by doing a project without one.

Furthermore, I have long held that "it works" is not the sole metric of software project completion, or even the best one; there are real qualitative differences between various kinds of "working" code, and anything that encourages "but it works" responses is saying that quality doesn't matter all that much and that all developers are created equally.

• [We value] Customer collaboration over contract negotiation

This has a "forget the spec, just start programming" smell that frightens me. "Contract negotiation" is not just deal-making and paper-pushing: it's deciding what the customer wants. Not only does the development team have to know what the customer wants, but the customer has to know too, and the very fact of going through "contract negotations" helps set the plan for the whole project.

Some years ago, a customer wanted me to build a large software system, and they wanted me to "just start programming". In spite of their desire for "customer collaboration", I insisted on writing a formal specification. Several weeks and 100 pages later it became clear that the project was well beyond what my customer could afford, so it was scrapped. My pushing back and forcing extended "contract negotiations" saved my customer an enormous amount of time and money.

• [We value] Responding to change over following a plan

When I see "a plan", I think "a specification", and anything that gets away from this is trouble. I think it's self-evident that change will happen during a project: sometimes external factors change the requirement, sometimes the customer didn't really think it out clearly up front, or perhaps somebody just had a great idea partway through the project.

Hopefully one has built a system that tolerates change, but one cannot send the message that change (especially late in the game) has no costs, or at least costs no more than change early in the game. Going through a round of "OK, let's find out what you really want in this change" not only nails down the details, but it creates a climate of taking change seriously.

In fairness, when one reads Principles behind the Agile Manifesto one does find good points: stepwise refinement has been a great idea (at least) since Niklaus Wirth wrote about it in 1971, and fostering teamwork is essential to any project. And any project that doesn't involve the customer on an ongoing basis is doomed.

But the whole "Agile Programming" idea seems (at best) a platitude fest, and I don't see how any actual professional software managers could have participated in this silliness. Looking at the manifesto, I can't help but believe that more projects have gone south due to following the warm-fuzzy festo points in favor of professional software-development disciplines: why would anybody want to encourage this?

And I wonder what Joel would say about this?

Comments

[Shawn Lauriat]

I'll give you an "Amen!" on that one.

Yeesh, undocumented feature creep...sounds nightmarish.

[Hans]

Have you actually tried it, and found that it doesn't work? Or are you just guessing?

[Steve Friedl]

Of course I haven't tried it; I am actively opposed to the principles it espouses.

This is not like Extreme Programming, which might be goofy or it might be clever - you can't really tell from reading the descriptions. It sounds plausible, so the only way to really find out is to try it.

I am given slight pause by some of the luminaries behind this, but I can't help but think they've just gone off the warm-and-fuzzy deep end.

[Kent]

First of all, I've never seen a single project delivered only because of the availability of good tools. Only good people make project successful.

Second, I'm doing mostly web sites development and I am the one who discuss features to add to the web site with a customer. What I noticed is a quick and dirty prototype makes things much more clear than any spec which a customer doesn't even know how to read properly or doesn’t have time to read.

That is the idea of agile programming. It defines a number of methods that allow you experiment with features without the fear of be bogged down with increased number of bugs.

[Simon]

Great, thought-provoking post. As someone who went through (and survived) the dotcom bubble when XP was at its zenith, I can safely say that the methodology never lived up to it's hype. For example, while the goal of involving the customer is admirable, very often customers have no desire to be heavily involved; they just want the job done. Similarly, I haven't yet met the customer who is happy for work to start without knowing at least a rough estimate of the total project cost. Of course, estimating the cost means knowing the scope...

Most methodologies contain at least a grain of usefulness. Some of the ideas of Scrum (a related XP approach) I have found useful, particularly daily stand-up meetings. These can help keep the team focused on what is important and reduce the risk of communication failures, especially toward the end of a project.

What really gives people that "warm and fuzzy" feeling is achieving something, and in software development that requires a clear specification, a rigorous and efficient process, good tools and skilled developers. Methodology is a just a tool, not the entire solution.

[Steve Friedl]

Oh, I don't believe that the customer should ever be uninvolved - this is a recipe for disaster. The customer may say that he doesn't want to be involved, but the only way that can work is if he agrees to take the system as shipped: even if they think they can live with this arrangement, the bracing air of "reality" always kicks in after delivery, and at the end of the project they're going to be unhappy. And blame the developers.

When I see how Agile Programming is implemented, I see things that I'm ok with - you have to involve the customer, stepwise refinement, and so on. This is a good thing.

But the principles expressed in the Manifesto are full of feel-good stuff that runs counter to practices I believe are important.

[Vaughan]

Many good points have been made by everyone.

Having worked myself in an agile / scrum environment I can say that the method definitely has its merits. The problem I found though is not in its fundimental principals but in its application. Agile and Scrum are not a replacement for good development practices but can be used to get the most out of a development team.

What management often seem to do is think that Agile and Scrum are a panecia (cure all) to get development back on track (or is used as a tool for denial... "We don't need to change our chosen solution software. We need to chage our process"). What I found was that, if internal processes/tools are bad Agile/Scumm accelerates the speed at which a critical problem occurs (whilst in the short term providing for higher productivity) when used with a in conjunction with a well planned, and well documented specification the agile / scrum process maximises team development output (and can acclerate a project).

• [We value] Individuals and interactions over processes and tools:

This does not mean the process and the tools are not important. It just means the focus is on the team, the current situation and how to move forward.

• [We value] Working software over comprehensive documentation

This is intended to convey the fact that some organisations produce more documentation than development output (rather than to say documentation is bad). The agile approch (as I have experience) is to document business level requirements, document the business process and document the necessary technical specifics of the solution.

However, this statement is not necessarily "core" to the agile / scrum approach IMO.

• [We value] Customer collaboration over contract negotiation

I wouldn't see this as saying "forget the spec" myself.. this is talking more about how a company and the customer work together. Its saying that its better for both the customer and the business to be cooperative / collaborate over the development and specification process rather than to waste time and resources debating and arguing over the fine print of a project unnecessarily. This does not mean important information should not be conveyed through debate / discussions.

TBH, though.. its basicly a nice way of saying "Customer, be flexible! you might have imagined a jet fighter but we are going to build you a perfectly servicable twin prop that will still be better than the hand-glider you currently run").

• [We value] Responding to change over following a plan

Ok, this one does have alarms going off all over the place ;).

What it is trying to illude to though is that if a plan is failing it is a fools errand to continue to go with the plan until the project has failed. What it highlights is that adaptability is important to success.

I compeltely understand where you come from though on all your points, and have seen business employ Agile/Scrum as you describe (which makes a very depressing work environment for the team).