Ed.: I've made some minor changes to this post to improve flow and revise the conclusion.
A few weeks ago (October 16 to be precise…) I happened across an intriguing blog post by developer David Christiansen entitled Is there REALLY any rigor in Waterfall? Far from being a standard pro-agile polemic, Christiansen describes his own intellectual journey from a self-professed waterfall-neutral “fence-sitter” to agile/iterative-friendly convert through firsthand experience and research. Rather than accept prima facie the rhetoric on the problems with phased delivery, he researched the model’s pedigree and coupled it with his own experience and had his own revelation:
“For the last couple of years I’ve sort of ridden the fence on where I stand on project management. I’ve conceded that waterfall might have a place in the IT world in the past, that it’s a tool we should have in our toolbox, even if we don’t use it very often…
The waterfall approach to managing software development projects is one of the most costly mis-diagnoses ever. It structures projects for failure. It creates a cost-change curve that is unacceptably expensive (you know, the cost of a change grows exponentially as the project progresses) and is largely responsible for the high failure rate of IT projects. Waterfall creates a project structure that is rigid and forces you to make decisions and predictions when you know the least, then abide by them no matter what realities you expose in the process.
I am off the fence. Waterfall doesn’t belong in any IT project manager’s toolbox. It is based on and littered with myth.”
While I do enjoy reading about another developer’s Conversion on the Way to Damascus, David’s piece stands out to me because outside of Craig Larman, he’s the only person I’ve come across who makes reference of Winston W. Royce’s role in the story of the waterfall model. While Royce isn’t really responsible for the waterfall model – that came about as a result of our industry’s 30 year failure to RTFM – knowing who he is and the history behind phased delivery is vitally important to putting the rationale for agile/lean/iterative models into their proper context.
Is there a One True Way?
A week later, David’s post was picked up in a feedback thread to a rather pedantic advice column on InfoWorld that was responding to a student’s query on the value of requirements vs. testing (btw, I think both the instructor and the student are wrong) with the ensuing fracas prompting David to write a follow-up post where he expresses some disbelief about the fervor that seems to pervade the pro/con waterfall debate, ie. that there’s only “one true way” to deliver software: (emphasis mine)
Frankly, I’m perplexed by the belief among many technology practitioners in the ideal, the “one true process,” that will solve any problem, whether it is Waterfall, Spiral, Agile, or whatever comes next. It is a fixation I believe is unique to IT - I don’t remember seeing this type of zealotry when I worked as a manufacturing R&D engineer at Bell Helicopter. Engineers there had a much more experiment, do-what-it-takes sort of attitude that couldn’t be captured, analyzed, and then reduced to a simple prescription for project success.
With all due respect (and a wink and a nod), David’s probably not thinking of some of the holy wars that are waged in other disciplines like physics, quantum mechanics, climatological science and medicine when it comes to “one true way” zealotry. Hell, look at the trouble Robin Warren and Barry Marshall had to overcome in gaining acceptance of their evidence that a bacterium and not spicy foods were responsible for the vast majority of gastric and peptic ulcers. Intransigence and zealotry can be found just about anywhere.
But I digress: I also have to disagree with David's observation that the "experiment, do-what-it-takes" attitude of the Bell R&D engineers can't be captured and turned into a prescription for better project outcomes. In fact, this is the very essence of the models used in manufacturing (ie Toyota Production System, Lean) that agilists have adapted into a variety of methodologies from Scrum to XP to Crystal Clear to Lean.
Thus, while I agree that there isn’t an out-of-the-box “silver bullet” that will make every project successful, I am of the opinion that practices which belong to the family of empirical process controls have the highest probability for successfully delivering working software than any other process because they have significantly better visibility into project activities and an internal feedback-loop to support self-correction based on the realities of the situation at hand rather than a six-month-old plan.
Agile Values
David concludes his post by observing that in the final analysis it may be better to rely on human characteristics such as “brilliance, bravery, strength, and intelligence” to solve our problems than a naive and idealistic belief in a singular “right” way to build software. I think David’s got this half right, because I think he’s not fully aware of the values that back-up agile/lean/iterative practices – and I’ll grant that the zealots probably don’t know them, either.
For example, take the first value statement of the Agile Manifesto:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools.
Next, consider the Four Values that Kent Beck defines for project success under eXtremeProgramming practices:
We will be successful when we have a style that celebrates a consistent set of values that serve both human and commercial needs: communication, simplicity, feedback and courage.
Finally, consider Scrum creator Ken Schwaber’s comments on the challenges and rewards for implementing agile practices:
Of all organizations that attempt to become a world class software organization (by adopting agile practices like Scrum) only one third will make it.
The inspect and adapt practices of Scrum is where the pain comes in, because when you start doing that you discover everything in your organization that’s not working.
Agile is hard work; it’s a long process and it is extraordinarily satisfying
These aren't the words of idealistic zealots, but reasoned pragmatists, and they demonstrate a closer affinity to the core "human" values that David seems to cherish. While he may be dissuaded or alienated by the zealots who proclaim "one true way", the fact of the matter is that agile/lean/iterative disciplines (and that is what they are) work because they are built upon the fundamental understanding that building software is about people and how they can work best in a very complex and creative enterprise.