It’s a constant theme in software engineering: systems are too complicated, requirements too ephemeral, projects too risky. For the last twenty years we have been pouring intellectual fuel into the problem of large-scale software engineering, software quality, and overall implementation risk, and depending on who you talk to, we have either made a little progress or none at all. Recently so-called Agile Development methods have been getting a lot of press. A story in eWeek.com Friday highlighted the extent to which Microsoft is beginning to reply on agile methods, and in particular a variant called Scrum, to make shipping product easier, and faster.

Scrum, its name taken from the world of Rugby, and meant to evoke players huddling together to exchange strategy before heading out to play their roles, was developed by Ken Schwaber and Jeff Sutherland back in the mid-nineties. It actually predated much of what came to be known as Extreme Programming. The reason I like these approaches is that they take advantage of the essential mutability of software. The one thing that makes software engineering different from other forms of engineering and construction is the ease with which changes can be made.

I have made the point in this space previously that software development is like all other complex engineering tasks: it involves the assembly of myriad parts into a complex whole that achieves specific goals, or requirements. But the ease with which software can be changed ultimately lowers the risk of getting the requirements wrong. This may be surprising to those who have participated in the archetypical project flameout driven by just this sort of misunderstanding, but I think even they will admit that the flameout cost less than it would have on, say, a suspension bridge project.

Getting requirements right is hard. It involves iterations of analysis and design, and rests on the ability of specialists to derive the subtleties of a problem from ordinary users and other experts. When you’re building a bridge, or a hotel, you must invest the time and money needed to fully explore the problem space, because the costs of error are high. In software, unless the system is intended to run the nation’s air traffic control system, or something of similar criticality, most implementing organizations won’t or can’t commit those kinds of resources. In such an environment the attractions of agile methods are easy to understand, and may in fact offer a path to “getting it right” that the typical business can afford.