So, You Think You're Agile?

The software development industry has generally converged on the Agile Methodology as the one-size-fits-all universal approach.  Agile has many attractive features to justify the infatuation. Working groups are largely self-organizing, short sprints produce tangible results at regular intervals, daily goals are set and reinforced through informal scrums and the overall direction is honed and re-honed via a tight feedback loop with the client.

image courtesy of commons.wikimedia.com

Agile avoids the Problems of Big Analysis: the painstaking process of determining functional and technical requirements down to the last detail before the first line of code is typed.  One can see the draw of this so-called "old-school" methodology.  The developers (not to mention the client and the Product/Project Managers) set out with clearly defined goals. Scope is established and signed-off  by all parties, designs have been finalized, the opportunity for surprise greatly reduced.  The actual work of implementing a well-defined product is greatly simplified when all related factors has been rigorously identified beforehand. One highly-evolved exponent of this approach would be the Capability Maturity Model Integration (CMMI).  Evolved? Yes, but delivery is going to take awhile.

In the rapidly-changing world of business app development, Big Analysis is also fraught with well-known perils. Thanks to the inexorability of Moore's Law, client needs (and wants) are evolving at an ever-accelerating rate. Products of all kinds are in constant peril of becoming obsoleted by competition; in the digital world, this can happen before a product is fully defined.  If any product expects to survive any length of time in the modern market, it must move from idea to distribution/deployment as rapidly as possible.

The desire to produce software quickly is certainly nothing new.  Software development is expensive: analysts, developer hours, sysadmin time and QA iterations are all costly. Even more costly are the implications of deploying or distributing a deeply flawed product. In the 90's, we spent much of our pub time discussing the precepts of "Rapid Application Development" and the Pyramid of Fulfilment. Reaching back into the elder days, we find that IBM was a leading exponent of agile development (before the term was coined) via a significant investment in APL (A Programming Language), an interactive scripting language which places strong emphasis on packing a lot of semantic meaning into a relatively small set of symbols.  This encouraged developers to  bring clients to their terminals to model program behaviour iteratively with immediate client feedback.  Modern vendors of APL still frequently practice this, always have.

The Agile Methodology has come down to us as a set of broad rules, designed by long observation and experience, to turn ideas into concrete implementations with a minimum of overhead in terms of time and budgetary resources.  It encourages project planners to choose those elements of Agile which are suitable to the project at hand and disregard the irrelevant. Every Agile shop I have had the opportunity to work in has had what amounted to a set of individualized, custom processes intended to produce the vision, such as it might be, with a minimum of time and confusion. Unfortunately, that intention is all too rarely realized.

To give the matter some deeper thought (a thing which Agile neither encourages nor does it explicitly prohibit), we observe that the more successful advocates are aware that certain elements can not be ignored.  For example, Agile dictates that project members must be dedicated to that project, not stretched across multiple projects, each of them vying for a slice of attention.  A multi-tasked developer cannot achieve the level of focus that true agile development demands when they are spending hours in unrelated meetings, or hampered by delivery schedules for other products. The Agile developer is expected to be sensitive to the reality that many aspects of a project may be under-analysed or weakly defined and they need to be responsive in identifying those aspects as they arise. This is not easily accomplished by a distracted developer or, worse, an overworked one.

Another element of Agile methodology which is all too often gleefully ignored is recommendations about work hours.  A tired programmer makes mistakes, they miss things.  The software development industry has always demanded extra hours from programmers, sysadmins and project managers alike. There will always be events which necessitate it: milestone delivery dates, clearing developmental bottlenecks, operational emergencies.  This is accepted and well-established.  But if your teams are putting in 60+ hour weeks, month after month, it points to a flaw in the planning and estimation process.  A naive manager might perceive this as getting 50% added value with no added cost but as it proves there are serious costs for this practice. Before very long the measure of concrete progress slows and things get missed, people start making mistakes. More than once, I have seen a code base regress under these circumstances. Those things that are getting missed are often the things which were not identified by the analysts or the architects because, hey, we're Agile and continuous discovery is part of the job.

There are two other aspects to Agile Methodology which are critical to successful execution but which are often ignored: the Maintenance Sprint and Peer Review.  Developers are explicitly instructed to implement any given task in most convenient way possible. Don't worry about the best way of doing a particular task, just find one which works well for now.  This philosophy is based on the observation that tasks are frequently revisited as analysis-by-discovery continues and the time spent to determine  that optimal method and subsequently code it often turns out to be wasted as new information may invalidate that code.  And this is the condition which makes both the Maintenance Sprint and Peer Review indispensable.

The Maintenance Sprint is the only opportunity you have to optimise your program, to spend time on improvements previously deferred for those parts of the business logic which have 'settled'.  It is a chance to organize the creative, task-oriented out-flux into a coherent model; to provide a sober, pessimistic Yang to counter-balance the effusive optimistic Yin of the creative developer. The Peer Review puts a second set of eyes on the code which can only serve to increase the quality of that code.  Even the most accomplished developer benefits from reviewing code with someone of similar experience, to spot the odd oversight, correct assumptions or to validate an apparently ideal solution.  Junior programmers engaged as readers in this exercise get to see how their more experienced peers go about things.  These are the means by which we tune our programs and sharpen our programmers.

A related risk is the problem of controlling scope in a context where it is acknowledged that all requirements are not known up front.  The subject of negotiating scope in an Agile environment is beyond the scope of this particular essay but it certainly bears consideration as scope-creep continues to be the leading cause of death among projects.

Agile, when done well, is a powerful and transformative methodology but is up to the project planner to ensure that the methodology used is complete.  An incomplete methodology tends to lead to an incomplete product and then it never gets out the door.