In Defence of Agile

State of the Things

The war is over and we won. Agile methodology, whatever one might understand by that, proved to be the most efficient way of running software development projects. Team members are happily coding, testing and deploying products. Management adopted Agile methodology and recognised its value. Satisfied customers are eager to try out new features. Other industries are queueing up to join the revolution. Waterfall showed the white flag.

Or so some of us believed. The truth is there was never any war. There are no good or bad guys here, no winning or losing sides. We are just thousands and millions of professionals working in the IT industry and exploring the best way to deliver quality products. And there is no such thing as the Agile methodology. There is an Agile approach and there are many methodologies adopting it in one way or another. And no, it is not over, or ever will be. In fact, the whole Agile philosophy involves constant improvement. And while we are on the subject, other industries do not envy our precious agileness. In terms of efficiency and productivity, we are way behind other trades, mere apprentices standing on the shoulders of giants.

There is no widespread adoption of Agile as we might think. Those of us lucky enough to work in developer-centric, modern environments^[1] might be excused for not noticing that the rest of the world is lagging behind. Many organisations are struggling with trust, skills or staffing issues. For some adopting mainstream Agile does not make any sense. Some don't see business value. Others do Agile in name only, just to keep up with the current trends. Or they let their development teams do this Agile thing all the way they want, as long as the Gantt chart is up to date and the change management process is strictly followed. And please, log your hours before you leave for the day. So here is the ugly truth that I have recently had a chance to experience in a rather brutal way: most organisations are struggling with adopting Agile.

What does it even mean Agile? There are many organisations in Agile Industrial Complex^[2] that would gladly explain it to you. For a hefty fee. Join one of many courses they offer and you will learn how to apply a rigorous technique which will lead you to better efficiency. You will also learn why other techniques are not "true agile". Or you can hire a consultant to take you on "agile journey". Their presence indicates many things wrong with Agile today. In this article, we will attempt to answer what it means by Agile, what problems it solves and how it helps with the challenges of developing software. But let us start where it is customary to do so: at the beginning.

Not So Humble Beginnings

Our industry^[3] emerged in 1954 with the creation of the first COBOL compiler. Unsurprisingly, the creator of the compiler was met with the protest: "But Grace, then anyone will be able to write programs!"^[4]. In the decades that followed, indeed masses started to write programs. And the world has changed. We see software in all aspects of our lives, from driving a car to teaching to smoking cigarettes to brushing teeth. Across the world, there are many garages that act as incubators for new ideas. This plethora of innovative and successful products is what makes the industry so attractive.

But first IT projects were nothing like that. Computers were big and expensive, resources were sparse. Only governments and organisations with large budgets could afford to run IT projects. That stands in contrast to other trades that humankind conceived over millennia. It was usual to start slow and small, and over the time appropriate practices would emerge. Construction, cars, health, education and virtually every industry started in that way. The fact that IT was initially dominated by big projects shaped the way we created software for a long time.

Most software development projects followed the same path. Some smart people conceived the idea and passed it down to others for further analysis. Technical sages prepared the grand design. The code was written according to the plan and handed over for verification. Finally, happy users were trained to use the system. In the meantime, project managers monitored the whole execution and made sure everything went according to the plan.

The following diagram represents the well-known approach to managing Software Development Lifecycle (SDLC) called classical waterfall model:

And the Plot Thickens

Let's keep that diagram in memory, while we lay down a few concepts related to the process of creating software. Most issues are repercussions of adopting the waterfall model and are provided in non-specific order.

You shall not fail. Most IT projects fail^[5]. Some miss deadlines. Some go over budget. Some scale down. Some deliver the wrong product to the wrong people at the wrong time. Many get all of these issues or get canned altogether. Most fail. Imagine 70% of bridges collapsing over time or 70% of operations causing health complications for patients. It is our duty to reverse this trend and rebuild the reputation of the industry. In the long term, we will all benefit.

It's the product, stupid. So far, I have purposefully put emphasis on the phrase "IT projects". Projects have their beginning and end. They have their own rhythm and budget. They have project managers who make sure that everyone adheres to the plan. But neither developers nor customers are focused on projects. We create and use products, which are much more persistent in their nature. Budgets are important but much more important is customer satisfaction. It is good to have a plan, but it is the unplanned that makes the difference. And the earlier the product is delivered to the customers, the better.^[40]

Fallacy of role specialization. The workflow presented above naturally imposes assembly-line work organisation. At each station, actors (workers) add their part and hand over the semi-finished product to the next in line. Tasks are clearly defined and project managers with the plan act as conveyors. And so business analysts gather requirements, architects design, developers code, testers verify and support does maintenance. There are some serious implications of such work organisation. Firstly, people have a tendency to slip mindlessly into roles ascribed^[6] due to situational attribution^[7], nature of conformity and power of authority. This is done at the cost of personal development, virtues^[8], and contribution to areas outside of specialization. Secondly, it leads to team fragmentation. Teams are divided into factions^[35], often split up in different locations, which only toughens the divide. Thirdly, such work organisation erodes the sense of responsibility for the end product as each actor can only be accountable for their part. Lastly, it provokes the "blame game" up and down the stream.

Dangers of isolation. Factions in disintegrated teams are growing apart both functionally and organisationally. Moreover, the vision of each faction narrows to inputs and outputs used for their operations. That leads to Plato's Cave^[36] effect. Factions develop their own sets of beliefs and practices based on limited access to information and crippled capabilities. To get a grip over disintegrating teams, companies often inaptly utilize methods like intrusive communication, meetings overload and forced team-building exercises. That only leads to even stronger feeling of isolation.

"Walking on water and developing software from a specification are easy if both are frozen"^[9]. Software is built from specific requirements gathered in the first phase of waterfall SDLC. As this is happening, an inevitable change to the requirements will occur. There are many different reasons for specification changes, from new functionality to technical difficulties to infrastructure requirements to defects. Change occurring at different phases of the project has different costs associated with it. The following diagram presents the common cost implications of change:

By the nature of change, it is not known when and if it will occur, or what the scope of the change is. Also, the classical waterfall SDLC does not assume any reassessment of requirements or reaction to defects. Much effort has been put into making waterfall model more adaptable. The general assumption is to add a mechanism to feed back the issue up the stream to decision makers. This is the best shown in this diagram^[34]:

This is quite difficult to follow, no matter how process-oriented teams are. Even the most reactive amendment to the waterfall model cannot mitigate the fact that change is intrinsically against the natural flow in the model. But the fact is that during the software development lifecycle requirement changes will happen, and they will happen often. As each change in effect leads to better-suited software, we need to revise the perception of change: from a problem to a competitive advantage. Any rigorous process around the change restricts this window of improvement.

"But we’ve always done it this way".^[10] Variations to specification are what most management will try to identify, track and contain. But software development teams also have to respond to changing environments. This includes, but by no means is limited to, changes in:

• team dynamics: leavers, joiners, upskilling, deskilling, personal circumstances, career progression, project fatigue
• working environment: office setup, hot summer, January blues, safety concerns, impact of terrorism
• company: reputation, culture, reorganization, financial position, leadership
• industry: changes in processes, design, architecture, practices and utilities with the effects beyond non-functional requirements; shifting industry focus can cause talent drain in other areas^[11], elusive nature of technological breakthroughs, workforce internationalisation and ease of migration, unemployment level, inclusive/exclusive nature of tech workers^[12]

Some of these environmental variations might be difficult to identify. Software development teams have to respond quickly to such changes, minimizing their negative effects or preferably turning them into an advantage. While doing it, the team has to keep a balance between maintaining its integrity and adaptation to a changing environment.^[13]

Agency of response. This agency is the body responsible for adapting to the environmental change. It might be located inside the team. It means that the agency is closer to the problems, but might be lacking the perspective, skills or required authority to apply appropriate responses. It could also be located outside of the team. But its distance to the team and lack of understanding of internal dynamics might result in inefficiency^[14], ignoring the problem or applying wrong medicine to wrong illness.  Also, an authoritarian nature of external agency might cause resentment within the team. Placing the agency both inside and outside might lead to overlapping responsibilities and conflict. Lack of the agency leads to lower productivity as the environment develops over time.

Adaptable system.^[15] The entropy of any closed system increases over time as proved by Second Law of Thermodynamics^[16][17]. Increased entropy leads to lower work outcome. This is why it is so important to maintain order in Complex Adaptive System^[18] like software development team. As stipulated by cybernetics, feedback mechanism plays the key role in maintaining order. The environment that the team operates in, constantly changes, in consequence requiring adaptation of team processes and practices. The classic waterfall model puts the burden of calibrating the system on an external body (upper management, steering committee, etc.). These attempts to adapt to changes are often inadequate (too little, too late) or do not have buy-in from the team.

Communication by specification. The more disintegrated the team becomes, the more hindered the communication between its factions is. This is often reflected in the resulting product.^[39] Direct communication is replaced by overreliance on the documentation passed between factions. Factions often resort to work-to-rule attitude when each directive in the form of specification is complied with very literary, even when actors are aware of potential negative effects. The examples of such behaviour are developers precisely implementing the specification even though they are aware that it is not compatible with other parts of the system or previous work.  This self-inflicted damage is caused by two factors: urge of proving the point (desire to be heard) and creating a safety net for any future blame game. This behaviour lowers actors' initiative and reduces the sense of ownership. Another consequence of communication by specification is the Broken Telephone game^[37] effect: the greater distance from the source and the more hops in specifying requirements, the more distorted the meaning becomes. Obviously, such effects are very undesirable.

The map is not the territory.^[19] The map with perfect fidelity would become the terrain itself and therefore rendered useless. In the less drastic scenario, imagine a map of the UK with the size of mile by mile. At this scale, all granular details are already lost, but the map is still not very handy for navigation. On the other hand, such map is detailed enough that any organic changes happening in the territory are not reflected on the map. Therefore, it becomes out-of-date the moment it is printed. A balance has to be found between map accuracy and its usefulness and maintainability. Also, there is only one source of truth: the terrain. When map and terrain differ, follow the terrain. That leads us to the conclusion that even the best model is only a reduced representation of the reality^[20], or in other words “all models are wrong, but some are useful.”^[21] Unfortunately, for many people, “the map appears more real than the land.”^[22]

Specification as a roadmap. Technical specification provides a model that developers use to implement the required functionality. Ironically, after reaching some threshold, the more details the specification contains, the less useful it becomes. There is a growing trend to replace traditional formal specification with approaches like Design by Contract, Behaviour-driven Development, Consumer-driven Contracts and many others. Although used for slightly different purposes solving different problems, they share one common thing: they concentrate on why over how. The specification becomes a roadmap with inviolable rendezvous points. The actual mileage might vary, but as long as the key points are visited, the goals are met. This means that the implementation is "negotiated" between different factions of the team on the way. Most projects manager might flinch in the face of such atrocity, but this is merely recital of the facts that “truth can only be found in one place: the code.”^[23]. There are two aspects to consider before taking this approach. Firstly, it requires the team to have a high sense of ownership. Secondly, there is a risk that developers are handed keys to power and effectively become interpreters of black magic of source code.

Cost of Time. In the simplest model, to deliver a product, we need to hire a number of analysts, architects, developers, testers, infrastructure engineers and support. In the waterfall phased approach, only one group at any given time can be fully utilized. In the meantime, the other groups remain idle or, what is worse, use the extra time to overdo tasks at hand, negatively impacting colleagues. While it is tempting to assign groups to other projects for the period of low utilization, this would increase Distance To Product (DTR). The distance plus cost of task switching lower overall productivity. From the above, it is obvious that a model, which offers full occupation of all groups at all times, would provide a much higher Return On Time Invested (ROTI).

Vicious circle of architecture. In the waterfall model, a lot of responsibility is placed on architects to provide the initial technical design. This is an ungrateful task as hardly ever enough information is available at this stage to provide comprehensive design and so architects are doomed to failure. This causes disparity between what is really implemented and what was originally planned. In the meantime, architects provide more design in line with Parkinson's Law. As the technical design grows, so does its distance from what developers really do. This, in effect, causes architects to provide even more designs to "get a grip on things" and thus creating a vicious circle. To the point that architecture faction is expanding to meet the needs of expanding architecture faction.^[24] The mechanism can be observed in other factions of disintegrating teams. What makes it particularly noticeable in the case of architecture is the compounded effect of self-preservation instinct. It is more and more frequent that the whole software development team takes responsibility for providing design, thanks to enhanced education and improved tooling. In effect, we don't need architects for architecting, the same way we don't need typists for typing. That feeling of inevitable extinction only strengthens the vicious circle as more design is created by architects in order to prove the worth of architecture. Of course, the situation described here applies to technical architecture, not enterprise architecture in cross-cutting domain.

Defectiveness of Taylorism. The process of creating software is unpredictable by its nature, we never really know what we end up with. That renders useless the big upfront plan preferred by the waterfall model. Even medium-term planning is crippled with a high dose of uncertainty. As such scientific management^[25] approach is inapplicable, coding is not a repeatable task and there is no tangible measure of performance (like locpm, lines of code per minute); each day in the life of software development team is different. In fact, the more creative a task is, the less Taylorism can be applied.^[26]

Self-organisation.^[28] A self-organizing system  would consume energy and order from its environment to improve its order^[27]. That order must be directed towards increasing productivity. That means that development teams are encouraged in various ways (recognition, job satisfaction, training and personal development, position, shame, pizza, threat, etc) to tunnel their energy towards building product. Without these incentives the system order can easily be optimized to serve non-productivity purposes like increasing individual ego, revenge, schadenfreude. Self-organising team proved to be very efficient form of management.^[29]

We are Uncovering Better Ways^[30]

It is Agile movement advocates' belief that the issues highlighted above are not just related to the waterfall model. They are built into the model making its integral part. Therefore, a better way of working must be harnessed. Based on experiences from the multitude of previous IT projects, practices that had been in use for decades^[31] and methods successfully employed in other industries^[32][33], Agile software development approach was formed.

The Agile movement addresses the issues of the traditional model by recognising the fact that software development is not just an act of writing code according to the prescribed design. It is a complex mix of technology, psychology, sociology, economics, science and management.

Agile software development approach was built on key concepts:

• Cross-functional, multi-skilled, self-organizing, domain-specific teams
• Continuous improvement
• Frequent delivery
• Adaptive rather than predictive model
• People-oriented rather than process-oriented

We can see that using these simple concepts resolves, or at least significantly improves, the issues highlighted above. Agile teams work closely towards common goals and therefore avoid pitfalls of disintegration. The specification becomes an outcome of the whole team effort, avoiding problems of insufficient model accuracy. Requirements change is a very welcome opportunity to improve the product. Self-organization is an important aspect of improving productivity and responding to changing environment. Also, the impact of external agency of response is more likely to provide positive reactions, often reducing negative second-order effects.^[38] The team work is organized around frequent releases, naturally removing any inefficiencies. Capacity of all team members is fully utilized at most times, thus reducing their distance-to-product and improving overall ROTI.

The following iterative model is a typical representation of Agile approach:

By no means is the Agile approach a magic wand, that can fix all issues. The issues range from lack of long-term project visibility to limited career progression to dependency on team collaboration. There are types of software systems or organisations that would clearly not benefit from adopting Agile. Many methodologies that were built on Agile are of various quality. This might be off-putting for newcomers. In fact, issues of agile approach entail a whole separate article.

Still, the Agile approach is the most powerful concept in modern software development lifecycle management. It has proven to be efficient and provide better results. What makes it so attractive is that the essence of Agile is to improve and evolve constantly. And nothing can summarize it better than these humble, but wonderful words of the manifesto:

"We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more."

---

Remarks

[1] For the record, I don't believe that developer-centric technocracy is a good thing. Like with everything in life, balance is required, and if we forget the service-oriented nature of our work, the great pendulum of history will eventually swing back and we might not like what comes next.

[2] Martin Fowler (2018) The State of Agile Software in 2018, available: https://martinfowler.com/articles/agile-aus-2018.html

[3] By the industry, I mean mass, professional software development. Obviously, computer programming, hardware development and related science subjects came before this time.

[4] It is commonly reported, but the unsourced quote

[5] Standish Group (1995-2018) CHAOS Report

[6] Based on anecdotal evidence and personal experiences as I reject conclusions of Stanford Prison Experiment (Haney, Banks, Zimbardo (1973) A study of prisoners and guards in a simulated prison). The subject was further explored in The BBC Prison Study with better scientific rigour (Haslam, Reicher (2006) Rethinking the psychology of tyranny). Both studies suffer from the WEIRD people bias (Henrich, Heine, Norenzayan (2010) The weirdest people in the world)

[7] Fritz Heider (1958) The psychology of interpersonal relations

[8] Example being forfeiting personal values and believes in the face of duty. See Good Samaritan Experiment from Darley, Batson (1973) From Jerusalem to Jericho: A study of Situational and Dispositional Variables in Helping Behavior

[9] By Edward V. Berard. I was unable to find the exact source and date.

[10] Grace Hopper in The Baltimore Sun (1975) Navy computer grandmother keeps moving, see also: https://quoteinvestigator.com/2014/11/27/always-done/

[11] Skills shortage has been a serious concern for decades in the industry. Its ever-expanding nature only exacerbate the issue, e.g., developers moving to DevOps because of career progression, financial incentives or No Country For Old Men stigma;  or new graduates choosing Data Science over Software Development

[12] Historically, tech workforce has had a clear bias towards specific race (Caucasian/Asian), gender (male), age (below 35), personality trait (introvert). As a welcome change is happening towards a more balanced  situation, so does change team dynamics and requirements.

[13] Stafford Beer (1959) Cybernetics and Management

[14] Due to team homeostasis, the tendency of a system to be resilient towards external factors and maintain its key characteristics, Douglas Shepard Riggs (1970) Control Theory and Physiological Feedback Mechanisms

[15] William Ross Ashby (1956) An Introduction to Cybernetics

[16] Rudolf Clausius (1867) The Mechanical Theory of Heat – with its Applications to the Steam Engine and to Physical Properties of Bodies

[17] Constantin Carathéodory (1909) Examination of the foundations of thermodynamics, available: http://neo-classical-physics.info/uploads/3/0/6/5/3065888/caratheodory_-_thermodynamics.pdf

[18] Ahmed E, Elgazzar AS, Hegazi AS (2005) An overview of complex adaptive systems

[19] Alfred Korzybski (1931) A Non-Aristotelian System and its Necessity for Rigour in Mathematics and Physics, available: http://esgs.free.fr/uk/art/sands-sup3.pdf

[20] I used the allegory of map and terrain as a model. And then, I used this imperfect model to draw a general, strict conclusion that all models are imperfect. See what I did here? In this case, it was a fun, intentional measure, but we should be on guard with superficial analogies that "are useless in science and harmful in their practical consequences" (Spyros G. Tzafestas (2017) Systems, Cybernetics, Control, and Automation)

[21] Box, Draper (1987) Empirical Model-Building and Response Surfaces

[22] D.H. Lawrence (1936) Study of Thomas Hardy

[23] Robert C. Martin (2008) Clean Code

[24] Obviously, a paraphrase of Oscar Wilde's “The bureaucracy is expanding to meet the needs of the expanding bureaucracy.”

[25] Also known as Taylorism (Frederick Winslow Taylor (1911) Principles Of Scientific Management): "It is only through enforced standardization of methods, enforced adoption of the best implements and working conditions, and enforced cooperation that this faster work can be assured. And the duty of enforcing the adoption of standards and enforcing this cooperation rests with management alone"

[26] Katia Caldari (2007) Alfred Marshall’s critical analysis of scientific management

[27] Von Foerster (1960) On self-organizing systems and their environments

[28] Ashby (1962) Principles of the self-organizing system

[29] Tribus, Myron (2001) Lessons from Tomas Bata for the Modern Day Manager

[30] Robert C. Martin, et al (2001) Manifesto for Agile Software Development, available: http://agilemanifesto.org/

[31] Edmonds (1974) A Process for the Development of Software for Nontechnical Users as an Adaptive System

[32] Iacocca Institute (1991) 21^st Century Manufacturing Enterprise Strategy: An Industry Led View

[33] Presley, Liles (1995) Agile Aerospace Manufacturing

[34] Winston Royce (1970) Managing the Development of Large Software Systems

[35] Faction: a group within a larger group, especially one with slightly different ideas from the main group (Cambridge Dictionary)    

[36] Allegory of the Cave: a thorough experiment put forward by Plato (Plato (380BC) Republic) concerning human perception in situation of limited cognition and its role in shaping concepts acquired by subjects

[37] Broken Telephone (aka Telephone aka  Chinese Whispers) an internationally popular children's game. Players form a line and the first whispers a message to the next in line. The message is passed onto until the last person and then compared with the original to a comical effect.

[38] “Changing some aspect of a complex system always introduces Second-Order Effects, some of which may be antithetical to the original intent of the change.” - Josh Kaufman

[39] Conway's Law ( Melvin E. Conway (1968) How do Committees Invent?): "Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations."

[40] Based on Pareto's principle, 80% of the required functionality is covered by 20% of the product.