CodeProject
Most people - even the overwhelming majority of programmers - would say that the main activity of a software developer is "writing source code". But this is a (though quite understandable) misconception - and if you take a look at the available figures on the issue or if you - as a software professional - are honest to yourself, the misconception immediately turns out to be an enormous one.
The world is full of software systems that are already in operation, and they have to be maintained - writing a new software system from scratch is a rare highlight for most software developers. On the other hand, the importance and especially the business value of maintenance is greatly underestimated, as well as the amount of actual working time spent on it. I think the main reason for this misperception lies in the fact that maintenance activities are spread over a much longer time span compared to the far more intense development period. Also, maintenance costs per year are relatively lower than the initial development costs.
Consequently, far too little effort goes in the future maintainability of a software product during actual development, which in turn leads to software systems that are hard to maintain and cause substantial technical and financial problems, or at least prove to become much costlier than necessary and/or expected. - Or, in business terms: the Return On Investment of a software becomes negative far sooner than necessary, thus making it necessary to replace it with a new product, which in turn produces high costs, and so on... - Some people even predict something like a global "IT meltdown" (see [1] for an example), and their reasoning is quite conclusive, although this sounds a bit overdramatic...
1. Where are we - and where are we (unintentionally) heading to?
As an introduction to the problem, I'd first like to present some figures from an older but widely recognized study by Boehm ([2], from 1981):
- Software maintenance costs in some selected (very) large US organizations lie between 60% and somewhat under 80%.
- The total maintenance effort in a larger sample of organizations (487 namely) yielded the figure of 49% - compared to 43% attributed to development activities.
These figures are quite old (almost 30 years), and are cited here deliberately, because they may help to shed some light on an overall trend of the software development industry: The relations strongly tend towards the maintenance side in the course of time. This is mainly because of two reasons:
- The natural aging process of software systems. They inevitably and naturally tend towards entropy with every change or update, and the used technologies (programming languages, frameworks etc.) become more and more outdated, so it's harder to find experts with the required knowhow, and they will take higher rates.
- Software has to fit into its environmental world, and therefore has to evolve constantly. Major changes like recently the introduction of the Euro or the Y2K problems required much maintenance effort for existing software. And there is much more to come - just think of the new format for IP addresses, the need to add digits to US social security numbers, or the frequent legal changes in individual countries that have to be reflected in many software systems.
Along these lines, Jones (2006, [3]) comes to the following conclusion:
Mass-update and other maintenance projects will potentially absorb almost 70% of the world’s software professionals during much of the 21st century. Mass update software projects can top five trillion dollars in overall costs before the middle of the twenty first century.
As of now, the average portion of maintenance on the overall lifecycle costs of most software systems is expected to be somewhere between 60% and 80% (see for example [4] and the references therein - concrete figures naturally vary somewhat across individual studies). However, in some areas, we have a phenomena referred to as the legacy crisis, with maintenance costs reaching the 90% mark (see e.g. [5]): Old systems - often programmed in a very idiosyncratic 'style', full of 'clever' hacks, with technologies that are totally outdated and very hard to understand from a present point of view, with no documentation at all or, even worse, with unreliable documentation - cause serious problems, and sometimes even bare disasters. If we don't do something about that and learn from it, the consequences are clear:
Concern is growing that the development of new software is outpacing our ability to maintain it. With large portions of software budgets being devoted to maintenance, few resources remain for new development. [...] If these trends continue, eventually no resources will be left to develop new systems, and we will enter the Middle Ages of the information age. [6]
The above are only some figures and statements about a project's maintenance costs that came immediately to my mind. I could add more, but the overall picture would stay the same: Software maintenance accounts for about the half up to 90% of a software project's total cost. In all but the most exotic or trivial cases, maintenance will be the biggest single cost factor - much bigger than development! Where exactly a project is located within the 50-90% range, depends largely on the project's concrete nature and the related business structures.
From my own experience and feelings, I'd say that the average amount of a software project's maintenance effort normally lies somewhere in the upper 60% range - "two-thirds of total costs" or "twice the costs of initial development" usually are pretty useful rules of thumb...
2. Why are we there ?
So, if maintenance is such a big issue, and the problems are growing bigger at a high rate, what are the main (technical) reasons then? The picture is not (yet?) overly clear here, mainly because the research on the issue lacks a solid theoretical background (What exactly is meant with the term maintenance? How can something like the "performance of a maintenance-related activity" be measured objectively? Where can we get realistic sample data and comparison values?...). But overall, it appears to be true what a programmer would say anyhow out of his bare intuition: If you want to ease the anticipated maintenance burden, then you have to write good (= clean, well-crafted) code, you need a proper design, you must understand the related 'real-world' business, and you must also have good documentation at hand.
A 2008 empirical study ([7]), based on data collected via an e-mail survey, found the following main problem factors for software maintainability:
- Inadequacy of source code comments
- Documentation is obscure or untrustworthy
- Changes are not adequately documented
- Lack of traceability
- Lack of adherence to programming standards
This list is interesting particularly because most of the above points have to do with proper documentation. And we all know that documentation is one of the first things to be omitted when the deadline is coming closer - sometimes with the comment "We can do that later", which always is wrong...
Another well-known code smell is code duplication, which in the worst case effectively means "error duplication". Fowler et al. (in [8]) have the following to say about duplicated code:
Number one in the stink parade is duplicated code. If you see the same code structure in more than one place, you can be sure that your program will be better if you find a way to unify them.
Large portions of duplicated code (aka. Code clones) are often the result of Copy/Paste-Programming - a "technique" which can be regularly observed when the deadline is nearing. But in most cases, it would be better to miss the deadline instead...
Other strong evidence locates the root problem more in a software system's complexity, be it architectural complexity as for example in [9]...
... we identified strong correlations between software design complexity and change effort for error corrections performed during the maintenance phase.
...or a complexity concept which is closer to the source code, as in [10], which looked at the correlation between the Cyclomatic complexity software metric and the defect rate of the respective file:
... if we know nothing else about a file except that it has a high Cyclomatic Complexity value, we can now have more reason to believe that it is likely to cause problems ...
Individual research outcomes and methodologies may be subject to discussions, but the overall bottom line is always the same: The more complexity, the higher the probability of future problems. This strongly corresponds to my own personal experiences - if you have ever struggled to understand a system which is basically a Big ball of mud, then you know what I mean...
There's another thing to mention about a software system's complexity, namely that it grows bigger at a much higher rate than the complexity of the real-world domain that it relates to:
For every 25 percent increase in problem complexity, there is a 100 percent increase in complexity of the software solution. That's not a condition to try to change (even though reducing complexity is always a desirable thing to do); that's just the way it is. [11]
Basically, complexity in software systems comes from two different sources: either from the problem domain itself or from the technical stuff that comes with software development: programming language(s), system architecture, used (or not used) frameworks and components, hardware properties and the like… Because a software developer usually can't do much about the real world's complexity, it's even more important to control the technical aspects of a program.
On the one hand, It is common sense among software developers that "Drowning in complexity" is a danger which is inherent to coding, and I don't think there's a single developer out there who never shot himself in the foot by doing something too complicated. But on the other hand, I've never seen complexity being monitored and controlled systematically during development. - I know that it's possible without too much effort and I heard of it being done, of course. But I've never seen it in real life. - As of now, it is largely a matter of gut feelings in common practice whether something is "too complicated" or not...
The (growing) importance of readability and understandability
While there may be varying opinions on what exactly should or should not be understood under the term maintenance and how one can objectively measure that, one thing remains true in any case:
There is one common thread, however. An essential step in all types of software maintenance is understanding the software system to be maintained. Before a maintainer can change software in order to fix a problem or add a feature, he or she must understand the software as a whole and the particular modules that need to be modified. Many experiments have indicated that program understanding, which includes understanding of software requirements and design, takes approximately half of all maintenance efforts. [12]
So about half of the work of a maintainer goes into code reading, browsing through the project and reading related documents, in an effort to figure out how the software works and why something was done that specific way. Half the time - this correlates to about 25-45 percent of the total costs, if we take the above 50-90% range as a basis. Even for a small software system, this will easily be a six digit number (in €)!
There seem to be no empirical studies on what individual factors influence readability and understandability most. But some anecdotal evidence as well as my own experiences from working with other people's code (which is always a hard task, but sometimes it can become really frustrating...) strongly suggest that the following are among the main factors supporting understandability:
- Domain expertise
- Low complexity and a clear structure (source code, overall design, system architecture etc.)
- Adherence to coding standards (both coding style guidelines and design standards)
- Documentation quality
It is not surprising that this list is roughly a subset of the problem factors that were found to influence the overall maintainability of a software system. What makes it still noteworthy is mainly the fact that the importance of readability/understandability will likely gain even more weight within the 'maintenance complex' during the next years.
At least in my corner of the world (Germany, and to a smaller extent other German-speaking countries), there's a persistent skills shortage in the IT area [13] - and things appear to become even worse, if you look at the numbers of the educational system. I doubt that this situation is all too different from that in the rest of the world. Simply put: The world is short of developers. This is likely to have the following consequences:
- A good developer will become an even more valuable resource than he/she already is today, and - as shortage inevitably leads to higher market prices - a developer's work will most likely become higher in price. And remember: More than two out of three members of this rare breed will be occupied with some sort of maintenance work!
- Constantly employing software developers all the time will become less and less attractive and affordable for smaller companies that need software only as part of their product, but are not in the IT business themselves (which is by far the majority of cases).
- Outsourcing and Offshoring will become more widely used strategies, even for relatively small companies. Also, Freelancing will become a more common professional form for software development professionals than it is today.
All in all, we can take for granted that an increasing number of different professionals will work on a software system in the course of its lifetime. - Developers, who were not involved with the initial development, who probably only have a vague idea of the relating business, and only have the code itself (and hopefully some good documentation) as a working basis. That being said, it becomes immediately clear that readability/understandability will become an even more important key success factor for a software project.
The 'Broken Window' thing
Furthermore, it is widely acknowledged that issues like e.g. Adherence to coding standards, Completeness of documentation, Degree of code duplication are particularly subject to what is known as the Broken Window Phenomena. Basically, it says that you have to keep your environment clean and fix problems immediately when they occur, as long as they are small and manageable. Otherwise, bad habits of all sort will soon appear to be acceptable, and vandalism will rise its head. Originally, the theory comes from the context of urban sociology, and in its most general form it reflects a simple everyday wisdom ("Don't let the door open for chaos, not even a bit."), but it turned out to be highly applicable to programming/code quality issues as well.
If you express the broken windows approach in terms of software development, you might get something like this: Once you start to allow your code being poor in some respect, it is inherently promoting a looser attitude on code quality among your team, and this can quickly - much quicker than you might suspect! - lead to a complete disaster.
We’ve seen clean, functional systems deteriorate pretty quickly once windows start breaking. There are other factors that can contribute to software rot [...], but neglect [of poor code quality, tw] accelerates the rot faster than any other factor. [14]
Personally, my strong impression is that every software developer out there fell into that trap at least once. The conclusion: You better be pedantic and paranoid, when it comes to code quality issues.
So what?
The bottom line of all this is far from new and sounds almost trivial (at least for my ears), but still there seems to be a strange sort of collective denial of the facts, both among software developers and management people. If you talk about it to involved persons, they will most likely fully agree with you, but the insights do not really have any practical consequences… So I feel I have to state it here again loud and clear: Maintenance is by far the most important, time-consuming, and cost-intensive aspect of a software system - and it's extremely underestimated and undervalued in contemporary professional practice. In my opinion, we need a mindset change among current industry practitioners (both developers and business people) as well as in the educational system concerned with software development issues.
The software industry as a whole needs a better understanding of what maintenance really is. It's not "defect removal", and it's also not "writing a program, then delivering it to the customer and working on it subsequently only in reaction to customer complaints" - it's more like "creating a long-living system (with the long-term impacts of short-term decisions in mind), and then working on its alignment with the real world for the entire rest of its lifetime".
Software development essentially is a two-phase process:
- The first one is the initial planning and coding phase, which is quite intensive, but relatively short in terms of the overall lifetime of a software system.
- The second one spans the much bigger part of the software’s lifetime (in usual cases well above 90%), and here the software has to be kept aligned with its technical and business environment, and eventually it has to be enhanced in functionality. (The amount of effort for these activities grows with the success of a software system.) This will be easy or hard depending on how good the software is designed and coded. So you lay success during the planning/coding phase.
For every single software developer, it's two to three times as likely to work on an existing system (a "brownfield project"), than beginning with a white sheet of paper (a "greenfield project"). Beginning programmers should know that they will spend most of their professional lifetime with software maintenance - not programming, as they might suspect…
Automating, monitoring, measuring
"You can't control what you can't measure. [15]
It is obvious that better maintenance tools and technologies are an urgent global priority. [3]
While the above are long-term, fundamental considerations, a great deal can be done right away – especially when it comes to C#/.NET. The required software is either completely free or has a ridiculously low price. All that needs to be invested is know-how and time – and most of the so incurred expenses need to be issued only once, while the benefits are permanent.
It’s not overly hard or complicated to set up an automation server, and it does not require much resources – a Windows license and a standard machine are all you need for the beginning. Once set up, the server soon will do much more than just performing simple builds (i.e. doing Continuous integration). It’s a perfect means to automate just about everything - most notably it can serve as central analysis and monitoring point for your software system, it can run various analysis tools and present detailed reports and statistics about the results.
I will outline a scheme for such an automated analysis system in an upcoming series of posts, making heavy use of free and Open Source Software from the .NET ecosystem. For now, let’s end with the conclusion, that there’s much more to software development than just writing code…
References
<iframe src="http://ads.geekswithblogs.net/a.aspx?ZoneID=5&Task=Get&PageID=31016&SiteID=1" width=1 height=1 Marginwidth=0 Marginheight=0 Hspace=0 Vspace=0 Frameborder=0 Scrolling=No> <script language='javascript1.1' src="http://ads.geekswithblogs.net/a.aspx?ZoneID=5&Task=Get&Browser=NETSCAPE4&NoCache=True&PageID=31016&SiteID=1"></script> <noscript> 
</noscript> </iframe> 
