According to the 2004 Standish Group chaos report which assesses the success of software development projects, 54% of projects were late, 18% completely failed, and only 28% were delivered on time and within budget. The average cost overrun was about 100% and the average schedule overrun was around 120%. That is catastrophic planning by anyone’s standards, and yet enormous amounts of resources are applied to get it right.
It is clear that error is endemic in the software estimation and planning process. Despite an abundance of techniques for estimation, projects get it wrong over and over again. One response to this is to attempt to increase the level of detail, the accuracy of individual estimates, the micro-management of schedules, and to apply greater control to individual processes, in an attempt to anticipate problems and respond to them promptly. Another approach is to relax control and modify the whole delivery process into one of incremental functionality. The former seems like control freakery, whereas the latter seems like a loss of project direction.
But why is the problem so difficult, and why do so many project managers find the whole area so perplexingly fluid? And why do our project management methods often fail to shape up to the challenge?
Myths about requirements
Everyone knows how to build something. You decide what it is you want, then design it, then make all the bits and assemble it. Just like a house. Some project managers still have this mental picture of software development and their planning corresponds to a traditional linear process. They create a product breakdown structure, with requirements defined for the whole product and each separate component, until eventually they have a complete set of sub-components with designs that are used to assess the resources needed to make them.
That fragmentation of the product into components is almost never adequate to capture the interconnections between the requirements. Instead it creates the illusion of a product divided and conquered, and the corresponding appearance of project control. With the apparent separation of components, the very connections between them become problematic, hidden, and underestimated, despite the apparent accuracy of the project plan.
The moment any changes are required between components, the interfaces are modified and the effects cascade, sometimes uncontrollably. And change they will, always. Requirements change because we are never in the position of building separate components which will seamlessly fit together to deliver the expected functionality. There is always a compromise. Change is inherent in software development and consequently so is uncertainty. The assumption that requirements can be captured at the outset is a myth and the best that is ever achieved is a partial set with some estimate of the likelihood that they will change.
Sources of change
Any software project that lasts more than a few weeks will see some change in the technology available for development, possible even a change in the underlying platform. As code is developed, it is possible to observe it working empirically and this throws up questions of performance, functionality, scalability, and testability.
That in turn has an influence on the rest of the project, perhaps requiring a reordering of the effort, a change in priority, a change in the scale of effort required, the use of different tools, different methods, more than likely some redesign. Any one of these makes the existing project plan unstable and forces it off schedule. Normally plans include some contingency to accommodate the uncertainty but it is usually used up in the early stages of the project.
During the course of a project it is common to hear mention of future-proofing with questions being raised about the adequacy of the existing designs. Sometimes changes in design become necessary and this adversely affects the project plan. Attempts are made to keep as much as possible of the existing work rather than redesign too much of what is already done. So the efficiency of the design tends to degrade through this process of accommodation, but what can the project manager do? They have a plan, they have milestones, they have sponsors who need reassurance.
Project Manager responses to change
One common response is to try to force developers to stick to the existing schedule and to meet the milestones, to cut whatever corners are necessary to deliver the product, just in order to preserve the integrity of the plan. After all, the financial viability of the project depends on its delivery on time and on budget. Developers can be put under intense pressure to get back on track.
As the implications of the early stage of development become apparent, estimates are seen to be less and less reliable, and so a struggle develops between the developers who are trying to offer more realistic estimates, and the project managers who consistently assume that the estimates are inflated, and consequently try to argue them down.
With some projects, it is possible to adopt a more flexible approach. Typically where the functionality can be delivered incrementally, where the customer accepts such a model, and where the architecture of the project permits a relatively clear separation between components, a more agile approach to development can be used.
This is often highly iterative with the developers committing to deliver a unit of functionality in a given time, and are then left to make the resource decisions for themselves. Where the structure of the project permits this level of developer control, the evidence says that the projects are more successful. Project managers accept a much lighter role as the function is absorbed into the development team itself.
The problem with estimates
An estimate is a calculation and in order for any developer to provide an honest figure, they need to establish some prior data. Until some development work has been done, the developer will not be able to offer any reliable estimate. This work might require a prototype, experimenting with the code for critical functionality, or examining the likely problems of testing. Once the developer has obtained some data, they can offer an improved estimate, but it will necessarily contain some uncertainty.
Any estimate is based on assumptions, some of which will be wrong. Requirements inevitably change during the course of a project, either because they prove infeasible or the customer wants something different, or because the development is constrained to use particular technology or integrate with other systems. Sometimes such changes are required by platform changes and updates. So estimates are critically related to requirements which also contain uncertainty.
The myth of future-proofing
It is common for developers to be expected to future-proof their code. Sometimes this simply means accommodating potential changes to the platform on which it runs, but frequently it also involves some measure of anticipating the integration with other future projects.
Of course, there are multiple possible futures, all but one of which will turn out to be wrong. Since there are substantial costs involved in abstracting functionality to insulate code against change, future-proofing is inherently risky. Instead of anticipating the future and getting it wrong, it is often much more cost-effective to save the expenditure and pay for necessary change when it is known.
The problem with reuse
Once we have designed a house, we can reuse the design many times over to produce more of them. In software development, there is a belief that abstracting functionality into general purpose components enables their reuse. And there have been successful technologies built on such reuse. ActiveX and the COM components of Windows are clear examples of reuse, but they have their problems.
But behind the concept is reuse is the necessary agreement to keep the interfaces to such components constant. So although the code behind the interface can change, the interface itself stays the same. This gives the impression that change can be accommodated. However it is the interface to the component that provides access to the functionality. Any additional functionality therefore requires additional interfaces.
So although components might get reused, they gradually acquire additional interfaces, increasing complexity and the maintenance overhead. Often the time required to develop additional interfaces within the context of existing functionality is not cost-effective and new components are developed anyway, creating upgrade and compatibility problems. Despite the best of intentions, reuse is often not effective.
The problem with historical comparisons
When estimating a new project it is common to look back at previous projects of similar size and complexity and look at the actual time taken. This is assumed to provide the basis for a ball-park figure but it can be seriously misleading.
Even a project only a few years old will have used different development tools, different libraries and even different programming languages, and will have worked on a different platform without current updates. It will have had substantially different requirements otherwise the old project would simply have been revised. Making the assumption that historical projects are comparable can mislead project planners into completely unrealistic estimates.
Indeterminate factors
Performance, usability, and scalability tests almost always require a working system with as near as possible, the actual working loads. This is almost always not cost-effective and so customers experience many system problems in order for developers to be able to identify and fix them. The problem is that there are simply too many variables to control in complex systems, and too much uncertainty to be able to design effective solutions.
Nevertheless, there is constant pressure in a project to adopt an optimistic outlook. Often looking on the bright side early precedes some very dark testing scenarios later.
Linear projects and iterative development
All software development involves a large degree of iteration, whether it is the constant re-testing, or the incremental addition of new functionality. But normally that takes place within some form of linear plan.
Project managers are normally trained in planning methodologies which are linear, starting from product definition and breakdown, through various stages involving designing and building to testing and delivery. And of course, all product development is linear in the sense that it moves from a concept through to a product. Iteration has to take place within that linear progression.
The extent to which iteration is used depends on the nature of the project and although agile development is much favoured by developers who want to mitigate pressure from project managers, not all projects either need or benefit from such approaches.
Those projects which have both vertical and horizontal integration, which require lower layers to have stable and proven functionality, may not be able to accommodate the constant reworking typical of agile development. Some parts of some projects definitely need to be linear.
But the converse it true as well. Fixing designs and code too early can create later rework which destabilizes designs. A balance has to be struck. The agile versus waterfall debacle has served to clarify some of the important issues in managing software projects but realistic managers understand the need for elements of both approaches.
Project constraints
Projects in the real world have to meet financial targets and despite the need for iteration in planning, there are milestones which have cash consequences. No matter how flexible the development schedule might be, normally financial planning is less flexible. There is always a tension between the needs of development, and the needs of the project sponsors.
Adaptive planning therefore has limits, even where the customers are happy to work with an incremental delivery of the product.
On balance
Iterative and linear approaches have to be mixed together on most projects to accommodate the business needs for financial planning and developer needs to accommodate change. Initial requirements, estimates and designs are expected to change so adaptive planning is essential. Neither waterfall nor agile provide the panacea because all software development involves a substantial amount of indeterminacy coupled with linear business needs, so some mixture of approaches is essential.
Estimation is a calculation involving some probability of error which needs to be managed. The use of estimation tools is helpful but not definitive. Future-proofing is often not cost-effective, and reuse sometimes involves costly compromises. Project managers who force compliance with a rigid plan also force compromises in design and functionality which can critically lower product quality and increase risk. The better approach is to accommodate change within the planning process which needs a degree of iterative flexibility.