I have spent over twenty years steeped in IT projects and about the same amount of time researching and using complexity science. I have also spent a lot of time trying to reconcile the two and have come to the conclusion that to a great degree complexity is a red herring in the software delivery environment and could cause us to tie ourselves in unnecessary knots.

It doesn’t matter whether this or that phenomena is emergent (this is a vague concept in complexity science) or at the edge of chaos (so is this) or is self-organising (guess what, so is this); it is  all irrelevant unless it helps us understand how to influence such phenomena within our projects and the truth is that with the current state of complexity science, it can’t. So getting too smart about complexity is a distraction (or red herring). In the real world of software delivery the important questions are; what does the science of complexity tell us about the behaviour of software projects and how can it help us reduce waste, time, and costs whilst increasing quality and the experience for everyone concerned .

What is Complexity Anyway?

There are two sides to complexity. Firstly there is our personal subjective view that tells us something is complex when it makes no sense, or when there is a lot of rapid change and uncertainty flying around.One of the problems is that we tend to perceive the consequences of complexity only when it becomes a problem when what we should be doing is paying attention to it throughout the project.

The other view of complexity is the more objective one of complexity science. Complexity science is all about understanding the principles of the nature of patterns of behaviour and structure and how the relationships between the components give rise to such behaviour. There is a load of real science and mathematics behind complexity science, but often this gets diluted down to unsubstantiated bullet points where the real meaning and relevance gets lost. The truth is if you want to use this stuff then you have to put some effort in.

Our perceived, or as I call it, ‘mind’s eye‘ view of complexity has little in common with the objectivity of complexity science and this difference can cause us to attempt to apply inappropriate approaches in complex environments. If you want to use what complexity science is telling us  then you have to view your environment using complexity science and not your own precepts. Click here to find out why we have to be careful when interpreting complexity science.

What is the Meaning of Complexity in Software Development Projects?

Although there is no absolute measure of complexity the science has identified many general characteristics of the structure and behaviour of complex systems. In general we can say that a complex system is one that has many interdependent parts and whose behaviour is significantly more complex than that of the individual components. Given this we can most certainly say that software delivery is complex because it has lots of components that interact in many ways and of which many of the components (that’s us) can adapt their local environment and hence influence lager scale behaviour; great! But where does that get us!

There are two aspects to complexity, one is the structural complexity of the project and this is all about the structure of each of the components and the interactions between them and much of this is defined by us when we set up and initiate the project. One of the things complexity science tells us is that in general the more structurally complex a system is the more potential it has to show increased complex behaviour. So the greater the complexity in the project structure, the greater will be the complexity of its behaviour.

The second aspect of complexity is the dynamic behaviour, or what the project does when people start doing things. The dynamic behaviour of a software project can be viewed as multi-dimensional because there are countless measurable characteristics of the project that are changing all of the time, things like, costs, number of defects, information flow, number of active tasks, amount of code, unresolved software changes and the list goes on.

If we tracked the value of one of these characteristics with time it may look something like this.

Some of these characteristics are good to track because significant change can suggest the need for additional costs, or additional resource requirements or give us an objective measure of quality and all of this part of the normal use of metrics. We may also use statistical process control on such data to identify statistically significant jumps in behaviour (shown in red) that may indicate a process change or systematic error.

The big point is here is that analysing the complexity of any specific characteristics doesn’t tell us anything. Complexity is all about the way in which the sum total of the behaviour reveals itself to us and again complexity science can help us. Complexity reveals itself in the number of occurrences of extreme events and in the varying level of certainty we can have that the project or process does what we expect it to do and that if we change something the result is what we expect it to be.

What can Complexity Science Tell us about Software Delivery?

So now we can say that we want complexity science to help reduce complexity in all its forms so as to decrease the number of extreme events and to increase our ability to predict the project’s behaviour and optimise our ability to make changes that have a predictable outcome. In practical terms complexity science is about helping us to regulate[1] the behaviour of our projects to reduce extremes and to reduce uncertainty.

However the science also tells us that there will always be a significant level of uncertainty, so get used to it. There are several principles that we can extract from complexity science that can help us regulate our projects.

Project Behaviour may not be as Complex as you think

I tend to try to keep complexity as simple as I can and for this purpose I introduced the idea of a spectrum of complexity. This is a continuum of complex behavioural characteristics that starts with highly stable and ordered behaviour then moves through various levels of complex behaviour to end with chaotic behaviour. My first contention is that in reality and by their nature software delivery projects can only span a limited range of the spectrum of complexity.

In my view it can be argued that software delivery, and for that matter any human based system can only exhibit low to medium levels of complex behaviour.  This means that they retain a reasonable level of predictability for significant periods of their lifetime. Projects in this region of the spectrum also have periods of non-extreme sensitivity to change, which means that there are times when they can have a level of predictable outcome to change.  I believe that the reasons why projects have this limited range of complex behaviour are as follows:

• Projects are creative enterprises and as such there is an inherent level of information flow and feedback, coupled with individuals carrying out tasks that can be interdependent and operate in parallel. Such environments are unlikely to generate highly stable, ordered or cyclic behavior. Very simple projects consisting of one or two people may buck this trend.

• Humans have an aversion to true chaos and in fact I would contend that they cannot exist in this domain. True chaos is a place where there is no predictability from day to day, or year to year and this is coupled with large and frequent swings in the magnitude and nature of the behavior. As individual’s this would mean we would never know what is happening or what to do about it, or what the outcome would be. To us chaos would be complete randomness, although in complexity science they are very different concepts^[1].  Being intelligent agents we would try to adapt our local environment to reduce complexity. We would also engage others to make larger scale adaptations and at the end of the day as we approach chaos we may well cease to function so that the system self regulates away from approaching chaos.

So my contention is that, although we may perceive very complex or chaotic behaviour it is in fact just normal complex behaviour. I believe that highly experienced software developers and project managers learn this through experience and hence become more rational as things around them may seem to fall apart.

How can Complexity Science Help us ?

Complexity science can help identify some general principles that can be adopted to optimise our ability to predictably interact with our projects to reduce aspects of uncertainty and these will be discussed below and more detail can be found elsewhere on this website (when I’ve written it).  There is a need for us all to increase our understanding of the nature of systems and complex behaviour so that we can adapt these principles to our own environments. More importantly the more we understand about the true nature of complexity the more we come to realise just how magnificently complex complexity really is and eventually we relinquish the false belief that we have any real control over our project’s behaviour and consequently we start to reappraise what it means to manage and lead software delivery projects and to generally interact with and influence them.

Relentless Simplification

We should be relentless in continually simplifying our projects to reduce complexity. To do this we need to understand the nature of software delivery in terms of complexity science so we can identify how, where and when we can apply simplification. It is this understanding that is the hard part and will be dealt with in more detail in another article. However in general we should be looking at simplifying the following:

• Processes; making them repeatable and automated and more robust against external influences.

• Communications; ensuring clear channels of communication and reducing noise by filtering out erroneous information.

• Change; keep changes to a minimum, keep them simple and drip feed them.

• Product; keep it simple using minimum features based upon tried and tested technologies.

• Coupling; minimise and filter external influence, ensure interfaces and interdependencies are simple and well defined.

• Create sub-systems; Focus on creating self-sufficient sub-projects that have the simple coupling to other sub-projects.

The important point when simplifying is to not only look at processes but understand the influences between components of your project because it is the influences that are driving the behaviour. A great way of understanding the dynamics of a system is to use influence diagrams.Information on influence diagrams can be found through investigating techniques in systems thinking or more originally from systems dynamic modelling.

Optimise Adaptation

Complexity science also tells us that in general having adaptive agents (people) who can adapt themselves and their local environment will regulate complex behaviour. We should strive to support and optimise localised adaptation by not over planning and encouraging, supporting and teaching people to become more adaptive.

Project Initiation

It’s important to realise just how important this stage of a project is. Complex systems are highly sensitive to their initial conditions, meaning that small differences in project initiation will make huge changes in its behaviour and can be the difference between success and failure.

Prepare for the Complexity Storm

Complexity science and a knowledge of software delivery enables us to see that a project has a progressive and natural increase in complexity and this reaches a maximum that is related to the number of simultaneous tasks being performed through the project. I call the period around this peak, the ‘complexity storm’.

If we have an overall plan and a known approach to software delivery then we can have a good stab at identifying when the complexity storm is going to hit. This fact also tells us that we should expect an increase in variation of behaviour and an increased difficulty in predicting behaviour as the complexity storm hits. There needs to be a general acceptance that during the complexity storm the project will be less predictable and change is more risky. For Agile-like projects there is a general focus on simplification and a much less sequential flow at the task level. This means that Agile does not have a complexity storm, but instead has a continual complexity shower and this leads to a more consistent level of complexity and uncertainty.

Know the Rules of the Game

All systems have inherent rules that influence their behaviour and over which we have little or no control. For example all mechanical systems have to obey the law of gravity and other established laws of dynamics. The law of gravity cannot be influenced by us (unless we fly to the moon). The inherent system rules have a major influence upon the system and in some manner help to restrict its behaviour. This is particularly true as we move toward highly complex and chaotic behaviour, because it’s the inherent rules that are the primary restriction on the system’s behaviour. We can usually determine the behaviour of these rules and this can help us understand the general dynamic nature of the system and help us develop better dynamic models.

I have not as yet found any complexity science based research that has consolidated the rules that may apply to human social projects. This may not be surprising because it’s a huge task but it is obvious that we can start to identify generic rules of human behaviour and other aspects of projects. A simple and obvious rule is that there are only 24 hours in a day and people can only work effectively for a given number of hours. This rule restricts the rate of change of dynamic behaviour and the rate at which influence can propagate through a project. For large scale projects and for changes that are to be applied across a project this ‘work day limit’ rule can have a significant impact.

There are also rules or general principles of human behaviour, for example, we can only do one task at a time, and in general we like to conform to group behaviour. These types of rules are used in agent modelling and will have a significant influence on the social dynamic within a project as well as the individual’s interaction with processes. Although for software projects we do not currently have a consolidated list of inherent rules (I may try to start one on this website) the logic suggests that we should all acquire a better knowledge (not based upon our biased experiences) of human nature so as to understand better the consequences on the dynamics of interactions within the project environment.

Choose when to make Changes

It is easier to make predictable changes outside of the complexity storm as the system is less sensitive to change, by this I don’t mean that it won’t respond to change, but that its response will be more confined and potentially more predictable . We should also keep changes small and infrequent during the complexity storm.

It is also important to realise that all systems have a response time to a change or perturbation, part of this response time is due to the natural rules of the system but it is also influenced by the behaviour of the system. When you make a change don’t just think about what you want to do and what you want to happen, but have a model of your project and try to understand how the change may propagate through the project to help assess how long it may take to respond and the sequence of disturbances that it will create.

Continual What-if Planning and Prediction

A complex system is continually changing and its level of predictability changes unpredictably and you can do nothing about this. But what you can do is monitor how long you predictions are valid for and as this window decreases you need to increase the rate at which you make predictions and re-plan. If the period in which a prediction is acceptable is decreasing then this is a sure indicator of increasing complexity. You may have times when the project is pretty unpredictable; the important thing is to not lose your head and to monitor when the level of uncertainty decreases.

What-if planning prepares a team to take action before things start to go too pear shaped. From a manager’s viewpoint it also lets the team know that you are on top of things. It is no good putting a plan together and then leaving it for a few weeks. In general plans, which are simple predictive models, need to be updated every two or three days and during the complexity storm they may need updating daily, or you may even have to forget about them for a while.

There are many ways of estimating and predicting, but one that is underused is dynamic modelling. Dynamic models do not have to be sophisticated and should be applied to critical processes. I have often modelled the testing, fix, retesting cycle using a dynamic spreadsheet, where the actual values are used to dynamically update my model parameters by taking rolling averages (see example below).

The study of systems dynamic modelling can help us understand how we can generate models of our dynamic systems using influence diagrams and then to create quantitative simulations based upon the models that can be then be used for estimation and prediction.

Improve Decision Making and Judgement

Decisions influence the project’s dynamics so we better make good ones. The problem is that our quirks of thinking make us pretty bad decision makers and add to that a lack of understanding of the complex nature of projects and it just gets worse. Our decision making process needs to be more objective and using simple tools such as defining criteria, using scoring methods and decision trees can make a major improvement in the quality of our decisions.

Leadership and Management

It is incumbent upon managers and leaders to incorporate an understanding of complexity into everything they do. Some of the aspects of management and leadership that I believe are specifically important in complex project environments are:

• Making everyone aware of the complex nature of projects, especially in understanding the importance of localise adaptation and the effects of the complexity storm.

• Being a relentless advocate of continual simplification.

• Getting the balance of planning and standardisation versus freedom to adapt just right.

• Clarifying information flowing though the project and across its boundaries.

• Being relentless in resolving problems as they arise so as to reduce perturbations.

• Presenting a cool head as uncertainty increases.

• Ensuring that facts, not emotions drive the system.

• Decoupling the team or project from external noise and interference.

A Reality Check

This is all very well and to be honest I think a lot of it is just good working practice; however it takes time and resource to do all of these things on a relentless basis. So there is a trade-off between the size and importance of your project against the effort in regulation. Agile based approaches can be effective because Agile focuses on simplification of the system and has sophisticated automated tools to help make this an effective and continual process. The big problems are with big projects where I know from my own experiences that the effort in regulation is well worth while.

[1] In the chaotic domain the system’s state still has a dependency upon its previous state so there is a cause and effect relationship, however in true randomness there is no cause and effect relationship.

[1] We also have to incorporate the science of cybernetics, which is the study of self-regulating systems.