A few days ago, I had a chat with a friend of mine about the project he was working on. That discussion made me frustrated because the team was clearly doing something that had been done way too many times before: software that is legacy right from the start. We all know how that story goes, yet we so often fail to avoid it.
The typical story
It starts with a business entity who is willing to buy custom software. They have a real need, and after doing the initial plausibility review, they start the massive task of mapping all the requirements for the software. As with any good organisation, there are several stakeholders who all have their pet requirements, plus some that got thrown in mostly because someone expected requirements. Getting the requirements right and getting them now is a big deal, as this is the step where one must speak or be forever silent. End result: a lot gets required. The requirements then get patiently prioritized into must-, should-, and nice-to-haves. All of them make some sense when considered separately, but when crammed into the same document, the end result resembles a hoarder’s basement.
The process moves on, and a vendor is selected. The vendor has browsed through the requirements and has a vague idea of what they are committing to. An architecture gets cooked up. It is rather ambitious and complicated, but takes most of the requirements into account beautifully (because that is how we techies roll). If we get overly ambitious, we plan to make a product out of it at some point, and thus add another layer of unnecessary complexity. The sales guy (who owns the project) is concerned about the overall budget, so a much-hyped CMS is tossed into the mix because it "gives us a bunch of features for free". We have now set up the stage for disaster.
The project gets underway. The best technical dudes and practices are used, there are automated tests and iterations and Scrum Masters and all that jazz. A lot of magic happens, and we finally arrive at the final phases of the project: code freeze and testing, which is code for "the period where we catch up with the stuff we couldn't do on time". We are now in Hurry Mode, and technical debt is accrued in a manner not unlike a gambling addict having a binge at a casino. The original architecture – those layers upon layers of complex beauty – gets chunks of mud thrown at it and the end result becomes a… well, something that "we'll refactor ASAP". It's not a lie if you believe it, right?
What happens next is the go-live. The development is now "finished". Refactoring doesn't happen because the allocated money has been spent and the team will soon be kicked out. In the best case scenario, one guy is left to keep the lights on; worst case, there is a handover to the support department. Whatever happens, only one thing is certain: a lot of tacit project, systems and business domain knowledge gets lost. The support department has an SLA which measures them on their speed in rebooting the system, meaning they will produce exactly what they are measured for: fast response times. Meanwhile, because the development work ended, the actual understanding of how the system works and how it should be refactored and changed slowly disappears.
The final nail in the coffin comes in the form of change requests. The guy in charge of maintaining the system is already a lame duck because of his limited knowledge of an overly complex system. Small changes with even smaller budgets are implemented in the only possible way: as quick hacks. The system now rapidly degrades and soon reaches the inevitable state - it has become too expensive/slow/scary to modify.
At this point, the need to replace the system appears and the cycle starts over.
When we talk about sustainable software development, there are two variables that are not accurately quantifiable but incredibly important: the complexity of the software and the understanding of the software. When changes to these go hand in hand, the software development pace is sustainable. When a gap emerges and complexity increases faster than understanding, the sustainability of the development effort decreases.
The first milestone: Requirements
The equation is simple: the more requirements, the more complex the system. To create a sustainable system, it is imperative to initially only accept the requirements that are really, absolutely, truly needed. Leave the rest to be implemented later, when a more complete understanding of the system is formed. This is one aspect where agile development really helps, as the iterative and incremental approach allows you to concentrate on the most valuable features first. Then again, this needs to be already understood during the purchasing process. If the scope is too broad and fixed, the project has already lost many of the benefits that agility brings with it.
The second milestone: The initial architecture
The design of the system architecture is another step where unnecessary complexity can be easily added. Once again, simplicity is the key. For a customer, it is difficult to get visibility into this part of the project unless the proposed architecture and solution is explicitly outlined somewhere, and even then it will be very difficult to steer the architectural choices.
Something to be careful of is doing product development during a delivery project. This is a sure-fire way of adding layers of complexity to the software – and it rarely ends well. A good warning sign to watch out for is if the vendor wants to keep all the rights to the developed code.
Another classic misstep is the addition of a third-party product for the wrong reasons. I've witnessed a project fail mainly because a portal product added great complexity and destroyed build and deployment times, all in the name of giving second-priority features for "free". If a content management system or portal (or similar) is selected in order to save time and money, one must bear in mind that it will add significant complexity and should only be done if it provides a big part of the "must-have" features.
The third milestone: The hurry mode
I don't know exactly why it happens, but I've never, ever seen a project that’s not in some state of hurry during its last weeks. It seems to be a law of software development. This is when the architecture usually gets broken by people trying to cram in too many features in too little time. Every quick hack inevitably increases complexity.
Considering that projects usually have little wiggle-room in terms of budget or schedule, in order to avoid making legacy code, we need to understand that the final rush will happen and prepare to mitigate it.
For example, we developers are often optimistic and happily let the customer bring in big changes during the project if the schedule seems relaxed. That’s fine – we are agile, after all – but why do we forget to remove something from the scope when accepting the change?
The fourth milestone: The handover
This is a curious step as complexity doesn't increase, but the understanding of the system gets decimated. The effect is same: the sustainability of development suffers. Because of budgetary constraints, it is seldom possible to keep the original team on board, so we need to look into other ways of addressing the issues.
This is a topic that we, at Futurice, have put quite a lot of thought and experiment into with the Lifecycle Management concept. We've discussed it in more detail in a series of earlier blog posts. The short summary is that in order to retain maximum knowledge, the maintenance team needs to participate during the development project, and development must not cease when the software goes live. This way, the initial time-to-market will be faster, less complexity will be added in form of features that are not really needed, and the money that got spared can be used for narrow-bandwidth continuous improvement of the system. Lower priority features can be added during post-production development.
The final milestone: The change requests
We all recognize the following conversation:
- "It's just a little change, it can't cost much"
- "Sure, that's easy to add. Should take a few hours".
Often making a few changes is easy, fast and cheap. But again, every quick hack adds complexity, and there will never be budget for refactoring. There is no single action I can think of that would address this properly, but if we've managed to pass the previous milestones without too much damage and have a sensible maintenance and post-production agreement in place, we just might be able to keep the engine running properly.
Building custom software solutions that can be used, maintained and developed for years after go-live is possible and happens constantly. To do so with limited resources is very difficult. It is not enough to get the technologies, the team and the work practices right. The relationship between the complexity of a system and the knowledge about the system needs to be well understood. When we take care to avoid the complexity-increasing or knowledge-decreasing pitfalls during the entire lifecycle of the custom software, we will be able to craft software that doesn’t expire before the business case it was made for expires.