An Approach to Software Production - Part 2

Integrate Early

System integration is often one of the most painful exercises a software developer can do. It doesn’t matter whether it’s integrating a front end with a backend or some legacy subsystem, integration usually involves more than just software development. It also involves strong communication and problem solving abilities. It’s best to make integration continuous - from both automated and human perspectives.

Our self organising, vertically organised teams start integrating from the get go. As explained we actively encourage cross functional teams that include design, front end developer and back end developers. This creates an environment that fosters strong communication and collective ownership of the entire system - a massive gain in terms of productivity, team member involvement and commitment. The subtlety here is that any member of the team can fix the problems they find - if they can. This is not about who owns what. This is about building the system in an environment where the lines of ownership are blurred. With this in mind, members are empowered to integrate “as they go”.

It’s not just the team setup that supports integration. We also make ample use of continuous integration (CI) tools such as Jenkins, Circle CI and others - depending on the choices of the team. Continuously integrating in an automated fashion is important because it catches problems early. As code is committed into the source code tree, a suite of tests are run against the code base and problems reported. Errors are found immediately and fixed quickly. With this in mind, developers can be sure that they are integrating, and thus minimising future problems, as early as possible. This shorter feedback cycle makes big sense.

Managing the Backlog

In the traditional waterfall model, deliverables are managed based on the scope and deadlines (milestones) of the project. Many things tend to be ongoing at any one time and deadlines inevitably lead to a final crunch time as cutoffs loom large. Multiple tasks are often thrown at developers and it is up to them to keep it under control. The unfortunate side effect of this situation is that team members become less productive, shortcuts are taken to get it “out the door” and ultimately a reduction in output quality is realised.

Another unintentional side effect of this mode of operation is an increase in complexity. With high levels of work in progress (WIP) the need to manage inter-dependencies and regression quickly emerges. Think about this, if we are working on many incomplete tasks it is only logical that it takes longer to achieve a complete working body of work. This means release cycles tend to be (much) longer than if we fully complete small bodies of work in a focused way (low WIP) and release as we go.

Maintaining a minimal backlog of WIP is a key point to understand because the cost of processing change is expensive.. In many projects and indeed in many aspects of life, the longer the list of things to do the more difficult it becomes to stay focused on the task at hand. It is too easy to context switch to a more “enjoyable” task. When we context switch, we lose time and the fidelity of thought. Cognitively we simply can’t seamlessly move from one task to another. Context switching must be minimised if optimal output is to be achieved.

Let me put it another way, no one can do more work than they can handle. Seeing this statement written down makes it seem obvious however in practice it is not so simple. Managing the backlog of stories and tasks being worked on is a critical skill for ensuring maximal development speed. Let’s get one thing straight, where possible we avoid parallel development that might see a team working on a large number of stories/tasks at any given time. It’s more than we (as humans) can handle. Thus, our goal with backlog management is to keep WIP to an absolute minimum.

With the aim of minimising context switching through lower levels of wip, new tasks can then only be pulled by developers based on the story and task priority and the ability to complete. The traditional system sees new work pushed onto the developer regardless of where they are. Our way is to pull tasks when the developers are ready and to maintain that flow of tasks moving through the system. It’s a subtle yet very important difference.

Release Fast

Returning again to our scope driven waterfall model that has traditionally occupied the software development space, traditional projects tend to be managed in order to deliver the scope of the project. Customer and user requirements are a secondary concern. Under this model, many times we have heard customers say “I know that’s what I asked for but that’s not what I meant”. To which developers reply, “You could’ve told me that!”. This common scenario sees both customers and developers left feeling unfulfilled.

With higher levels of WIP and their longer release cycles it’s no wonder the client isn’t getting what they wanted. Too much development goes into making a feature which makes it difficult and expensive to change later. Now we know better.

With levels of WIP minimised we can now start to deliver shorter release cycles. This is a nice side effect because it helps to provide customers more visibility of the project state. Product owners can actually see working features as they evolve. Moreover, this helps product owners prioritise the features they need now as the feedback loop is far more fluid and efficient.

However it’s not as simple as minimising WIP. The team setup and software infrastructure need to support continual this integration of new development into the existing code base. We’ve already covered our end to end team setup so we should again briefly discuss the role of automation and how it allows us to move quickly. In fact, this is one of the key components of our iterative “release fast” mantra.

To recap, old school projects in the scope driven world tend to develop code now and integrate later. This is a poor software development practice no matter which way you look at it. As mentioned earlier, in our way of working we make heavy use of continuous integration (CI) and continuous deployment (CD) tools. CI is of particular importance because it sees every incremental change committed to our main code base tested for functionality, integration and regression. This does not happen in the rush two weeks before “go-live”. It happens on every merge, every day. We hit problems early and we solve them quickly. Subsequently software development is done on working, well factored and tested code. Needless to say, the foundation from which new development can take place is far more solid.

If code is integrated, tested, well factored and client approved then what’s holding us back from releasing now. Nothing stops us - so we do it. And we do it often. In fact, because the client has been involved along the way we can start releasing and validating immediately. This incremental approach is an incredible gain over scope driven approaches.

Conclusion

This article has presented the next three aspects of the software production approach - integrating early, managing the backlog and releasing frequently. In the final part three I’ll discuss the final pieces of the system - validation and measuring the success of the product by using KPIs.