An Approach to Software Production - Part 1

Developing enterprise-level software is very difficult and failures are extremely expensive, so reliable ways of working that really deliver results are vitally important. Standard practices are luckily changing for the better.

There’s a lot to manage in an enterprise-level software development project and challenges come from a number of different aspects:

• Extensive product requirements
• Many stakeholders
• Complex range of legacy technologies
• Mission critical, strategically important, or both
• Fast and flexible development within existing software management processes

All easier said than done.

The old-school approach

Traditionally large, complex software projects have been managed using structured methods based on analysis, design, coding and testing - in that order. Developers are typically organised in functional teams and they usually have little visibility the workings of the other teams they deal with. Integration is not continuous, Release cycles are long and iterations were slow. Needless to say, projects operating under in this way often run over budget and deliver late.

In the beginning of my career as a software engineer, this was the standard practice. A number of projects were delivered but it was not easy and the long feedback cycles often meant the significant mistakes were made whilst expensive redevelopment was common. Integration was a nightmare.

A leaner approach

My first exposure to lean principles was in 2002 whilst I worked in the production planning department of a German research institution. Lean manufacturing was hot and its application to the software domain was often by way of a set of principles collectively called Extreme Programming. Rightly or wrongly, in my mind this was the beginning of Agile methods in a software development context. Start from the point of view of the customer and deliver those features which lead to the customer’s satisfaction. The idea of Extreme Programming (XP) was that it improved a software project in five essential ways; communication, solution simplicity, continuous feedback, respect between customer and developer, and confidence to move forward with new features and development. However it was widely misunderstood and difficult to implement in most environments.

In stark contrast to the traditional waterfall approaches, XP methods called for continual user feedback and iterating functionality based on that feedback. Requirement specification documents without end became a thing of the past. In fact, Extreme Programming was incorrectly associated as a methodology which called for no documentation. It does in fact call for documentation but only for documentation that adds value.

Although the roots of its history stretches back to the 1980s, the mainstream emergence of Scrum as a methodology in the early to mid 2000s saw it as far more palatable to the corporate environment. It’s perceived structure was stronger and it’s documentation process was clearly defined. Corporates took note and soon enough many were calling their projects Agile - something that couldn’t be further from the truth. Although not perfect, the value of agile methods could be seen as a way to tackle the difficulties surrounding software development.

Over the years, we’ve developed a set of ways of working which borrow heavily from lean and agile methods. Without a doubt this has been a large contributor to our track record. In this article, I will attempt to explain the management methodology that we use to run software projects. The methods are simple however it is our discipline in execution that has made it so successful. In point form, this means:

• Co-creating ideas
• Organising around self-organising, end-to-end teams
• Integrating early
• Managing your backlog carefully
• Releasing software fast
• Validate continuously
• Using KPI’s to measure success

Co-Creation

This is where it all starts. Here we aim to tear down the functional silos and bring people together to create interdisciplinary teams. Thus supporting co-operating in a way that produces a mutually valuable customer outcome. The important thing here is getting people collaborating to focus on value and how to maximise it in terms of the project, the business and importantly the end customers. Product owners, designers, developers, quality assurance and analysts all in one room, sitting together, building. You can probably start to imagine how this facilitates communication between the project stakeholders. Validation starts from the get go and our experience shows that if you want efficiency and the best outcomes, this is the place to start.

In fact, co-creation encompases a broad variety of stakeholders. Business people use co-creation to identify value whilst working in a customer centric way, engineers use it to incorporate design thinking into their agile development methods, and experienced service designers use co-creation as a way to better understand the problem worth designing for. The reason co-creation is so powerful is because it brings together diverse opinions. Diversity of opinion is what truly drives stronger outcomes because it supports a far higher level of efficiency in the end product. When opinions align, as often happens inside functional silos, inefficiency is typical and outcomes are subsequently hindered.

In other words, the real power of co-creation lies in the fact that it brings multiple views of the world into the analysis process. When the stakes are high it is dangerous to project one’s own understanding of what is right and wrong onto the solution at hand. No matter how smart or how widely read, one views the world through a particular frame and we look to confirm those views in the information we seek and the solutions we produce. Co-creation is the simplest way to start working around this natural human bias so we use it extensively.

Self organising, end to end teams

Traditional development teams setup based on function. That is, Database administrators, front end developers, back end developers, network administrators are each organised into functional silos. Intuitively, this makes sense as it allows project members to specialise whilst project managers can optimise their teams in terms of costs. Unfortunately, this couldn’t be further from the truth. The most visible consequences of this are longer lead times and higher energy spent coordinating activities. Further exacerbating this is each team has very little visibility or understanding of how their outputs interact with the other aspects of the system. Waste becomes common as handoffs often lead to re-work, quality issues, bottlenecks and delays. It doesn’t take much insight to realise that all of these issues lead to increase costs - in contrast to what we are actually trying to optimise in this setup.

The developers who work with us vary in their experience but overall the common thread between them is that they are good at what they do. Rather than organise by function, we organise teams vertically so they are cross functional - a speed optimised setup. With this in mind, we provide our teams with clear constraints with which they can work within and it is then up to them to organise themselves within this boundary to choose the best way forward to get a project done. With that comes a lot of responsibility and trust but our experience over the past years has shown that in this setting people tend to deliver their best. We encourage teamwork and collaboration so that project members can play to their strengths. It’s a potent mix but some might consider this as a leap of faith - one we are happy to make. Give project members the room to achieve along with the resources they need and the chances are that the outcome will be exceptional.

In my experience, horizontally organised teams as described above (cost optimised) more often than not waste considerable time on integration between the layers. In fact, the experience is one I hope I won’t have to work in again. Vertically organised teams (speed optimised) on the other hand are rewarding work in and progress is rapid because integration happens as the software is developed. Integration almost feels like a non-issue. Can you imagine?

Conclusion

Here I’ve presented the first two aspects of an approach to software production - co-creation and team setup. In part two I’ll discuss integrations, the backlog of tasks and getting your releases out the door pronto.