Business Software Is Hard

Essay 9

When Agile becomes process

What survives when a team keeps the ceremonies and loses the reasoning.

I have worked in quite a few organisations that describe themselves as Agile, and the pattern is often surprisingly similar. Every two weeks there is a planning meeting. Stories are discussed, assigned story points using the familiar Fibonacci-like scale, and debated until the team agrees how much work can fit into the next fortnight. Quite often the work is then divided between individuals, who go away and largely work on their own.

Each morning they come back together for fifteen minutes to say what they did yesterday, what they are doing today and whether they have any blockers. Two weeks later there is a retrospective. Sometimes useful things come out of it, but often the same observations are made and little changes. The team has sprints, stories, points, stand-ups, planning and retrospectives, so it has all the appearance of Agile software development. What is less obvious is how much adaptation is actually taking place.

I have written before about our habit of keeping a practice and losing the reasoning that produced it. Agile seems to be another example. We preserve the visible parts: the meetings, boards, points and cadence. The less visible parts, such as people working towards a common goal, learning together and changing the plan when they discover something, are easier to lose. Putting the process in place may be a sensible way to start, but if it works I would expect a team eventually to need less of the scaffolding rather than more of it.

Agile does not mean fast

Agile did not appear in a vacuum. The people who met at Snowbird in 2001 came from Extreme Programming, Scrum, Crystal, DSDM and several other approaches, and the history of the Agile Manifesto describes them as looking for alternatives to the heavyweight, documentation-driven development methods of the time. The Manifesto did not say that processes and plans had no value. It said that individuals and interactions, working software, customer collaboration and responding to change mattered more.

I think “responding to change” is the bit that gets lost most easily. Agile is often used to mean fast, modern, or simply working in two-week chunks. But the useful distinction is whether you can change what you are doing as you learn more. If the plan is effectively fixed before the work starts, dividing it into sprints does not create much agility.

Scrum is useful here because the Scrum Guide is more goal-oriented than many implementations I have seen. The Product Goal gives the team a longer-term objective, and the Sprint Goal gives the current Sprint a purpose. The detailed work can change as more is learnt, provided the Sprint Goal is preserved.

That is quite different from taking a project plan, turning it into a backlog, selecting a fortnight’s worth of tickets and assigning those tickets to individuals. In that version the sprint is mostly a container for pieces of an existing plan.

Five people can meet on Monday, divide the work between themselves and spend most of the week independently completing their parts. They may be well coordinated, but that is still largely division of labour. Scrum’s emphasis on a Sprint Goal points towards people contributing to the same outcome and adjusting the detailed work when necessary. That does not require constant conversation; software development often needs concentrated individual work. What matters is that people are working towards the same outcome and that useful information reaches the people who can act on it before the relevant decisions are closed.

Which practices survive

What I find striking is that the Agile practices do not all decay in the same way. Daily Scrums tend to survive. Planning survives. Story points often survive, and velocity may be tracked quite carefully. Sprint Goals are easier to lose. Retrospectives may remain on the calendar while producing little change. A Product Goal may exist somewhere, but not necessarily as something the team actually uses to make decisions.

There is an organisational reason for this. Practices that produce information for people outside the team have something holding them in place. A programme manager needs to know whether a date is threatened. A delivery manager needs some view of progress. A project manager needs material for a report. A daily meeting in which each person says what they did yesterday is useful for that purpose, even though the stated purpose of the Daily Scrum is to inspect progress towards the Sprint Goal and adapt the plan.

The familiar questions about yesterday, today and blockers were already only examples in the 2017 Scrum Guide and disappeared from the 2020 version, yet that reporting format remains common. It survives because it solves a real problem for the surrounding organisation. The same can happen to estimation. Story points and velocity become inputs into forecasts and capacity planning. There is nothing wrong with forecasting, but a forecast can gradually turn into a commitment, with the assumptions and uncertainty that produced it forgotten.

The practices that only help the team are less well supported. Nobody outside the team necessarily needs a Sprint Goal, and no weekly status report fails because a retrospective did not lead to a useful change. The meeting itself is visible, so the meeting survives, but the reason for having it can quietly disappear.

The Agile practices most likely to survive are the ones the surrounding organisation depends on; the ones that exist mainly to help the team adapt are much easier to lose.

Six practices in a column. The Daily Scrum, Sprint Planning, and story points and velocity each have a solid arrow to something the organisation consumes: status for the report, a view of progress, and a forecast and capacity model. The Sprint Goal, the retrospective and the Product Goal have a dashed line that runs out and stops, grouped under a note saying nothing outside the team depends on them, so nothing holds them in place.
Nothing here is a judgement about how useful a practice is to the team; every one of them is meant to help it inspect and adapt. What differs is whether anybody outside the team would notice if it stopped.

That pattern becomes easier to understand once Scrum sits inside a larger project structure. If the project owns the important commitments and decisions, the surrounding organisation has a use for the practices that produce status and forecasts, while the team has much less room to use the practices intended to support adaptation.

Scrum inside project management

Large organisations genuinely need commitments. Customer promises, regulatory deadlines and dependencies on other programmes are real, and senior management reasonably wants to know when important work is likely to finish. The answer is not to tell the business that planning is somehow un-Agile.

The tension comes when a project already has a delivery date, an agreed scope and milestones stretching months into the future, while the development team underneath it is described as Agile because it works in two-week sprints. It is worth asking what that team is actually free to adapt.

Sometimes the answer is not very much. The scope has been committed, the date has been published and important technical choices may already have been made. What remains is largely the order in which a set of tickets is completed. In that situation Scrum has effectively been placed inside conventional project management. That may even be the right thing to do for some work. If we are building something very similar to six systems we have already built, there may be little value in pretending the work is mostly discovery.

The difficulty is when both scope and date are treated as fixed while the work is still producing important new knowledge. The uncertainty has to go somewhere, and usually the team is expected to absorb it. Discoveries then look like delays rather than information that should cause the plan to change.

This is also where the language of commitment becomes unhelpful. A project plan produced early in the work may be a reasonable forecast: given what we know now, this is what we think the route, cost and date look like. Later, the qualifications disappear and somebody remembers the forecast as a commitment. Engineering is then judged against knowledge it did not yet have.

How much you can sensibly commit to depends on how much you already understand. A team building the seventh version of something it has built six times before is in a very different position from a team that has just inherited a fifteen-year-old system whose original developers have gone. Scrum’s own use of commitment is useful here: the Sprint Goal is the commitment for the Sprint Backlog, while the detailed work can still change as the team learns. That is different from committing in advance to every ticket selected at planning.

Starting with process

There is a reasonable defence of putting the process in first, particularly when somebody new joins a team and is trying to improve the way it works. If nobody reflects on how the team is working, scheduling a retrospective creates an opportunity that did not exist before. If people rarely coordinate, a daily meeting may help. If work arrives as a stream of unrelated tickets, asking for a Sprint Goal may encourage people to think about what they are collectively trying to achieve.

That is a sensible place to start. It is difficult to develop a habit before there is any occasion on which to practise it, and Agile writing has often described this kind of progression using Shu-Ha-Ri: begin by following the form, then adapt it as understanding develops.

The problem comes when the starting point becomes permanent. A team can still be running exactly the same meetings in exactly the same way five years later. Other parts of the organisation may also have started depending on them: velocity feeds a forecast, story points feed a capacity model, and the stand-up becomes a convenient place to collect status. What began as scaffolding for the team becomes part of the reporting infrastructure around it, which makes it harder to remove even when the team no longer needs it in the same form.

Moving up

So after introducing the process, I would ask what has changed. Has the team started working towards goals rather than merely completing tickets? Can it change the detailed work when it learns something? Has any ceremony become unnecessary or changed shape? If nothing changes except that the meetings are now firmly established, the process has not led to the behaviour it was supposed to encourage.

One way of thinking about the progression is:

These are not official Scrum levels, and I would not expect every team to move through them neatly. They are just a way of distinguishing between having the process in place and developing the behaviour the process was meant to support.

The awkward part is usually the Sprint Goal, because teams are used to thinking in tickets. A list of tickets fits neatly into two weeks; a useful goal does not always do that. If the goal is achieved after a week, the team can continue doing useful work towards the Product Goal. If the natural goal looks more like three weeks of work, the question is whether there is a meaningful intermediate outcome rather than inventing a goal which is just another name for half the ticket list. If this happens repeatedly, it may also be telling you something about the cadence. Scrum deliberately uses fixed-length Sprints, so changing the Sprint length whenever the work changes is no longer really Scrum. But there is no reason to treat two weeks as a magic number.

Less scaffolding

The same progression should be visible in the ceremonies. A team which only reflects because a retrospective appears in the calendar has not yet internalised the behaviour. A more mature team will raise problems and suggest changes when they notice them, while the scheduled retrospective can remain as a backstop. The Daily Scrum can become less important for the same reason if the team already coordinates when coordination is needed.

At some point this can start to look more like a Kanban-style way of working. I do not mean that Kanban is simply a higher level of Scrum, because it is a different method. But it can demand more maturity than beginner Scrum. Scrum gives a team regular prompts to plan, coordinate, inspect and reflect. A freer flow-based system provides less of that scaffolding, so the team has to recognise for itself when work in progress is too high, when priorities need to change, when somebody needs help, and when it is worth stopping to discuss how the work is being done.

That is why Kanban is not necessarily the easy alternative for a team that finds Scrum ceremonies tiresome. Without the underlying habits, a Kanban board can become little more than a queue of individually owned tickets. The lighter process works better when the team has already developed the judgement that the heavier process was supposed to encourage.

There is a technical equivalent. If a team is going to adapt its implementation freely as it learns, it needs to know whether the business behaviour still works. Regression tests around business transactions or other stable business boundaries are particularly useful because they let the implementation underneath change. Tests tied closely to classes, methods and current implementation details can make that harder, because changing the design also means changing the tests even when the business behaviour has not changed.

Those tests do not make a team Agile, but they make adaptation safer. The team can change how the system is built while preserving what it is supposed to do.

I would not assume the progression ends with more perfect Scrum. Scrum can be a good way to create occasions for planning, coordination and reflection, particularly for a team that does not yet do those things well. A mature team may eventually need less of that scaffolding, provided it has actually learnt the behaviour rather than merely stopped holding the meetings.

Moving up the progression should mean needing less external prompting to behave in an Agile way: work towards goals, share responsibility, reflect when something can be improved, and change the plan when what you learn says the old plan is wrong.

We have become quite good at performing Agile. The harder part is preserving what the practices were meant to achieve.

A practical test

Take your team’s current sprint and try to state its Sprint Goal without looking at the board. If the goal turns out to be the list of tickets, ask what the team would be free to change on Wednesday if it learnt something inconvenient on Tuesday.

Then go through the ceremonies one at a time and ask who outside the team depends on what each one produces. The ones with an external consumer will survive whether or not they help the team. Which of the others would anybody notice if they stopped?

Look back at the last three retrospectives and find something that actually changed as a result. If that change is hard to name, the meeting has outlived its reason while keeping its slot.

Then look at the plan the team sits inside. What is treated as fixed: the scope, the date, or both? When the work produces new knowledge, where does the uncertainty go, and does a discovery arrive as information or as a delay?

Finally, take a recent refactoring in which the business behaviour did not change. Did the tests have to change anyway? A team cannot adapt its implementation freely while its tests are pinned to the implementation it already has.

References