Business Software Is Hard

Essay 3

Drawing the transaction

Why is writing business software so hard? Essay three.

The previous essay recovered a way of drawing the program as a handful of meaningful modules. That tells us what the software is made of, but it leaves another question unanswered: what changes together when the business moves from one valid state to another?

I worked with a colleague who would repeatedly pull design discussions back to the entity model. He was much less interested in the structure of the application code above it than I was. He wanted to understand the facts the system persisted, how they related to one another, and how they changed. I think he was right about something important. Classes and frameworks come and go, while the persistent data often remains a recognisable description of what the business knows.

Later, we were discussing a new system with its entity model in front of us. The entities and relationships told us what state could exist, but they did not explain how a business process would change it. I started numbering some of the relationships and annotating the parts of the model that each step acted upon. A query established something about the current state. A later statement updated one part of the model. Another inserted a related record.

One drawing was not enough. We ended up drawing several because the process consisted of several separate transactions, with calls to external systems, messages, waits and decisions between them. We had started with the persistent facts and found ourselves drawing the programs that changed them.

The larger process

Opening an account, settling a trade or onboarding a client may take minutes, hours or days. A person may approve something, work may wait in a queue, and another user may complete the next step tomorrow. It would make no sense to hold one database transaction open across the whole thing.

The process is often better understood as a chain of smaller database transactions with interactions between them. Each successfully committed transaction makes its changes durable; together they move the process forward. The process may branch, repeat or include parallel work, but the same questions apply at each transaction boundary: what changes together, and what has to happen before the next change can begin?

Three database transactions — capture order, record payment, release order — drawn left to right, each inside its own dashed atomic boundary. A call to an external payment provider sits between the first two; a human approver's decision between the last two. A bar beneath the row is solid under each transaction and absent in the gaps, marking where the atomic guarantee holds and where it ends.
Each box here is one atomic boundary; later in this essay we open one of them up. What the picture is for is the gaps: the bar is solid only where a transaction guarantees all-or-nothing, and the questions that are easiest to miss all live in the spaces between.

The gaps are often where the harder questions live. A local transaction commits, then we call an external system. What happens if that system succeeds but our call times out? If someone has to resolve the uncertainty tomorrow, what durable state tells them what happened and what remains to be done?

Drawing the process gives us somewhere to examine delivery, retries and recovery before choosing mechanisms for them. The deployment view shows where the participating systems run. The process view shows how the work moves between them. The transaction boundaries show which changes can commit or roll back together. Calling an external system while a local transaction is open does not make its effects part of that rollback.

Making the transaction visible

Inside each transaction boundary is a program that observes state, makes decisions and performs changes. In a typical layered application, its implementation may pass through a controller, several services, repositories and an ORM. The transaction exists at runtime, but there may be nowhere we can inspect it as a whole.

In the applications and tools I have worked with, reconstructing that program has largely been left to the engineer. I can navigate classes and find callers, but I do not get the complete transaction programs laid out beside each other. Before I can reason about them, I have to work out what they are.

The whiteboard drawing gave us a representation of one of those programs. It showed the significant statements and decisions, and the persistent state they observed or changed. The database optimiser could choose join orders and access paths within a statement; our drawing showed the logical steps of the transaction.

Suppose an operation approves an order. It checks whether the order is awaiting approval, updates its status and inserts an approval record. Put the relevant entities on the board and annotate them with those steps. The drawing below is a small example: the solid line shows the data relationship, while the dashed arrows show the transaction’s actions.

The approve order transaction drawn beside two entities, order and approval, joined by a solid line showing that an order has an approval record. Three numbered steps run from the transaction to that state: step one reads the order's status and requires it to be awaiting approval, step two sets the status to approved, and step three inserts the approval record.
Steps 1–3 execute within one database transaction. The relationship line describes the data model; the numbered arrows describe this operation. Nothing here settles what happens when two attempts read that same status at once — that is the argument the picture is meant to start.

Putting those statements inside one transaction means the order update and approval record commit together or are rolled back together. It does not, by itself, establish that the approval is valid under concurrency. If two attempts both read that the order is awaiting approval, what prevents both from approving it? The answer depends on the statements, constraints and isolation or locking rules we choose.

That is already a useful question exposed by a very small drawing. Someone can point at the first read and ask how its result is protected. We might discover that the update must itself require the expected status and that the operation must check whether it actually changed a row. The drawing has not proved correctness, but it has made an assumption available for examination.

A short description alongside it can record who may run the transaction, what must be true when it finishes, and the concurrency and failure behaviour it relies on. In this example, we would want the approved status and approval record to agree. We would also want to know what a caller receives when the order can no longer be approved.

The drawing can sit beside the SQL. It need not reproduce every expression or implementation detail; it should let us connect the statements to the business facts they establish and change. If an audit record belongs in the same atomic change, it belongs in the picture too. If something happens later, that boundary should be visible.

Putting transactions beside each other

We can draw one important transaction over six relevant tables, omit hundreds of other transactions and tables, and still have a truthful picture of that operation. That makes a large system approachable one transaction at a time.

Making another transaction visible lets us put them beside each other. Approval and cancellation may each look reasonable alone. Together, they prompt a more demanding question: can their execution overlap in a way that leaves an approved order with its stock reservation released?

We can trace what each operation observes, which changes conflict, and what prevents an outcome neither intended. We might find that both rely on the same order status, or that one checks a fact which the other is allowed to change. Those are concrete dependencies we can investigate in the statements and the database’s concurrency behaviour.

The representation could be SQL scripts, diagrams, or both. A complete script may be the best way to examine the exact conditions and changes. A diagram may make the relationships easier to discuss. Several scripts placed side by side may be enough to see a problem that was almost impossible to follow through separate call paths.

This is the reason to make the transaction a design object. Without a usable representation, understanding even one operation requires reconstruction. Reasoning about several interacting operations then depends on keeping those reconstructions in our heads. We should not be surprised when important assumptions escape examination.

The representation does not perform the reasoning for us. It gives us a practical place to start, and something another engineer can question. We can retain the assumptions with the transaction, revisit them when another operation is added, and compare the intended behaviour with what the implementation actually does.

Transactions across modules

Transaction boundaries also give us evidence about module design. Often a transaction sits inside one module and is comparatively easy to understand. But some business operations span distinct responsibilities even when the modules are well designed.

Consider an online shop with separate modules for inventory and customer accounts. Cancelling an order may need to release reserved stock and adjust the customer’s account as one atomic change. Inventory and customer accounts still have good reasons to be separate modules: they own different responsibilities and hide different decisions. The cancellation brings them together because the business requires those changes to happen together.

We should expect transactions like this in a substantial business system. The clue to poor boundaries is the pattern of crossings. If transactions repeatedly cut across much of the system, pulling the same modules into operations that ought to have a more local home, perhaps related responsibilities have been scattered across several packages.

This takes us back to the first essay and Parnas’s approach to modularity. Drawing the transactions gives us concrete evidence with which to question the decomposition. Is this a necessary collaboration between distinct responsibilities, or have we separated things that belong together? Where does coordination belong, and is one module reaching into state another is supposed to own?

The two views describe different decompositions of the same system. The module view asks which responsibilities and decisions belong together over the life of the software. The transaction view asks which observations and state changes must belong together at one moment.

A system can have beautifully named packages and still permit states the business regards as impossible. A transaction crossing several modules can be exactly right because the invariant spans those responsibilities. Transaction design deserves at least as much attention as package design in systems whose main purpose is to maintain important business state. It gives us a way to examine whether the structure supports the work the system must perform.

A practical test

Take one important business process in a system you know well. Start by drawing its database transactions and the interactions between them. Where does an external system become involved? Where does the atomic guarantee end? What happens if the outcome of an external call is unknown?

Choose one transaction and draw its significant statements and decisions over the relevant part of the entity model. Show what it observes, what it changes, and what must remain true afterwards. Record the authority and concurrency assumptions on which it depends.

Put another transaction beside it, choosing one that may inspect or change the same facts. Try to explain what happens when their execution overlaps. Then compare their boundaries with the module view. Look for necessary collaborations and for patterns suggesting that related responsibilities have been scattered.

Finally, compare the drawings with the implementation: the actual statements, commit boundaries, constraints and failure handling. Starting away from the code helps us describe the intended operation; returning to it tells us whether that is what we have built.

The useful result is something two engineers can put between them and reason about. We can point at a read, a change or a commit boundary and ask a precise question. The transaction was always there. Now we have a way to examine it.

References

Background

Where the detail lives