Business Software Is Hard

Essay 4

Yeah, but does it work?

What the Test Pyramid stopped telling us about business software.

I have already written about a system where adding one column to an existing screen required changes across roughly forty files. It was a large Java business application, carefully structured into many small classes, services, interfaces, repositories and mappers. There was another fact about that same system which makes the story more interesting: we had extremely high automated test coverage. At one point we had pushed it to effectively one hundred percent.

From outside the development team, that looked reassuring. Test coverage was one of the few simple numbers anybody could see. The software was difficult to change, but the tests were green and the coverage report was excellent. Inside the codebase, the picture was less comforting. Almost everything was tested through small class-level unit tests with mocks around its dependencies. The same decomposition that had produced all those implementation classes had produced tests for all of them, so changing the implementation often meant changing a matching web of tests.

The coverage number told us that nearly every line had been executed under a test. It did not tell us that the important behaviour of the system had been tested. In a system built around calculations and persistent business data, some of the questions I most wanted answered were much simpler: given these facts in the database, what numbers should this screen show? Given the database in state A, when this business operation runs, what should the state be afterwards? We had tested the pieces thoroughly while leaving questions like those surprisingly exposed.

That experience made me see the Test Pyramid in a different light. I do not think the original idea was wrong. I think we inherited a simplified memory of it, combined it with an ambiguous word, unit, and then applied the resulting rule after some of the assumptions behind it had changed.

My argument here is deliberately about database-backed business software rather than software in general. A compiler, a graphics engine, a numerical library and an internal accounting system do not necessarily need the same testing shape. In the systems I am talking about, correctness is heavily tied to persistent business facts, calculations over those facts, and the transactions and longer-lived processes that change them.

What the pyramid was correcting

The Test Automation Pyramid is usually associated with Mike Cohn’s Succeeding with Agile, published in 2009, although Cohn had discussed the idea earlier. Its familiar three layers were unit tests, service tests and user-interface tests. The context matters. The problem being corrected was an automation strategy dominated by expensive tests driven through the user interface. Those suites were slow, brittle and costly to maintain, so the advice was to move much more testing below the UI and keep only a relatively small amount at the top.

Martin Fowler’s later description makes the intermediate idea clearer. Tests can operate below the UI through a service or API surface and still exercise a substantial part of the application. His term subcutaneous test captures the idea nicely: enter just under the skin of the UI and test the behaviour without paying all the costs of driving a browser. Fowler also points out that many arguments about the exact shape of the pyramid are really arguments about what people mean by a unit.

That is not quite the collective memory I encounter on projects. The version many developers seem to carry around is simpler: lots of unit tests, fewer integration tests, and very few end-to-end tests. Unit tests are fast and desirable; integration tests are slow and expensive. The natural conclusion is to turn as much as possible into a unit test, mock everything around the subject under test, and keep the other layers small.

There are several assumptions hidden inside that interpretation. A unit is often assumed to be a class. A database is treated primarily as an external dependency to isolate from rather than as part of the behaviour under test. Integration is assumed to be expensive, and fast feedback becomes almost synonymous with isolated execution. None of those assumptions is absurd, but none of them is a law either.

The original pyramid was a corrective to too much testing at the top. It was not an instruction to test every class independently. If somebody objects that the Test Pyramid never said a unit was a class, I agree with them. My criticism is of the way the idea has often been operationalised in Java teams, not of a claim Cohn made.

The level error in testing

The word unit has much the same problem as the word module. It sounds precise while leaving the important decision unstated.

In a Java codebase it is easy to interpret a unit as a class because classes are the most obvious named pieces of the program. Once that decision is made, a testing style follows naturally. Every class gets a test. If a class depends on another class, introduce or use an interface and replace the collaborator with a mock. If a class is difficult to instantiate independently, restructure it until it can be. Repeat this across the codebase and the bottom of the pyramid becomes enormous.

There is real value in small isolated tests. An algorithm, parser or tricky piece of logic may be easier to exercise that way, failures are easy to localise, and the tests can run extremely quickly. The problem begins when class-level tests become the main evidence that a business system is correct.

This is the same level error I described in the first essay, now applied to verification. We made the class the primary unit of design and then, unsurprisingly, made it the primary unit of testing. If the more meaningful structural unit is actually a package or module, the corresponding test should often enter through that module’s public surface. Its internal package-private classes can then be split, combined or replaced without forcing the test suite to know that the implementation changed.

Tests create coupling too. A test that knows the private structure of a module is coupled to that structure just as surely as production code can be. This was part of the forty-file problem. Some of those files were tests. They were doing exactly what we had asked them to do: pinning down implementation detail. What they were not doing was making the implementation easy to change.

A package-level test is potentially much more durable because it is attached to a boundary we intend to preserve. Some external collaborators may still be mocked where that is useful, but there is no requirement that every internal class become an independently tested unit merely because it happens to exist.

A different middle for business software

Packages are only the structural part of the answer. For database-backed business software, much of the behaviour I most care about does not exist meaningfully without the database.

A physical transaction may read several relations, establish preconditions, update one set of rows, insert another, rely on uniqueness or foreign-key constraints, and depend on an isolation model when competing work occurs at the same time. A read operation may use joins, grouping, window functions and database-specific semantics to derive the figures shown to a user. Mocking repository calls can tell us that the Java code asked for the values we expected it to ask for. It cannot tell us that the SQL means what we think it means, that the production database enforces the constraints we think it enforces, or that the whole state transition is atomic.

This is where I depart from merely saying that the middle of Cohn’s pyramid needs to be larger. I am proposing a different organising unit for this domain. The middle I care about is the physical database transaction, exercised against the real type of relational database used in production.

That is not a claim that database integration testing has never existed. Of course it has. Later descriptions of the Test Pyramid explicitly include database integration tests, and plenty of teams run them. The distinction is what we organise the test around. I am less interested in proving that a repository can serialise a row correctly than in proving that an important business transaction produces the correct durable change. The database is not merely an integration point at the edge of that test; it is where much of the transaction program runs.

The basic form of the test is straightforward. Put the database into a small, understandable starting state. Run the named transaction against the real database. Inspect both what it returned and the persistent state afterwards. Given these business facts, when this authorised operation executes, then these facts should have changed and these invariants should still hold. The same approach works for reads: given known persistent facts, when this view or calculation is requested, then these are the figures the business should receive.

Concurrency needs more care. We can and should write tests for important known races: arrange two transactions so that a particular interleaving occurs and assert the expected result. Those tests are valuable regression evidence, but they do not prove the absence of every possible anomaly. Correctness under concurrency also requires reasoning about transaction scope, database constraints, isolation and the invariants being protected. A hundred concurrency tests cannot compensate for an atomic boundary drawn around the wrong work.

There is a practical prerequisite here too. I have talked about running the named transaction as though every application has an obvious list of them. Many do not. The boundary may emerge accidentally from framework propagation across a service call graph, and nobody can enumerate the transactions as design objects. That is part of the problem described in Drawing the transaction. If you cannot point to the important physical transactions in the system, that is a finding before it is a testing problem.

The old cost argument

One reason developers learned to avoid database-heavy testing was cost, and that history deserves some sympathy.

Database tests used to be awkward to provision and could be painfully slow. Shared development databases allowed tests and developers to interfere with one another. Spinning disks made I/O expensive. Starting another database instance was not something a test suite casually did. If the alternative was a millisecond test against mocks, the economic pressure toward isolation was obvious.

That cost model has changed substantially. Developers can now run the same database engine used in production locally, often in an isolated container provisioned for the test suite. NVMe storage makes local I/O cheap by historical standards, and CI systems can parallelise work that once ran serially. I have worked on suites containing hundreds of database-level tests that completed end to end in a couple of minutes. That is slower than thousands of mocked class tests measured in milliseconds, but the interesting question is not which number is smaller. It is whether the feedback is fast enough for development and what evidence we obtain in return.

Containers solve provisioning, however, not test data. Anyone who has inherited a database test suite built around one giant golden dataset knows the other failure mode: fixture files containing thousands of rows, tests depending on mysterious shared records, and one change breaking apparently unrelated tests because somebody altered row 4,712.

The starting state should be treated as part of the test’s explanation. I prefer small per-test fixtures containing only the business facts needed to understand the scenario. Sometimes it is useful to establish that state through the same transactions that create it in production; sometimes direct declarative setup is clearer because the setup itself is not what the test is trying to prove. I would rather make that choice locally than turn test setup into another universal rule. What matters is that the GIVEN is small, owned by the test and comprehensible to the reader.

AI makes the question of independent evidence more important. An agent can generate an implementation, the abstractions around it, the mocks for those abstractions and a large suite of tests that confirm that all of those generated pieces agree with one another. The quantity of tests therefore becomes even less persuasive. For database-backed behaviour, exercising the real persistent state gives us evidence against something outside that self-consistent set of generated assumptions. If an agent writes the SQL, the repository interface, the mock repository and the unit tests, I am much more interested in the test that runs the SQL against the database and checks the resulting facts.

The useful historical stance is not that the old advice was foolish. It is to ask which constraints made the advice sensible, and whether those constraints still dominate the system in front of us.

Above the physical transaction

A physical transaction is still not the whole business process.

Drawing the transaction described business processes as chains of physical transactions separated by external interactions, messages, human decisions and waits. Those larger processes are the top of the testing shape I have in mind. There should naturally be fewer of these tests because they involve more moving parts and usually take longer to arrange and execute, but they answer a question that neither a package test nor a physical transaction test can answer: does the larger business process work?

A useful business-process test might start with the system in a known durable state, perform the first business transaction, control an external response, perform the next transaction, and observe the state as the process moves forward. These are good places to test retries, rejection, ambiguous external outcomes, recovery paths and the durable state left behind for a human to resolve later.

The test does not have to be driven through the UI. Depending on what we are trying to prove, it might enter through an HTTP API, a service boundary, a test harness, or the user interface itself. The entry point is secondary. What makes it a business-process test is that it follows the meaningful process across its physical transactions and external boundaries and verifies the durable business outcome.

External systems will often be substituted or controlled at this level. That is a much more meaningful place for a test double than between every pair of internal classes. The boundary being replaced is one we genuinely do not control.

I learned the need for this level of testing from another conversation. I once worked with a product owner on an application whose automated testing I was quite proud of. When I explained how much we had tested, his response was immediate: “Yeah, but does it work?”

I realised that I could not answer him as confidently as I wanted to. I could say that the individual parts passed their tests and that the build was green. What I could not point to was a test that started from a meaningful business state, carried out the operation or process the application existed to perform, and demonstrated that the resulting business state was correct.

That was not necessarily a missing UI test. It was a missing business-process test.

I would still keep a small number of assembled-system smoke tests, and some of those will naturally go through the real UI. Their purpose is different: to establish that the application is wired together, starts correctly, can reach its dependencies, and supports a few critical journeys through the deployed interface. They are useful checks on assembly, but I would not make them another level of the pyramid. The top of the pyramid is the business process.

Coverage is not confidence

For the kind of business system I am describing, my test portfolio therefore has three meaningful levels: focused tests of modules and genuinely independent pieces of logic; a substantial body of tests around physical database transactions, reads and calculations against the real database; and fewer tests of the longer-lived business processes made from those transactions. Alongside that I would keep a small number of assembled-system smoke tests, but I see those as checks on wiring and deployment rather than another business level in the pyramid.

I do not particularly care whether the resulting picture is literally a pyramid or something else. Replacing one remembered diagram with another would repeat the mistake. The distribution should follow where the meaning and risk live in the system.

That brings me back to the one-hundred-percent coverage application. Coverage measures something real. It can tell us which lines, branches or conditions were not executed by a test suite, and that can be useful. What it cannot tell us is whether the suite asked the questions that matter.

A codebase can have near-perfect coverage because every implementation class has been exercised through mocks while nobody has tested that the database calculates the right result from realistic business facts. It can have thousands of tests and still leave an important transaction boundary unexamined. It can even be difficult to refactor partly because its tests know too much about how the current implementation is assembled.

High coverage and reliable software may be correlated on some teams, but the percentage is not the cause of correctness. Once it becomes a target, we risk optimising the thing that is easiest to measure rather than accumulating the evidence that is hardest to fake.

Future work: coverage at the right level

If modules, physical database transactions and business processes are the units that carry the meaning of the system, our tooling ought eventually to report coverage in those terms as well. Ordinary line and branch coverage can still tell us something about code inside a module, but the more interesting missing measures are at the transaction and process levels.

Once transactions become explicit and nameable, we could ask which important physical transactions have real-database tests, which meaningful outcomes have been exercised, and which known interactions have concurrency tests. A simple transaction-coverage figure would still be crude: one happy-path invocation of approve_order tells us nothing about rejection, duplicate invocation, retry or a known race with cancel_order. A useful report would expose those paths rather than compressing everything into one number.

The same idea applies to longer-lived business processes. If a process is described as a sequence of physical transactions, external interactions and decisions, business-process coverage could show which important routes through it have actually been exercised, including rejection and recovery paths rather than only the successful one.

These measures could become targets, and that would not necessarily be a mistake. If we are going to optimise for coverage, transaction or business-process coverage at least directs attention towards behaviour the business cares about. The danger is treating the headline percentage as sufficient. What I would really like the tooling to show is which transactions, meaningful transaction paths, business-process paths and known interactions have been exercised and which have not. If those measures become targets, at least they are targets aimed at the parts of the system we most need to understand.

A practical test

Take a database-backed business system whose test suite is considered good and ignore the coverage percentage for a moment.

Choose one package that represents a meaningful part of the application. Can you test its behaviour through the surface the package deliberately exposes without knowing which internal classes currently implement it?

Then choose one important physical transaction. If you cannot identify where that transaction begins and ends, note that first. If you can, put the production type of database into a small known state, execute the transaction and inspect the resulting state. Does the test exercise the real SQL, constraints and atomic boundary? For an important concurrency case, can you arrange the specific competing operations you are worried about while also explaining why the transaction scope and isolation protect the invariant more generally?

Choose one important read or calculation and start from known persistent facts. Can you prove that the figures returned to the application are the ones the business expects?

Now choose one longer-lived business process. Which physical transactions make it up, which external boundaries does it cross, and is there a test that exercises its important success and failure paths? It does not matter whether that test enters through the UI, an API, a service boundary or a test harness so long as it proves the process and its durable outcome.

Then keep a small number of smoke tests for the assembled application itself. Can it start, reach its real dependencies and complete a few critical journeys through the deployed interface? Those tests answer an assembly question rather than defining another level of business behaviour.

The Test Pyramid was useful because it challenged an expensive testing habit that had become normal. We should be willing to apply the same habit of thought to the pyramid itself. The question is not how many tests belong in each geometric layer. It is whether we are testing at the level where the system’s meaning actually lives.

References