Business Software Is Hard

Essay 4

Yeah, but does it work?

What we stopped asking of our tests. Essay four.

I have already written about a system where adding one column to an existing screen required changes across roughly forty files. There was another fact about that large Java application: we had extremely high automated test coverage. At one point we had pushed it to effectively one hundred percent.

From outside the development team, that looked reassuring. Coverage was one of the few simple numbers anybody could see. Inside the codebase, almost everything was tested through small class-level unit tests with mocks around its dependencies. Changing the implementation often meant changing a matching web of tests.

The coverage number told us that nearly every line had been executed under a test. It did not tell us whether the important behaviour had been tested. Given these facts in the database, what numbers should this screen show? When this business operation runs, what should the database state be afterwards? We had tested the pieces thoroughly while leaving those questions surprisingly exposed.

On another application, I once explained our automated testing to a product owner. I was quite proud of it. His response was immediate: “Yeah, but does it work?”

I could explain which tests passed more confidently than I could demonstrate that the application performed the business operation he cared about. We needed a test that started from meaningful business facts, carried out that operation, and checked the result.

Those experiences changed how I thought about testing database-backed business software. In these systems, correctness depends heavily on persistent facts, calculations over those facts, and the transactions and processes that change them.

What the pyramid was correcting

The Test Automation Pyramid, commonly associated with Mike Cohn’s Succeeding with Agile in 2009, challenged an expensive habit: relying heavily on tests driven through the user interface. Its unit, service and UI layers encouraged teams to test more behaviour below the UI. Fowler’s later description explicitly included tests that exercise substantial application behaviour through a service or API.

The version I encountered in Java teams was narrower: lots of isolated unit tests, fewer integration tests, and very few end-to-end tests. A unit usually meant a class, its collaborators were mocked, and the database was treated as an expensive dependency to keep outside the test. That was how we applied the pyramid, rather than something its original authors required.

In the business systems I have seen, we have often made that trade-off badly. We have optimised how quickly tests execute while weakening what they can tell us about whether the software works. Replacing the database with mocks removes the real queries, constraints and transaction behaviour from the test’s scope. Those things still need testing somewhere. When they receive little coverage elsewhere, we have bought speed by leaving important behaviour untested.

Fast feedback helps us find and correct mistakes while we work. Its value depends on what the feedback tells us. A fast green build is poor reassurance if the tests cannot detect the failures that matter to the business.

Tests create coupling too

The word unit has much the same problem as the word module. It sounds precise while leaving the important decision unstated.

Small isolated tests have real value. An algorithm, parser or difficult calculation may be easier to exercise that way, with fast execution and failures that are easy to localise. The problem begins when class-level tests become the main evidence that a business system is correct.

This is the level error from the earlier essays applied to verification. We made the class the primary unit of design and then made it the primary unit of testing. If the meaningful structural unit is a package or module, its tests should often enter through the public surface it deliberately exposes. Internal classes can then be split, combined or replaced without forcing the tests to follow the refactoring.

Tests create coupling too. A test that knows the private structure of a module is coupled to that structure just as surely as production code can be. Some of the forty files were tests. They were pinning down implementation detail and making that implementation harder to change.

Testing through the module boundary leaves room for mocks where they help. It removes the assumption that every internal class needs an independently isolated test merely because it exists.

Testing the transaction program

Much of a database-backed application’s behaviour does not exist meaningfully without the database. A transaction may read several relations, establish preconditions, update rows and insert others, relying on constraints and isolation to protect business invariants. A query may use joins, grouping and window functions to derive the figures shown to a user. Mocking repository calls cannot establish that this SQL means what we think it means.

The transaction program described in the previous essay gives us a useful organising unit. Put the production type of database into a small, understandable starting state. Execute the named transaction. Inspect what it returned and the resulting database state. Given these facts, when this authorised operation runs, these changes should occur and these invariants should still hold.

Tests can inspect the changes through the same connection and roll back for cleanup; the code managing transaction completion can be tested separately. Module tests and transaction tests can also overlap. One transaction may coordinate several modules, while a single module operation may constitute the whole transaction. One test can provide both kinds of evidence where their boundaries coincide.

The same approach applies to reads and calculations. Given a few known orders, payments or ledger entries, does the query return the figures the business expects? Checking that the result is non-empty tells us very little about whether the calculation is right.

Expected results need a basis outside the implementation: a business rule, a worked example or an independently checked calculation. An agent can generate code and tests that share the same misunderstanding, just as a developer can. Running against the real database checks its behaviour; the worked example tells us whether that behaviour is right.

Concurrency needs deliberate treatment. We can arrange competing transactions so that an important interleaving occurs and assert the expected outcome. That gives us regression evidence, while reasoning about the statements, constraints, isolation and protected invariants explains why the design should be safe more generally. Running many schedules cannot replace understanding the assumptions the transactions make.

There is a prerequisite here. I have said “the named transaction” as though every application has a clear list. Many do not. The boundary may emerge from framework propagation across a service call graph. If nobody can identify the important transactions and their boundaries, that is already a finding. Making them visible lets us decide how to test them.

Making database tests practical

I have worked on suites containing hundreds of database-level tests that completed end to end in a couple of minutes. That was fast enough to use during development, and those tests exercised the real queries, constraints and business state changes.

Shared development databases and difficult provisioning gave database tests a deserved reputation for being awkward. Local instances, containers and faster storage have made other arrangements practical. Costs still need measuring in the environment we use, but avoiding the database should not be the unquestioned default. We can run focused tests while editing and select relevant database tests for the work in progress.

Containers help with provisioning; they do not solve test data. A giant golden dataset creates its own maintenance problem: thousands of fixture rows, mysterious shared records, and one change breaking unrelated tests because somebody altered row 4,712.

I prefer small per-test fixtures containing the business facts needed to understand the scenario. Sometimes production transactions are the clearest way to establish that state. Sometimes direct declarative setup is clearer because the setup itself is not what the test is examining. The GIVEN should belong to the test and explain it. A reader should not need to understand an entire sample company to check one calculation.

Does the larger process work?

An individual database transaction is still not the whole business process. As the previous essay described, a process may involve several transactions connected by messages, external calls, human decisions and waits.

A process test can start from known durable state, perform the first operation, control an external response, and observe how the next operation moves the process forward. This is where we can exercise rejection, retries, ambiguous external outcomes and recovery, including the state left for a person to resolve later. There will usually be fewer of these tests because they involve more work to arrange and execute.

They need not enter through the UI. An API, a module boundary or a test harness may provide a suitable entry point. What matters is that the test follows the process across its transactions and checks the business outcome. Behaviour implemented in the UI still needs its own tests; entering below it only covers the behaviour that route exercises.

External systems will often be controlled or substituted so that failures can be reproduced. This tests our response to those conditions. Separate integration checks establish whether our assumptions about the real system hold.

I would also keep a small number of assembled-system smoke tests, some through the actual UI, to check that the deployed application starts, reaches its dependencies and supports critical journeys. These check assembly and deployment without becoming another mandatory level of business testing.

A practical test

Take a database-backed application whose test suite is considered good and set the coverage percentage aside.

Choose one meaningful module. Can its behaviour be tested through its intended public surface without depending on its internal class structure?

Choose a transaction and a read or calculation. Starting from small, known business facts, can you exercise the real database behaviour and explain the expected results? For a known concurrency risk, can you reproduce the competing operations and explain the protection the design relies on?

Then choose a larger process. Is there evidence for its important success and failure paths, including what happens across external boundaries?

Explicit transactions and processes would give our tools better things to report: which outcomes, process routes and known interactions have tests, and which do not. A report exposing those omissions would be more useful than a percentage that treats one successful invocation as sufficient evidence.

That brings me back to the application with effectively one hundred percent coverage. Coverage showed us which code the suite exercised. It could not tell us whether the expected answers were sound, the important transactions were examined, or the business process worked. We had made the tests fast and the number impressive while leaving those questions open.

The product owner’s question deserved an answer we could demonstrate: does the software perform the work it exists to do?

References