The network we chose

What we could take away if the unit of allocation were the machine rather than the VM.

Ask a developer in a large organisation what a server is and you will usually get a description of a virtual machine: eight cores, perhaps twelve, sixteen gigabytes of memory, storage somewhere on the SAN. Thirty-two cores needs a business case. That picture is roughly what a physical server looked like fifteen years ago, and it has quietly become the unit we design around. If the largest thing you can be given is eight cores, anything bigger must be split across machines, and distribution becomes the default before anyone has made an architectural decision.

I want to try a thought experiment. Set aside the VM as the unit of allocation and ask what the hardware we can buy today would let us do with ordinary business systems. My answer is that it would let us remove a surprising amount of what we currently treat as the unavoidable cost of running a business.

What a server is now

AMD’s sixth-generation EPYC processors, launched in July, reach 256 cores and 512 threads in a single socket. A two-socket machine therefore has 512 cores, several terabytes of memory and room for a dozen or more Gen5 NVMe drives, each rated at around 14 GB/s of sequential reads and over three million random reads a second.

Now consider what a large company’s systems of record actually do. An order management system or a ledger handling a few hundred business transactions a second at peak is a busy one, and its working set usually fits in memory with room to spare. Put several such systems together and the gap between demand and the capacity of one machine is still probably two orders of magnitude. That gap widens with every hardware generation, while transaction volumes grow roughly with the business.

The argument here is confined to the internal systems that run a business: orders, ledgers, inventory, payments, customer accounts. Consumer-scale web traffic with global latency requirements is a different problem. For systems of record, one machine, or a few, has more than enough.

There is also a reason to ask this now. Since Broadcom bought VMware, perpetual licences have gone, products have been folded into bundles, and billing has moved to per-core subscriptions, with Gartner citing typical increases of 300 to 400 per cent. Most organisations are responding by moving to another hypervisor and carrying their estate of small VMs across intact. A migration like that is the one moment in a decade when the unit of allocation is open for question, and a like-for-like move wastes it.

The price of a hop

Latency figures are easier to feel at human scale, so imagine one nanosecond lasting one second.

The NVMe figures come from current enterprise drive specifications, which quote best-case latency at queue depth one. The SAN figure is a measured one: an all-flash array on end-to-end 16Gb Fibre Channel came in at around half a millisecond at the host. Plenty of enterprises still run that generation, but not all do. The remaining rows are typical orders of magnitude.

Newer arrays use NVMe-over-Fabrics, which carries NVMe commands directly over RDMA, Fibre Channel or TCP instead of translating them into SCSI, and they narrow the gap considerably. The row for them is my estimate: most published figures come from vendors, cover the fabric rather than the whole array, and describe reads, whereas an array usually mirrors a write to its partner controller before acknowledging it. On that basis local NVMe commits roughly 10 to 20 times faster rather than 50, and for a read miss the gap almost closes. An order of magnitude on every commit is still large, though, and the network is still in the commit path with everything that brings.

Writes on local flash are faster than reads because the drive acknowledges a write once it reaches its power-protected cache, while a read has to go to the flash itself. That asymmetry suits a transactional database well. Commits are writes, and most reads are served from memory before they reach a disk at all; when one does miss, local NVMe answers in under a day at this scale against most of a week for the SAN.

Those are raw operations. What an application actually experiences is a query, and here there is a recent measurement worth having. Cybertec ran pgbench over a single connection doing a simple indexed lookup, first over a Unix socket, then over TCP to localhost, then across a real network with one hop. The index read itself took 0.016 ms.

The first, second and fourth rows are Cybertec’s measurements; the others are estimates, and the switch row in particular depends heavily on the environment. Across that single hop, the database spent about 4 per cent of the elapsed time doing the work and the rest waiting on the wire, and throughput on the connection fell from over 50,000 transactions a second to under 2,400.

Adding connections recovers throughput. With more clients over the same network, Cybertec’s rate climbed back to around 82,000 a second, while each query still took about 0.43 ms. Concurrency hides network latency from the throughput figures, but it does nothing for the duration of an individual transaction, and every transaction still holds its locks for the full time on the wire.

Three consequences follow. A commit waits for its log record to reach durable storage, holding its locks while it waits; on local NVMe that is a few hours at human scale, on a Fibre Channel SAN most of a week, and on NVMe-over-Fabrics a day or two. A transaction making twenty indexed lookups across one network hop takes around eight milliseconds, which at the same scale is about three months, again with locks held; through a firewall and load balancer it is twenty milliseconds or more, most of a year. Over a Unix socket the same twenty lookups take about 0.4 ms, under five days, nearly all of it spent doing the work. Add the commit, and the whole transaction runs roughly 23 times faster on one box than across a single network hop and SAN, and about 50 times faster than through a firewall and load balancer. A faster SAN barely changes those whole-transaction figures, because the application’s own network round trips dominate them. Shorter transactions mean less contention, fewer deadlocks and narrower windows for isolation anomalies, so this is a correctness benefit as much as a performance one.

The third consequence is in the last row. In 1994 Waldo, Wyant, Wollrath and Kendall argued in “A Note on Distributed Computing” that partial failure is what makes distributed computing different in kind from local computing. When a remote call times out, you cannot tell whether the work was done. You wait what amounts to centuries and are then left guessing. On one machine processes still fail, but they fail unambiguously: the process exits, the service manager knows at once, and if every unit of work is a database transaction, the crash means a rollback. Ambiguous failure mostly disappears. What remains is clean crash failure, which we have known how to handle since the 1970s.

What goes away

Remove the network between services and their database and a long list of machinery loses its reason to exist. Sagas, outbox tables, compensating transactions and eventual consistency between internal systems go, because modules that must agree can live in separate schemas of one database and commit together. Retries, circuit breakers and idempotency keys on internal calls go; they stay at the edge, where the network is real. Service discovery, internal load balancers, service meshes and internal certificates go. Cache clusters go, because the database is already in memory. Read replicas and sharding go, as do message brokers used as glue between systems. Distributed tracing turns back into profiling.

Each of those has a licence, a team, a patch cycle, a runbook and its own ways of failing.

The effect on programming is quieter. When a query costs a few microseconds, chatty navigational access stops being a performance sin. Much of what we call the N+1 problem is really a network problem.

Operations change too. Hundreds of VMs each carry a fixed tax: patching, hardening, vulnerability scans, security agents, monitoring and backup agents, a CMDB entry. A handful of hosts carry it a handful of times. An incident that currently crosses application, network, virtualisation and database teams, with tickets bouncing while each proves its layer innocent, now happens on one machine with one clock and one log stream, where perf and eBPF can follow a request from socket to disk.

Organising the box

One box still needs organising, and Linux has had the tools for years. systemd is a front end to cgroups, and a few lines in a unit file pin a database to twenty cores on one NUMA node, cap its memory, weight its I/O and limit its process count. Slices nest those limits, so a business domain has a ceiling and each service within it has its own. Podman uses the same mechanism, and Quadlet makes each container an ordinary systemd unit.

A few rules help. Pin rather than throttle, because CPU quotas cause latency spikes and dedicated cores don’t. Align allocations to the hardware, since cores on one chiplet share a cache and an allocation that straddles chiplets or sockets pays for it. Give each consistency boundary its own PostgreSQL instance: what must commit together goes inside it, and nothing else does.

The underlying principle is to decide which tier each boundary belongs in. A call within a process costs nanoseconds. A call between processes over a Unix socket costs microseconds and fails cleanly. A network hop costs a millisecond and fails ambiguously. Current practice puts boundaries at network hops by reflex. Most boundaries within a business domain belong in the middle tier: separate processes, separate schemas with explicit grants, independent deployments and clear team ownership, at a hundredth of the cost of the network. Independent deployment never required network separation; we stopped noticing the two could be pulled apart.

There is one trap. If services still call each other synchronously over HTTP, even over loopback, the call graph brings back cascading timeouts, retry storms and threads held waiting on slow neighbours. The box makes integration through the database possible. Take that option, or most of the absences above never materialise.

Storage without the SAN

Local NVMe raises the question of what replaces the SAN’s other features. Replication is the easy one, and the replacement is better. SAN replication copies blocks and knows nothing of transactions; database replication ships the log and knows exactly where each transaction begins and ends. Moving replication to the layer that understands the invariants is an improvement.

Snapshots, clones, integrity checking and compression are where a copy-on-write filesystem earns consideration. ZFS provides all of them, along with per-dataset quotas that sit naturally beside the cgroups limits. Copy-on-write has a reputation for hurting databases, but the big box changes the calculation. Fragmentation matters little on flash. Checksums and compression cost almost nothing on 512 cores. And because ZFS writes records atomically, PostgreSQL can safely turn off full-page writes, which removes a large share of its log volume. The remaining costs (record sizes, a fast log device, double caching) need tuning and measuring rather than assuming. So does the organisational cost: ZFS ships as an out-of-tree kernel module, and some enterprise platform teams will refuse it on principle. If it doesn’t benchmark within a few percent, a split works: XFS under the databases, ZFS for everything else.

What about the cloud?

Everything above could be built in a cloud region. The cloud’s contribution to the problem lies in copying the unit of allocation; where the machines sit matters much less. Instance catalogues reproduced the small VM, and pricing that is roughly linear per vCPU gives nobody a reason to ask for a bigger one. Yet the largest instances now run to hundreds of vCPUs and terabytes of memory, and bare-metal instances are there for anyone willing to use them.

Two details change in the cloud, and both reinforce the argument. Durable block storage is attached over the network, so it behaves like a SAN, latency included, while local NVMe is fast but ephemeral, lost when the instance stops or its host fails. Database replication therefore stops being merely the better way to protect data and becomes the only way. And managed databases sit across the network from the application by construction, so the hop this essay removes is part of what the service sells. Providers also charge for traffic between availability zones, which means a chatty set of services spread across zones pays for its conversations twice: once in latency and once on the bill.

I saw a small version of this early in the development of a cloud system. It ran on a managed relational database with plenty of vCPUs, plenty of memory and SSD-backed block storage, and it was not cheap. A colleague copied the development data onto a laptop with a Gen2 or Gen3 NVMe drive, and the same queries ran two or three times faster there. It was not a benchmark, and I remember the ratio rather than the figures, but the laptop had nothing the managed service lacked except locality: its disk was inside the machine rather than across a network.

The cloud is genuinely good at elasticity, geographic reach and fast provisioning. Systems of record rarely need the first two, and steady, predictable load is where renting costs most relative to owning. 37signals’ departure from public cloud is the best-known recent case of a company doing that arithmetic in public. For an organisation whose core systems run at a flat rate all year, the useful question is what the rent is buying.

The objections

The strongest objection is the single point of failure, and the usual reply is that the database server is already one and is remarkably reliable. That reply has a gap. Database servers are reliable largely because they run one mature piece of software that changes rarely and carefully. Most outages come from change, and application code is the most frequently changed code in any estate. Put it on the same machine and the machine is only as reliable as its least disciplined tenant. The isolation that separate hardware used to provide for free now has to be engineered, configured and tested. It is achievable, and it is real work.

Hardware headroom and software headroom are different things. PostgreSQL does not scale linearly to hundreds of cores, and most application code has a single-threaded bottleneck somewhere. Enormous headroom is a defensible claim; 512 cores of headroom is not.

Per-core licensing can end the conversation before it starts. On PostgreSQL a big machine costs nothing extra; on a commercial database licensed per core, it is a budget catastrophe. The same applies to the hypervisor. A 512-core host and its DR twin under per-core virtualisation pricing means paying every year for over a thousand cores of software whose job is to carve the machine into pieces, whose separation you then spend effort working around. Run the box on bare metal, with systemd doing the carving, and that cost disappears.

Maintenance has a tidier answer than it first appears. Patch by switching to the DR machine every month. DR then stops being an annual ceremony and becomes routine, with evidence that it works. The trade between synchronous commits over distance and accepted data loss on failover remains, but it is now the only distributed problem left in the design, and it deserves the care that used to be spread across a hundred others.

Finally, microservices were largely an organisational answer presented as a technical one, and removing the technical justification exposes that. Teams do need independent release cycles and clear ownership. Processes, schemas and grants provide both without a network in between.

A practical test

Pick one business transaction in a system you know, such as placing an order or posting a payment, and trace it from the moment the request arrives to the moment its commit is durable. Count the network hops: load balancers, service calls, database round trips, the SAN. For each one, ask whether somebody decided it was needed, or whether it followed from where the VMs happened to be.

Then measure rather than estimate. Run the transaction’s queries from the application host, and again on the database host over a Unix socket. How much of the elapsed time is the database doing work, and how much is waiting on the wire? For how long does the transaction hold its locks?

Next, list the machinery that exists only because of those hops: retries, idempotency keys, sagas, outbox tables, caches, internal certificates. Which of them would still have a job if the components that must agree shared a machine and a database?

Finally, add up the peak load of your systems of record and compare it with one of today’s servers. If it would fit several times over, ask what the rest of the estate is buying.

Where this came from

None of this is new. Transaction processing monitors of the CICS era ran many programs on one large machine against a shared database, with the transaction as the unit of work and of recovery, and Jim Gray spent much of his career making that model rigorous. What has changed is that the machine is now commodity hardware running Linux, and the isolation tools cost nothing.

A very large enterprise would still have several machines, perhaps one per business domain with its DR twin, and between them it is distributed again. The difference is that the seams are few and chosen, and each is paid for deliberately. We spent fifteen years paying the cost of the network everywhere because the unit of allocation left us little choice. The hardware has moved on, and the choice is available again.