Testing Distributed Systems with Deterministic Simulation

Distributed systems are hard to test because the failures you care about — network partitions, clock skew, partial crashes — are non-deterministic. You can run integration tests for weeks and never hit the exact sequence that triggers a bug in production. We have been using deterministic simulation to solve this, and it has caught bugs that would have taken months to surface in the wild.

The idea is simple: replace the network, clock, and random number generator with deterministic implementations. Instead of real TCP sockets, use an in-memory message queue that you control. Instead of time.Now(), use a virtual clock that advances only when you tell it to. Instead of math/rand, use a seeded generator. Now every run is identical. If a test fails, it fails the same way every time.

We implement this in Go by defining interfaces for the runtime dependencies:

type Network interface {
    Send(to NodeID, msg Message) error
    Recv() (Message, error)
    Partition(nodes []NodeID)
    Heal()
}

type Clock interface {
    Now() time.Time
    Advance(d time.Duration)
}

The production implementation uses real TCP and the system clock. The test implementation uses a simulator that can drop messages, delay them, or partition the network between arbitrary nodes. A single test can simulate hours of wall-clock time in milliseconds.

The most valuable test we have is a chaos-style scenario: we start a five-node cluster, run a workload for 1000 virtual seconds, and inject a random sequence of failures — partitions, node crashes, disk stalls. We run this with a thousand different seeds every night in CI. Last month it found a bug where a node would lose a committed write if it crashed exactly between fsync and the replication ACK. The window was 3ms. We would never have hit it in production testing.

"If you cannot reproduce a bug in a test, you do not understand it. If you do not understand it, you have not fixed it."

The trade-off is that deterministic simulation requires your code to be written against interfaces, not concrete types. This is good practice anyway, but it adds friction. You cannot use net.Dial directly; you must use your Network interface. For existing codebases this means refactoring. For new code, it is a constraint that pays off quickly.

We open-sourced our simulator last month. It is not a framework — just a set of interfaces and a reference implementation — but it has been useful enough that three other teams at my company have adopted it. The repository is linked on the about page.