07 — Testing and Load¶

How to find out whether Limen is what we say it is, and what to look at while you do.

There are three levels, and they answer different questions.

Level	Question it answers	Command	Takes
Architecture tests	Does each principle hold in isolation?	`mvn verify`	~1 min
Proof script	Does one contract survive nine months of real life?	`./scripts/prove.sh`	~10 s
Load scenarios	Does it still hold at volume, and where does it break?	`./scripts/load.sh …`	1–5 min each

None of them is optional. The tests catch a principle being broken by a code change. The proof script catches a principle that was never true end to end. The load runs catch the principles that are true but too expensive to be worth having — which, as it turned out, is where all three of our real defects were hiding.

Level 1 — Architecture tests¶

cd backend/limen-server
mvn verify

These run against a real PostgreSQL and a real Redis, started as containers. There is no H2 and no in-memory substitute anywhere, because half the claims are enforced by Postgres itself and an in-memory database would quietly pass tests that production would fail.

Test	Claim under test
`AppendOnlyLedgerTest`	Money cannot be rewritten. Also that a payment delivered twice moves money once.
`BackdatedCorrectionTest`	The past can be corrected by forking and replaying, never by mutating.
`PartyNetworkTest`	A hierarchy is effective-dated edges, so restructuring is one write and history survives.
`HotContractStateTest`	Redis is an accelerator. Losing it costs latency and nothing else.

Level 2 — The proof script¶

cd backend/limen-server
./scripts/stack.sh up
./scripts/run-server.sh --background
./scripts/prove.sh

44 checks against a running server. It creates one contract, puts it in force nine months ago, raises premium for every month since, takes a payment twice, then discovers a child was never added at inception and corrects it. Each check prints what it asserted and what it saw.

What it proves, in the order it proves it:

Rules are data. The premium comes from a versioned rate schedule, and the response names the schedule that produced it. Nobody sent us a price.
A contract is one document. One key lookup returns cover, parties, price and status together. There is no assembly step to get slow.
A status change is a version. Activation appends. Version one still says QUOTE afterwards.
No batch run. Nine months of premium are raised individually, each on its own due date.
Exactly once. The same bank webhook delivered twice produces one movement and returns the original entry IDs.
The database refuses to lie. Raw SQL UPDATE and DELETE against ledger_entry, contract_version and decision_record are rejected by a Postgres trigger. This is the check to run in front of a sceptic, because no application change can loosen it.
The past is corrected, not rewritten. Ten months of consequence settle as one adjustment entry for the exact difference.
Nothing was rewritten to get there. Every original premium still carries its original amount, and the version that was branched from is still on the timeline.
The old statement can be reissued. The same effective date returns the old cover when asked as at last week, and the new cover when asked as at now. Both answers are true. This is the query a legacy system cannot serve at all.
Every figure is explainable. The corrected premium can be read straight off the decision record, down to the rule ID for each component.
Events were recorded with the change. They are written in the same transaction and drained continuously, so state and events cannot disagree.
Restructuring distribution is one write. Moving an agency to a new manager is one edge; last year's reporting line is still answerable.

If any check fails, either the code is wrong or the pitch is. Both are worth knowing.

Level 3 — Load¶

./scripts/load.sh seed 50000     # build a population through the real API
./scripts/load.sh read           # hot path, truth path and point-in-time path, side by side
./scripts/load.sh write          # the money path
./scripts/load.sh correct        # back-dated corrections, the expensive operation
./scripts/load.sh mixed          # a realistic blend, ramped until something gives

Every run prints a before and after snapshot: rows, bytes per contract, Redis memory, connection pool waits, GC, and server-side latency percentiles alongside k6's client-side ones. Throughput on its own is a vanity number. Throughput next to bytes per contract and connection pool wait time is an architecture you can defend.

RATE, DURATION, VUS and WARMUP are all overridable: RATE=2000 DURATION=2m ./scripts/load.sh read.

Nothing is inserted behind the API's back. A population built by a bulk loader would tell us nothing about whether the platform can create contracts at that rate.

What to check, and where¶

Look at	Where	What it tells you
`bytes per contract`	snapshot, "Stored truth"	Whether keeping every version is affordable. This is the answer to "storage will explode".
Redis `memory used`	snapshot, "Hot state"	The real cost of the hot path. This is the answer to "memory is not cheap".
`awaiting dispatch`	snapshot, "Outbox drain"	Whether events are keeping up with writes, or a backlog is growing.
`requests waiting for a connection`	snapshot, "Runtime"	Whether a pool is the ceiling rather than the architecture.
`p50` vs `p99` per endpoint	snapshot, latency table	Where the tail is. A good median with a bad tail means something is queueing.
`dropped_iterations`	k6 output	The load generator could not keep up at the offered rate. The system is saturated.

Measured, 8 August 2026¶

One MacBook, 8 cores and 16 GB, with PostgreSQL 18, Redis 7, the JVM and k6 itself all sharing them. These are not headline numbers from a tuned cluster; they are what one laptop does while also being the database and the load generator.

Writing 50 000 contracts through the API


Throughput	551 contracts/s (1 102 requests/s)
Latency	p50 39 ms, p95 72 ms
Failures	0 out of 100 000
Cost in Postgres	3.5 kB per contract, including both versions, its decision record and its events
Cost in Redis	34 MB for 50 000 hydrated documents, about 700 bytes each

At 700 bytes per hydrated contract, a 10 million policy book is roughly 7 GB of hot state. That is one Redis node, not a fleet, and it is the honest answer to whether in-memory state is affordable at national scale.

Reading, all three paths at once

Path	What it does	p50	p95
Hot	Redis lookup, document returned untouched	0.86 ms	98 ms
Truth	One Postgres row, one JSONB column, no join	1.9 ms	155 ms
Point in time	Same, as understood at an earlier moment	1.9 ms	158 ms

Two things worth saying plainly. First, a point-in-time read costs the same as a current read, because both are one row. Second, the truth path is fast enough on its own that Redis really is an accelerator rather than a crutch — which is exactly what we claimed, and now it is measured rather than asserted.

Back-dated corrections, the expensive operation


Volume	3 000 corrections in 7.6 s
Throughput	397 corrections/s
Latency	p50 55 ms, p95 85 ms, p99 172 ms
Failures	0

Each one read the version in force, re-priced nine months against the schedule that applied then, appended a version, compared the result to what had actually been charged, and posted one adjustment for the difference. This is the operation legacy systems take offline for a weekend. Here it is a 55 millisecond request, 400 of them a second, while reads continue.

Mixed book: 88% reads, 10% collections, 2% corrections

922 requests/s sustained, zero failures. Median read 636 µs, median collection 5 ms, median correction 12 ms. p95 rises to 66 ms on reads and 251 ms on collections at that rate. Saturation shows up as latency, not as errors, which is the failure mode you want.

The ceiling on this hardware is about 1 750 requests/s. Past that, dropped_iterations appears and the tail grows while the median stays under 2 ms. That is CPU exhaustion on a laptop that is simultaneously the database, the cache and the load generator, not a limit of the design. The next honest measurement needs the load generator on a different machine.

What the load runs actually found¶

This is the part worth showing to an engineer who has heard a pitch before. Three defects, none of which any unit test would have caught, all of which would have been found by a customer instead.

1. The cache was slower than the database it was protecting. The hot read fetched the document from Redis, parsed it into objects, and serialised those objects back into the same document. p50 was 225 ms while the Postgres path was 1.5 ms. Serving the stored document untouched took p50 to 0.69 ms, a 300-fold difference. The claim "one lookup, no assembly" had been true of the storage and false of the code.

2. The cache failed before the database would have. Redis was configured with 32 connections and the default 24-deep wait queue, while 200 blocking worker threads called into it. At around 600 requests/s the hot path started returning ConnectionPoolTooBusyException while the Postgres path stayed healthy. Because the boundary is blocking, every downstream pool has to be sized against the worker thread count. Both numbers now carry a comment saying so.

3. The outbox could not drain as fast as it filled. It marked events published one row at a time, draining about 414 a second against a write rate well above that. After 50 000 contracts, 62 712 events were still waiting. One statement for the whole batch cleared the entire backlog in under 40 seconds. The remaining limit — one poller, batch-size per interval — is now documented on the class, and the backlog is one query away from being visible.

The pattern is worth noting: every one of these was an implementation betraying an architecture that was sound. That is the useful kind of finding, and it is the argument for running these before the pitch rather than after.

Known ceilings¶

Named deliberately, so nobody has to discover them in a demo.

Ceiling	Where it bites	What removes it
One outbox poller	Write rates above `batch-size` per interval	Several pollers on disjoint slices, or Debezium reading the write-ahead log
Blocking boundary	One worker thread per in-flight request	Reactive endpoints on the read path, where it pays for itself
Single node, single database	Everything above	Read replicas for the truth path, then cellular partitioning by tenant
Correction cost grows with history	A contract with hundreds of versions	Replay from the nearest version rather than from the fork point
No load shedding	Saturation shows as latency, indefinitely	A concurrency limit at the boundary that fails fast instead of queueing

Not yet built, so not yet measurable¶

Product factory and component templates, campaigns, claims, renewals and underwriting, DMN through Kogito or fords-automation, scheduled milestones through fords-scheduler, communications through fords-comm, multi-tenancy and POPIA data residency, CQRS read models for finance and actuarial. See Architecture proof for the full boundary between what is proven and what is still a promise.