·9 min read·Deep Dive

Testing the Way a Restaurant Floor Actually Runs

A restaurant floor is a concurrency and reliability test that runs every night whether we prepared for it or not. Here's how we test PayTable against that reality — real usage patterns, a real Postgres, and tests we've proven actually catch problems.

By Igor Riera

A restaurant floor is a concurrency test that runs whether we’re ready for it or not. Several diners pay against one table at the same time. A kitchen has a hard ceiling on how many tickets it can fire at once. A handful of waiters share a handful of tablets across a full section. None of that waits for a deploy window — it happens every service PayTable is live, at full volume, with real money moving through it. So the test strategy is built around that shape: usage patterns we can actually name, a real database instead of a stand-in for one, and tests we’ve gone out of our way to prove actually catch what they claim to catch.

The suite grows with the product

PayTable’s backend is a .NET Clean Architecture app — CQRS with MediatR, EF Core on PostgreSQL, deployed on Azure. As of today it has 13,628 [Fact] and [Theory] test definitions across 1,215 files and 22 of its 53 projects. Five months ago that number was 5,025. We didn’t set out to hit a test count; the count is just what happens when every new endpoint and every new business rule brings its own coverage along with it.

The distribution tells you where the risk lives. The application layer (business rules and handlers) carries 5,135 test definitions, the data layer 2,598, the web API 1,886, the Blazor admin app 1,144, and payments — a smaller surface with a much lower tolerance for being wrong — 364.

The guest-facing side gets the same treatment. The Vue 3 + Ionic ordering app has 2,016 Vitest unit tests and 998 Cypress end-to-end tests, up from 705 and 200 respectively back in April. The Blazor admin has its own Playwright end-to-end suite of 378 test definitions. None of these numbers are the point on their own — they’re evidence that coverage keeps pace with a product that’s still moving fast, instead of falling behind it.

Real Postgres, because that’s where the real bugs live

Mocking a database is a reasonable way to test a single method in isolation. It’s a bad way to test whether a unique constraint actually stops a duplicate write, or whether a transaction actually rolls back the way you think it does. So roughly 100 of our backend test files are tagged Category=Integration and run against a real PostgreSQL instance through Testcontainers — up from 27 in April. That tier ran 223 tests in early June; by late August it was running 400.

The obvious way to do this — one container per test — gets slow fast, and slow tests get skipped, which defeats the purpose. So we run one shared Postgres container per test session, and instead of tearing it down and rebuilding it between tests, each test gets its own database created instantly from a pre-migrated template:

CREATE DATABASE test_run_042
TEMPLATE paytable_migrated;

Postgres can clone a template database near-instantly because it just copies the files rather than replaying every migration. The difference in practice: the full integration run takes 1 minute 12 seconds. With a container per test, it took 17 minutes. That’s the difference between a suite engineers actually run locally before pushing and one they route around. Production deploys run this integration suite as a hard gate — code doesn’t reach the release step without passing against a real database.

Testing the races, not just the paths

A restaurant floor produces races that a single-user demo never will, so we write tests that produce them on purpose.

An order at PayTable can be held back by more than one independent gate at once — a guest still needs to approve it, a wallet needs enough headroom, the kitchen needs capacity. Any one of those clearing the order shouldn’t be a race with the others clearing it too and firing the ticket twice. The release itself is a single conditional update that only succeeds if none of the pending gates are still open, something in the shape of:

UPDATE orders
SET released_at = now()
WHERE id = @orderId
  AND released_at IS NULL
  AND NOT EXISTS (SELECT 1 FROM pending_guest_approval WHERE order_id = @orderId)
  AND NOT EXISTS (SELECT 1 FROM pending_wallet_hold      WHERE order_id = @orderId)
  AND NOT EXISTS (SELECT 1 FROM pending_kitchen_capacity WHERE order_id = @orderId);

No advisory lock, no application-level mutex — Postgres’s own row visibility does the work. The test for this opens five separate DbContext instances and has all five race to release the same order at once, then asserts exactly one of them actually fires it to the kitchen. That’s a stronger claim than “the code looks correct” — it’s “we ran the race and only one winner came out.”

Background job claiming uses the same idea in a different shape: FOR UPDATE SKIP LOCKED, so ten concurrent workers pulling from a shared queue never fight over the same row — each one just skips whatever another worker already has locked and grabs the next one. The test runs ten concurrent callers against five available rows and asserts exactly five unique claims come out the other side — no duplicates, no rows claimed twice, nothing left contested.

Split payments get the same discipline. Two diners tapping “I’ll pay for this” on the same plate at the same moment resolve through a partial unique index plus ON CONFLICT — in sketch form:

CREATE UNIQUE INDEX ux_active_item_claim
ON item_claims (order_item_id)
WHERE released_at IS NULL;
INSERT INTO item_claims (order_item_id, guest_id)
VALUES (@orderItemId, @guestId)
ON CONFLICT (order_item_id) WHERE released_at IS NULL DO NOTHING;

Exactly one insert lands, the loser gets an idempotent response instead of an error, and nobody sees a duplicate charge for a plate of tacos. “Two people tapping the same button at the same time” is the most predictable thing that happens on a restaurant floor, so it gets a test like anything else.

Load testing against the busiest kitchen we can simulate

Concurrency tests prove correctness under contention. Load tests prove the system stays fast while that contention is happening at volume. We run Locust load and soak tests and keep the reports in the repo, under docs/operations/load-test-reports/, so the numbers are a record, not a claim.

A kitchen-throttle scenario — where the kitchen is the bottleneck, not the app — pushed 2,660 order creations through with zero 5xx responses and a P95 of 460 milliseconds. Disbursements, the payout side of the pipeline, came back at a P95 of 180 milliseconds. We also run two-hour soak tests, because a system that’s fast for five minutes and degrades over two hours is a different, more dangerous kind of slow.

Moving the backend to .NET 10 dropped median request latency from roughly 190 milliseconds to about 72 milliseconds. Faster code is a win on its own, but it also compresses the timing window that used to give a race a natural buffer — two operations that were “usually” far enough apart in wall-clock time to avoid colliding are now close enough together that they need to be provably safe rather than practically safe. That’s exactly the argument for keeping concurrency tests in the everyday suite and soak-testing after an upgrade like this, rather than treating a performance gain as a free win. It’s also why we design operations like closing out a tab or confirming a cash payment to be idempotent in the first place — if a call happens to repeat or race under tighter timing, the second attempt is a no-op rather than a double-charge.

Proving the tests themselves work

A test that never fails isn’t necessarily a good test — it might just be a test that can’t fail. So part of the discipline is testing the tests.

Our sustained-load memory gate — the check that’s supposed to catch a slow leak before it reaches production — doesn’t compare two snapshots in time, because two points mostly measure warm-up, not a trend. It samples memory usage across the whole run and looks at the trend line. We validated that the gate actually works by injecting a deliberate leak of 50 KB per operation into a test build and confirming the gate caught it. If it hadn’t, the gate would have been decoration.

The same instinct shows up on the frontend: fixes are routinely validated by breaking the code back on purpose and watching the corresponding test fail. If a test can’t fail, it can’t tell you anything. Animation and timing tests specifically run against a controlled clock rather than real wall-clock time, because we measured real timing on the same test varying up to 4x depending on what order the tests happened to run in — a test that’s only sometimes accurate isn’t one you can trust a release on.

Boundaries that are enforced, not just documented

Some rules are easier to keep if the build breaks when they’re violated. Nine NetArchTest rules pin down the boundaries of the payments module — for example, that the contracts project stays a dependency leaf and never reaches back into implementation details — and they run in CI on every change. One detail we’re fond of: the test project’s name was deliberately chosen to match the *Tests.csproj glob that CI’s test discovery already looks for, specifically so the architecture rules can’t quietly stop running because a naming convention drifted. A rule that isn’t running isn’t a rule.

Two team-wide standards sit on top of all of this. First, a new code path needs a test before it merges — coverage isn’t a cleanup task scheduled for later. Second, changes that touch restaurant data carry negative tests that prove one restaurant cannot see another restaurant’s data. In a multi-tenant platform, that isolation boundary is the one we’re least willing to leave to code review alone.

What runs after the tests pass

Passing tests earn a deploy, not a promise. Test and staging environments deploy automatically from their branches once CI is green. Production doesn’t — it’s a deliberate, manual release that deploys to a staging slot first, runs a smoke test against that slot, and only then swaps it into production traffic. On top of the automated suite, a set of live end-to-end journeys runs against our deployed environments every night, checking that the real thing, not just the test double of it, still behaves the way we expect.

None of this is really about hitting a test count. It’s about matching the test strategy to what actually happens on a restaurant floor: money changing hands under contention, every night, whether the code is ready for it or not. Real Postgres instead of a stand-in, races we manufacture on purpose instead of hope we never see, and a habit of proving the safety net itself holds weight before we trust it. That’s what “production-ready” means for a payments platform — not that it passed once, but that it’s built to keep passing under the exact kind of pressure a Friday night service puts on it.