Security that fails closed

PayTable is a QR ordering and payments platform we run for restaurants in Mexico. A guest scans a code at the table, orders, and pays — a card, a wallet, whatever the restaurant accepts. Every one of those payments passes through infrastructure we’re responsible for, which means every design decision in that path has to answer one question honestly: what happens when this is misconfigured?

The naive answer to that question is usually “the feature degrades gracefully.” A missing setting falls back to a default. An empty allowlist is treated as “allow everything, figure it out later.” A disabled safety check just… doesn’t run, and nothing tells you. That’s a reasonable posture for a lot of software. It’s the wrong posture for anything that touches a card.

Our answer is the opposite: in production, a misconfiguration should stop the deploy, not run a weaker system. This post is about what that looks like in practice, and about the other half of the same philosophy — keeping as little sensitive data in our hands as possible, so there’s less that a mistake, ours or anyone’s, could ever expose.

The platform underneath, the controls on top

PayTable runs on Microsoft Azure, in the Mexico Central region for production. Azure carries its own platform certifications — ISO 27001, SOC 1/2/3, PCI-DSS at the infrastructure level — and those matter, but they cover the platform, not our application. The controls in this post are the ones we own: what happens inside our API, our database connections, our deploy pipeline.

Transport is the floor everything else sits on. TLS 1.2 is the enforced minimum across the API, its deployment slot, and storage — defined in Terraform, not configured by hand after the fact. HTTPS is required with HTTP redirected, FTP deployment is disabled, and database connections require SSL. On top of transport, the API sets its own response headers on every reply: a Content-Security-Policy, X-Frame-Options set to deny framing entirely, X-Content-Type-Options set to nosniff, and a Referrer-Policy. None of that is exotic. What matters is that it’s declared in code — Terraform for the transport settings, middleware for the headers — and applies uniformly, rather than depending on someone remembering to check a box in a portal.

A misconfiguration should stop the deploy

Here’s the part of the design we think about the most. In production, the API throws at startup — refuses to boot — if any of the following is true:

• test authentication is enabled
• the token-signing secret is shorter than 32 characters
• payment idempotency is disabled
• the processor webhook signing secret is empty
• the reservations signing secret is missing while reservations are enabled

Each of these is a condition that, left alone, wouldn’t crash anything. The API would start. Requests would be served. It would just be serving them with a weaker guarantee than the one the rest of the system assumes — a signing secret an attacker could brute-force, a payment path that could double-charge on retry, a webhook endpoint that can’t tell a real processor callback from a forged one. That’s the shape of failure we’re not willing to accept silently.

A simplified version of the check looks like this:

if (environment.isProduction) {
    if (config.testAuthEnabled) throw new StartupError("test auth enabled in production");
    if (config.tokenSigningSecret.length < 32) throw new StartupError("signing secret too short");
    if (!config.paymentIdempotencyEnabled) throw new StartupError("idempotency disabled");
    if (string.isNullOrEmpty(config.webhookSigningSecret)) throw new StartupError("webhook secret missing");
    if (config.reservationsEnabled && string.isNullOrEmpty(config.reservationsSigningSecret))
        throw new StartupError("reservations enabled without signing secret");
}

The comment on the webhook-secret guard names the exact scenario it exists for: a wiped Terraform variable taking the Key Vault reference with it. That kind of change should stop the deploy — not quietly hand production a weaker system that looks identical from the outside. A failed deploy is loud, visible, and reversible. A quietly-degraded one is none of those things until something goes wrong downstream of it.

There’s one deliberate exception, and it’s worth explaining because it shows this isn’t a blanket rule applied without judgment. A token service used by only one feature fails closed at the point of use, not in its constructor. If that service threw at startup the way the checks above do, a missing configuration value for one narrow feature would take down every request the API serves — the blast radius would be the whole system for a fault that only affects one corner of it. Failing at the point of use keeps the blast radius scoped to the feature that actually depends on it. Fail-closed isn’t “throw as early as possible everywhere” — it’s “throw at the point where the failure should actually be contained.”

Secrets live in one place

Signing keys, connection strings, and processor credentials are stored in Azure Key Vault, with 90-day soft delete and purge protection enabled in production. The API authenticates to Key Vault using a system-assigned managed identity, and that identity holds exactly one role: Key Vault Secrets User, which is read-only. The application can read the secrets it needs — it can’t rotate them, delete them, or grant access to anything else. App settings pull values in as Key Vault references, so rotating a secret is a change in the vault, not a hunt through environment configuration.

CORS that fails shut, not open

Cross-origin requests to the API are checked against an explicit allowlist of production origins. The failure mode here is the same principle applied to a much smaller surface: if that allowlist comes back empty, or somehow resolves to a wildcard, the middleware doesn’t interpret that as “no restriction configured, allow everything.” It rejects every cross-origin request and logs a warning. An empty allowlist is a locked door, not an open one.

One path for every token to prove itself

Every role check in the API validates a session token’s signature through a single shared validation path. That sounds like a small implementation detail, but it’s the whole point — a session token can be forged or tampered with in a dozen creative ways, and each one of those is a place a validation routine could get subtly wrong. Having one place where that validation happens means there’s one place to get it right, and one place to audit, rather than a dozen call sites that each reimplement the same check slightly differently. Session tokens are valid for 24 hours.

Refresh tokens get more scrutiny, because they live longer and matter more. Each one is 32 random bytes; we store only the SHA-256 hash of it, under a unique index, never the token itself. A refresh token is valid for 30 days and single-use: presenting it revokes it and issues a new one, and we record the hash of that replacement so the chain is traceable. If an already-revoked token gets presented again, that’s not treated as a bug or a race condition — it’s treated as theft, and every refresh token belonging to that user is revoked immediately. The assumption is that a token which should no longer exist showing up again means someone has a copy they shouldn’t.

That same mechanism does quiet work on the restaurant floor. Staff sign in to shared tablets by PIN, and more than one person uses the same device over a shift. When a new staff member signs in, the previous user’s refresh token is revoked as part of that switch. A shared tablet never silently renews a session as the person who last logged out of it.

PINs, not passwords, on the floor

Staff authenticate to the tablet-facing side of PayTable with a PIN rather than a password, because that’s what fits a fast-moving restaurant floor. The PIN is BCrypt-hashed at a work factor of 10, the account locks out after repeated failed attempts, and PIN sign-in only works from a tablet that’s already been registered to the restaurant — a PIN alone, entered from an arbitrary device, isn’t enough.

The PIN endpoints are rate limited in two tiers: a per-caller limit, and a global backstop on the endpoint itself. The reason for the second tier is straightforward once you think about how the first one could be defeated — a per-caller key is often something the caller controls, like an IP address or a device identifier, and an attacker attempting to brute-force PINs can rotate that identifier faster than a single-tier limit can react. A global ceiling on the endpoint doesn’t care whose key is attached to the next request; it just won’t let the endpoint absorb unlimited attempts, from anyone, in a given window. Passwords elsewhere in the system — for the accounts that use them — go through ASP.NET Identity’s PBKDF2 hasher.

Keep the least data you can

The single biggest lever we have over what a compromise could ever expose is simple: hold as little as possible. PayTable never stores a full card number, a CVV, or an expiry date. What it keeps is a last-four, a card brand, and a masked number — enough to show a guest which card they used, nothing that could reconstruct the card itself. The processor token representing a card is deleted once a payment is captured or voided; it doesn’t sit around after its job is done. Authorization and settlement themselves run inside the processors’ own PCI-certified environments.

Audit logs that don’t become the leak

We keep an audit log of requests for accountability and debugging, which creates an obvious tension: a log detailed enough to be useful is also detailed enough to capture something it shouldn’t. The masking that runs before a request body is written to that log walks the JSON recursively and rewrites anything sensitive it finds; if the body isn’t valid JSON — a malformed payload or a form submission — it falls back to a regex pass instead of skipping masking entirely. The record only gets rewritten if the masker actually found something to mask, so an audit entry with nothing sensitive in it stays exact rather than getting needlessly mangled.

The processor client has its own, narrower masker for anything resembling a card number, and it keeps the first six and last four digits, masking everything between. The comment sitting next to that logic is the whole policy in five words: rather over-mask than leak.

Trusting inputs you don’t control

Two integrations in PayTable exist specifically to receive data from someone else’s system, and both get treated as hostile until proven otherwise.

Processor webhooks carry an HMAC-SHA256 signature, and we check it with a constant-time comparison — a non-constant-time comparison leaks timing information about how much of the signature matched, which is exactly the kind of detail an attacker can use to forge one byte at a time. An unsigned webhook call is rejected everywhere except a local development environment. Duplicate deliveries — which webhook systems produce as a matter of course, since “at least once” is the honest guarantee most of them offer — are neutralized by a unique external event ID, so a redelivered event doesn’t get processed twice.

Apple Pay merchant validation is a more interesting case, because of its shape: the browser hands the server a validation URL, and the server calls that URL while presenting a client certificate. That’s a textbook server-side request forgery pattern — an attacker who can influence the URL can make our server, holding a legitimate certificate, call somewhere it shouldn’t. Our validation only calls URLs beginning with https://apple-pay-gateway.apple.com/ — the trailing slash is there on purpose, because without it, a domain like apple-pay-gateway.apple.com.attacker.example would pass a naive prefix check while pointing somewhere else entirely. The merchant domain used in that call is taken from the request’s Origin header only when that Origin is itself on an allowlist. Apple’s response to a successful validation carries session material used later in the payment flow, and that response is never written to a log.

Recovering without drama

Fail-closed at startup handles misconfiguration. A separate set of decisions handles the ordinary risk of running a system over time. Postgres backups run automatically with 35-day retention. Blob storage has versioning enabled plus 14-day soft delete. Production deploys go to a staging slot first, run smoke tests there, and only then swap into production — so a deploy that’s going to misbehave gets a chance to show that before it’s serving guests. Database migrations are never something a slot startup triggers automatically; running one is a separate, deliberate step, and it defaults to a dry run. None of these are exciting. They’re the parts of the system designed for the day something needs to be undone cleanly instead of debugged live.

Checking our own work

We run adversarial security reviews on PayTable as a matter of process: one pass produces findings, and a second reviewer’s job is specifically to try to disprove each finding before it gets accepted onto the remediation list. That second pass exists because a list of findings nobody has argued with isn’t a security review, it’s a suggestion box. Forcing every finding to survive someone actively trying to knock it down is what makes the resulting list something worth acting on.

The point of all of it

None of the individual pieces here are unusual on their own — startup validation, a managed identity with a narrow role, a signed webhook, a masked log field. What makes them add up to something is the same decision applied consistently: a system that can’t verify it’s configured correctly should refuse to run, not run anyway and hope. And a system that holds less sensitive data to begin with has less riding on any single control working perfectly. Those two ideas cover most of what “secure by design” means to us in practice — not a feature, a default.