Two-Phase Deploy: Why We Gate Every Push

Two-Phase Deploy: Why We Gate Every Push

· 2 min read

At SpiderPublish, no content reaches production without passing through our two-phase deploy gate. This is not bureaucracy — it is the direct result of a production incident that taught us the cost of optimism in deployment pipelines.

The Incident That Changed Everything

In March 2025, an agent-driven workflow pushed a template change to production that broke the navigation component on 47 tenant sites simultaneously. The change was syntactically valid Liquid, passed all linting checks, and rendered correctly in preview. But it referenced a theme token that had been renamed in the latest settings migration, causing a silent cascade of broken layouts across production.

Recovery took four hours. We restored from the previous KV snapshot, but the damage was done — 47 sites served broken navigation to their visitors for the duration. That incident led directly to the two-phase deploy protocol.

How Two-Phase Deploy Works

Phase 1: Dry Run

Every deploy starts with dry_run: true. The system compiles all pending changes, resolves component versions, validates Liquid templates against the current settings schema, and returns a preview envelope. This envelope contains the exact diff of what will change, which tenant sites are affected, and a confirm_token with a five-minute TTL.

Phase 2: Confirm

To go live, you submit the confirm_token from Phase 1. The system verifies the token has not expired (410), has not been consumed already (409), and matches the current pending state (403 if something changed between dry_run and confirm). Only then does it execute the atomic KV bulk write that pushes all changes live simultaneously.

Atomic KV Bulk Write

We use Cloudflare KV bulk write to update all keys in a single operation. This means tenant sites never see a partially-deployed state — either all changes are live or none are. The bulk write operation is atomic at the KV level, providing the consistency guarantee that traditional file-based deployments cannot match.

Who Opts In

The two-phase gate is opt-in by design. The dashboard wraps it automatically with a visual confirmation dialog. The CLI interactive prompts handle it transparently. Direct API and MCP callers must explicitly choose to use it. On development tenants, skipping it is acceptable. On production tenants, we strongly recommend always opting in.