Rotating a Live HMAC Key Without Bricking Hardware
Production had been hashing board secrets with a development constant. The one-line fix would have locked out every deployed board.
What
Tink boards prove who they are with a secret, which the server stores as an HMAC-SHA256 hash computed under a server-side pepper. Reading through production environment variables turned up that the pepper had never actually been set, so the code had been falling back to a development constant for the entire life of the deployment. Setting a real one is a single line of configuration, and it is also the thing that breaks everything: the pepper is the HMAC key, so replacing it invalidates every hash already stored. Every provisioned board would fail authentication on its next poll and need a physical reset to recover.
The decision that mattered
Verify against the current pepper first, fall back to a retired one, and re-hash the stored value transparently whenever the old pepper is what matched. Boards migrate themselves the next time they call home, so the rotation is lazy and needs no coordinated re-provisioning or trip to the hardware.
A grace window in which two credentials are valid at once. The retired key keeps working until every board has checked in, and closing that window is a manual step with no natural deadline attached to it. I took a short, bounded exposure over bricking hardware that is not always physically reachable.
Good / Bad / Ugly
A live HMAC key rotated across deployed hardware with no downtime and no re-provisioning. A board reconnected on its next poll and reported in as though nothing had happened.
The code did warn that the pepper was missing. It used console.warn, in a serverless log nobody reads. It now throws on startup in production instead, which would have caught this on day one.
Production ran on a constant from the repo since launch and nothing surfaced it. No alert, no failing test, no check at deploy time. It was found by reading a list of environment variables by hand. The control existed in the code the whole time and was simply never switched on in the one environment where it mattered.
Shipped
Dual-pepper verification with transparent re-hash, live in production and confirmed against a provisioned board.
Next
A deploy-time check that fails the build when a required secret is missing, so the next one cannot ship unset.