Short write‑ups of problems we hit running a real commerce operation and what we changed in the code because of them. No product announcements; just the reasoning.
One morning every page in the back office threw errors for about two seconds, and the trail led somewhere strange — a queue that had been renamed in June and never given a worker, quietly growing to 6.9 million jobs and 18 GB inside Redis until the box was living in swap and one routine Lua script stalled past the busy threshold. How we traced a frontend error bucket to a key nobody was reading, why deleting it had to happen before fixing it, the eager-loading bug hiding behind the first fix, and the one-hour check that would have caught the whole thing in week one.
After a flush, the first visitor to each storefront page pays for the render. We have written the thing that pre‑pays three times: a link‑following crawler from a package, a two‑pass seeder‑then‑deep crawl with concurrency rules, and — the one we kept — a list of the pages people actually visit, fetched in order of traffic, at a concurrency that can never outbid a live shopper. Why each version was wrong for the next stage, and the surge‑protection bug that made the warmer most aggressive exactly when it should have been gentlest.
Fifty‑five kinds of email, one pipeline. How a newsletter to a hundred thousand people is prepared, sent through Amazon SES from a queue, tracked back event by event through SNS webhooks — delivered, opened, clicked, bounced, complained — attributed to the order it caused, and kept from becoming the retry storm that once throttled us at 1,700 calls a second.
Everything that is not a page view runs on Horizon — twenty‑six supervisors across two servers, from "urgent" to "historic backfill". It is the best thing about the system and was the source of its worst mornings: stray workers surviving a deploy and running a query 780,000 times an hour; a supervisor killed but its children kept alive so every queue ran twice; a retry‑after shorter than the job; ten thousand jobs in a queue from a loop that should have been one. What each one taught us, in the order they hurt — and how the beast was finally tamed.