Short write‑ups of problems we hit running a real commerce operation and what we changed in the code because of them. No product announcements; just the reasoning.
One morning every page in the back office threw errors for about two seconds, and the trail led somewhere strange — a queue that had been renamed in June and never given a worker, quietly growing to 6.9 million jobs and 18 GB inside Redis until the box was living in swap and one routine Lua script stalled past the busy threshold. How we traced a frontend error bucket to a key nobody was reading, why deleting it had to happen before fixing it, the eager-loading bug hiding behind the first fix, and the one-hour check that would have caught the whole thing in week one.
Everything that is not a page view runs on Horizon — twenty‑six supervisors across two servers, from "urgent" to "historic backfill". It is the best thing about the system and was the source of its worst mornings: stray workers surviving a deploy and running a query 780,000 times an hour; a supervisor killed but its children kept alive so every queue ran twice; a retry‑after shorter than the job; ten thousand jobs in a queue from a loop that should have been one. What each one taught us, in the order they hurt — and how the beast was finally tamed.