Short write‑ups of problems we hit running a real commerce operation and what we changed in the code because of them. No product announcements; just the reasoning.
One morning every page in the back office threw errors for about two seconds, and the trail led somewhere strange — a queue that had been renamed in June and never given a worker, quietly growing to 6.9 million jobs and 18 GB inside Redis until the box was living in swap and one routine Lua script stalled past the busy threshold. How we traced a frontend error bucket to a key nobody was reading, why deleting it had to happen before fixing it, the eager-loading bug hiding behind the first fix, and the one-hour check that would have caught the whole thing in week one.
Twenty‑nine steps between "push to production" and "the warehouse is on the new code", and not one of them makes a picker wait. A walk through one deploy as it actually runs — releases and symlinks, caches, migrations on one host only, a front‑end build that is skipped when nothing changed, checksums that decide what to flush, queues told to finish, workers reloaded from an anchor, SSR restarted only if it must, a broadcast that tells open tabs to refresh, and a few chores at the end.
No Kubernetes, no managed database, no serverless. aiku runs a multi‑country commerce operation on a handful of rented bare‑metal machines behind Cloudflare — a primary, a replica and a staging box — and we would choose it again. Why metal beats "cloud" for a workload like ours, what the boxes actually do, and the two incidents that are the honest cost of owning the iron.
Everything that is not a page view runs on Horizon — twenty‑six supervisors across two servers, from "urgent" to "historic backfill". It is the best thing about the system and was the source of its worst mornings: stray workers surviving a deploy and running a query 780,000 times an hour; a supervisor killed but its children kept alive so every queue ran twice; a retry‑after shorter than the job; ten thousand jobs in a queue from a loop that should have been one. What each one taught us, in the order they hurt — and how the beast was finally tamed.
The search box in the staff app finds a customer, an order, a pallet, a mailshot, an employee, a chat message — forty‑two model types — scoped to what you may see, typo‑tolerant, in one keystroke. How it runs on Typesense next to the application, why three places must change in lockstep when a new thing becomes searchable, the reindex command that rebuilds a section while people keep working, and the afternoon the search engine's raft log got stuck.