Short write‑ups of problems we hit running a real commerce operation and what we changed in the code because of them. No product announcements; just the reasoning.
Fifty‑five kinds of email, one pipeline. How a newsletter to a hundred thousand people is prepared, sent through Amazon SES from a queue, tracked back event by event through SNS webhooks — delivered, opened, clicked, bounced, complained — attributed to the order it caused, and kept from becoming the retry storm that once throttled us at 1,700 calls a second.
Selling across Europe and the UK means every order has to answer one question before a price is final — and the answer depends on two addresses, a postcode, and whether a VAT number is real. How aiku validates tax numbers against VIES and HMRC (with retries, rate limits and an audit trail), and the decision tree that turns a shipping address into a tax category, Canaries and Madeira included.
For years every line we sold carried the same VAT rate, so one rate per order was simply true. Then the range grew to include tea — zero‑rated in one market, reduced in another — and "which rate" became a question per line. The answer was not "read the product's tax code": it was presets instead of codes for staff, tax versioned like price on the historic line, frozen once sold, and a migration that re‑rated the open baskets to the penny.
When the application moved to long‑lived workers, the commercial APM agent we had run for years stopped telling the truth — and on the test runner it stopped the tests. We replaced it with NightOwl, an open‑source telemetry stack for Laravel: an agent on each box, a local buffer that survives outages, and every request, job, query, exception and log line landing as rows in a PostgreSQL database we own, on a server we already pay for, with retention measured in years. The best part arrived by accident — an AI assistant connected straight to that database with read‑only SQL.
Everything that is not a page view runs on Horizon — twenty‑six supervisors across two servers, from "urgent" to "historic backfill". It is the best thing about the system and was the source of its worst mornings: stray workers surviving a deploy and running a query 780,000 times an hour; a supervisor killed but its children kept alive so every queue ran twice; a retry‑after shorter than the job; ten thousand jobs in a queue from a loop that should have been one. What each one taught us, in the order they hurt — and how the beast was finally tamed.
Every storefront page is rendered once on the server — so crawlers and slow phones get full HTML — and once more in the browser, where it becomes an app. How Inertia server‑side rendering runs here (a Bun process under a supervisor, health‑checked on every deploy), what a 418 KB props payload taught us about where the second actually goes, the silent client‑side‑rendering incident, and the three levers we keep on the list.
From the packing bench, one tap books the shipment, prints the label and writes the tracking number onto the delivery note — for eight carrier integrations across four countries, each with its own dialect: SOAP here, REST there, XML that breaks on an ampersand, a service that refuses more than one parcel, cash‑on‑delivery that must go on exactly one box. What the shipment object has in common everywhere, what each carrier taught us, and why every integration ships with a "manual" twin.
Every count, total and trend in aiku is precomputed — by ~650 small "hydrator" jobs that run at the point of change and a nightly sweep that catches what they missed, into ~150 stats tables and 35 partitioned time series at five frequencies. Why we chose that over computing on read, the bug where a single‑day redo flattened a whole month, and the index traps in partitioned tables.
Every deploy was emptying the storefront cache and restarting server-side rendering, even when nothing on the front end had changed. The cause was one string baked into one JavaScript chunk. The fix was a checksum and a meta tag.
The same candle is one thing to the buyer, a box of twelve to the warehouse, and a litre of base to the factory. aiku models that with three corners — the trade unit (the physical thing), the stock (that thing, in this warehouse), the product (that thing, as sold in this shop) — and a number on every edge. Why a fraction is sometimes right and sometimes a bug, and why changing a number on an edge while an order is in flight is the most dangerous edit in the system.
The group catalogue — 62,000 master products feeding 317,000 shop products across 34 open shops — is where a thing is described once and pushed to every shop that sells it, so that in theory one person at one desk can run the lot. Majors and minors for multi‑currency prices, a "follow the master" switch at three levels, the scalar column we are retiring because it silently mixed currencies, and a price cascade that ran for a month multiplying RRPs by pack size before a repair put 1,635 values back.
A card payment arrives three ways at once — the browser says "done", the redirect says "done", and a webhook says "captured" — and only one of them is money. The payments layer we built this summer: server‑verified status, one processing path, locked idempotency, a recoverable state machine, a gateway log for every event, a sweeper, and a playbook that made the next provider a week's work.
Staff access in aiku is granted by job position — "dispatch clerk in the Spanish warehouse", "customer‑service supervisor for the UK shop" — not by ticking permissions. Positions expand to roles, roles to permissions, all scoped to the group, organisation, shop or warehouse they name. Why we chose that, how a permission is checked in the same place for the UI, the API and the MCP server, the "team id" that makes it all silently false if forgotten, and the things we refuse to do with an admin flag.
A live chat on every storefront, answered from the staff app by agents who see the customer's account, orders and history beside the conversation; messages translated both ways; a session that can become a ticket, a Slack thread or a summary; presence and typing over WebSockets; a dashboard of who is on the site right now and from where. And the tone rules we wrote down when an assistant started drafting replies.
226,000 images — product photos, family banners, logos, pallet photos, chat attachments — stored once and never resized by us. Every image on every page is a signed URL to an image proxy that resizes, crops and converts on the fly: AVIF and WebP with a fallback, a retina variant of each, the right width for the slot, cached at the edge. Why a proxy beat pre‑generated thumbnails, how the signature stops the proxy being a free resizer for the internet, and the "picture" element every card uses.
A trade buyer reordering forty lines, a dropshipper running a shop on someone else's platform, and a brand whose pallets we store — all log into the same customer portal, built on the same code as the staff app, living on the storefront's own domain. What each sees, what they share, why the dropshipper's order is the most complicated object in the system, and the two things the portal refuses to let a customer do.
Why a trading company wrote its own CMS instead of bolting a shop onto a page builder, and how two front‑end engineers turned "we need a product page" into seventy‑odd web blocks, a workshop that edits the live site, snapshots you can publish and roll back, and a storefront that knows its own catalogue.
Running a third‑party fulfilment business inside the same system as your own warehouse: pallets, boxes and oversize items that belong to someone else, stored items inside them, rental agreements with clauses, and a recurring bill that consolidates itself on the customer's cycle. Why a pallet's lifecycle needs twenty states and a separate "booked in" from "received", and how the 3PL customer sees all of it in their own portal.
The search box in the staff app finds a customer, an order, a pallet, a mailshot, an employee, a chat message — forty‑two model types — scoped to what you may see, typo‑tolerant, in one keystroke. How it runs on Typesense next to the application, why three places must change in lockstep when a new thing becomes searchable, the reindex command that rebuilds a section while people keep working, and the afternoon the search engine's raft log got stuck.
Our storefronts are server-rendered and personalised, which is the worst case for an HTTP cache. Here is how we put Varnish in front of them anyway — two hash buckets, a logged-in bit, tagged objects, surgical bans, and a warmer that only warms what people actually visit.