Support is full of tickets. Some customers were charged twice, and some were charged once but have no order.

01Symptom

POST /orders charges the customer's card through a third-party payment provider and then inserts an order row. On flaky mobile networks, users tap "Place order", see a spinner for 5 seconds, and the app auto-retries up to 3 times. Support is now flooded: some customers have two charges and two orders for one intent, and some have one charge and no order at all.

02Constraints

• One Go API service, 3 instances behind a load balancer; one Postgres primary
• 300 orders/sec at peak
• Payment provider p50 400ms, p99 6s, occasionally times out after 10s
• Client timeout is 5s with up to 3 automatic retries, and a retry can land on any instance
• The provider supports an idempotency-key header, but the current code never sends one
• Order rows must never be duplicated, and a customer must never be charged twice for one intent
• The API must stay available when the provider is slow — no unbounded waiting

03Evidence

• →Duplicated orders always have two different order IDs and two separate provider charge IDs, within seconds of each other, for the same user and amount
• →The charged-but-no-order cases cluster around requests that took 5–10s, i.e. where the client gave up and the server was still working
• →Restarts and deploys during the incident window line up with several of the charged-without-order tickets
• →No request ever carried an idempotency key, so the provider had no way to recognise a repeat of the same charge

→The question

Redesign POST /orders so that retries are safe. What identifies one intent across three instances, and where does a charge get reconciled when the process dies mid-call?

04Your prediction