Accept, then call back

The wait

In Lesotho most people pay from a USSD menu. The customer dials, a prompt appears, they enter a PIN, and somewhere in the middle of that they might put the phone down to go and find their wallet. EcoCash's pay-merchant call reflects this honestly: it is synchronous, and it returns only once the customer has finished walking through the prompt.

Our /initiate endpoint used to make that call inline. So when NALO, the partner routing traffic to us, posted a payment request, their HTTP connection stayed open for as long as the customer took. Sometimes that was four seconds. Sometimes the handset was off and it was never. The gateway timeouts we kept seeing were the architecture working exactly as written.

The tempting fix is a longer timeout, and it does not work, because there is no correct number to pick. You are not waiting on a computer. You are waiting on a person deciding whether to buy something, and any value you choose is either too short for a slow customer or too long for a dead handset.

Accept, then call back

So /initiate stopped calling the provider. It persists the payment as PENDING and returns immediately with a 202. The provider call moved out into PaymentService.DispatchAsync and runs off the request thread. The outcome arrives later, on a callback.

That is a contract change, and it is worth stating plainly: the response to /initiate is never the final answer. It tells you the payment was accepted and nothing else. The partner callback is the source of truth. We flagged this to NALO before it shipped, because a caller who reads the initiate response as an outcome will be wrong in a way that eventually costs somebody money.

Provider callbacks arrive on /payments/v1/providers/{provider}/callback/{secret}, authenticated by an unguessable path segment compared with CryptographicOperations.FixedTimeEquals, because the providers carry no OAuth token to present. M-Pesa correlates by the ThirdPartyConversationID we sent it, EcoCash by request_id. Where EcoCash's outcome is ambiguous we deliberately do not apply it and leave it for reconciliation, on the view that a wrong status does more damage than a late one.

The problem this created

Lifting work off the request thread moves it somewhere, and where it went was an in-process Channel<string>. That is fast and it is free, and it is also memory. Restart the App Service between persisting a payment and dispatching it and the payment sits PENDING forever. Scale to two instances and both could dispatch the same payment, which where money is concerned is the expensive kind of bug.

An external review flagged exactly this. The obvious remedy was a broker: the platform already runs Service Bus for partner callback delivery, so reaching for it again would have been the conventional answer, with a dedicated outbox table as the other one.

The persisted row is the outbox

We kept the in-process channel as a fast path and made the database the source of dispatch truth. Three things hold that up.

Dispatch state never lives only in memory. A payment is PENDING with no PAYMENT_INITIATIONaudit until a dispatcher has actually reached a provider, so the question “was this sent?” is always answerable from a row.

A sweep every thirty seconds re-queues anything persisted but never dispatched: PENDING, older than fifteen seconds, younger than twenty-four hours, no initiation audit, no live claim. A restart therefore delays dispatch by at most one sweep interval, and it never loses the work. That query is public static and pinned by tests, because it is the recovery contract and deserves to be read as one.

Dispatch is claimed atomically before any provider call. A conditional ExecuteUpdate sets DispatchClaimedAt only while the row is still PENDING and either unclaimed or holding a claim older than a five-minute lease, so exactly one racing instance wins. The lease is the part that matters most. A lock held by an instance that dies is held forever; a lease held by an instance that dies expires in five minutes and the sweep picks the payment back up.

What it cost

Recovery is not instant. A restart costs up to thirty seconds, and a dispatcher that dies mid-call costs the remainder of its lease, up to five minutes. Measured against a customer who is already standing at a USSD prompt, that is acceptable, and the partner callback carries the final outcome either way.

There is also a real gap in the testing. The EF InMemory provider cannot execute conditional updates, so under test the atomic claim degrades to check-then-act. The guard logic is pinned, but the actual race is only exercised against a relational database in dev, sandbox and production. That limitation is written into the decision record so nobody has to rediscover it.

What it bought

No broker sits on the money path. The failure domain stays the database the API already cannot work without, so there is one fewer thing that can be down at three in the morning. Scale-out is safe for dispatch correctness, and throughput stays bounded by provider round-trips, which is where the real limit always was.

The shape turned out to be reusable. Approved disbursements take the same route: persisted first, dispatched out of band, with DispatchedAt doubling as the claim marker on a conditional update that flips Approved to Pending exactly once. Every product on the estate that takes a payment inherits all of this without knowing it is there, which is the argument for building rails once and reusing them.