Why every busy restaurant needs someone walking the floor

In Order Dispatch Systems at Scale, a previous Kogan.com team member Jerry listed eventual consistency as one of the costs of going event-driven: parts of the system can be out of sync for a while, so our logic has to cope with stale data. In Patterns & Best Practices in Event-Driven Systems, I covered the patterns that make those systems work.

This post picks up the obvious next question. If our systems only agree eventually, how do we know they ever got there?

Three tables that look the same

Picture a pay-at-the-counter restaurant. Three tables have paid, and none of them has food yet.

Table 12: the pasta is nearly ready.
Table 18: asked for their dessert to come out later.
Table 24: their soup docket slipped behind the printer, and nobody is making it.

From across the room, all three look the same: paid and waiting. Only one of them actually has a problem. Table 12 needs a bit more time and table 18 is getting exactly what they asked for, but table 24 won't get anything until someone notices.

An event-driven order system has the same problem. Every service can be doing its own job well while one order falls into the gap between them. The usual answer is reconciliation: compare what should have happened with what actually happened, and deal with the difference. In the restaurant, that's the floor manager doing the rounds.

How much lag is normal?

Some, and that's fine. The order service, the warehouse and the customer's order-tracking page all catch up at different speeds, much like the till, the kitchen and the floor staff in a restaurant. A page built from a read model (a copy of the data shaped for display) will sit behind the system that recorded the change, which is the trade you make for the read being cheap; Microsoft's CQRS guidance covers it.

But waiting doesn't help table 24. Our workflow needs a way to notice missing progress and do something about it.

Who owns what

Before the floor manager can spot a problem, they need to know who to ask. The cashier knows the meal was paid for, but can't tell you whether it reached the table; only the runner knows that.

Services split their facts up the same way:

The order service owns the customer's request.
The payment service owns the payment result.
The fulfilment service owns the shipment booking.

A status page can combine all three, but when something looks wrong, check with the service that owns the fact in question. This is usually called data ownership. It sounds like a diagram exercise until the first time two services disagree about whether a customer has been charged, and you need to know which one to believe.

Putting a number on "eventually"

"Eventually" isn't something a job can check. It needs a rule, for example:

A paid order released for immediate dispatch should have a shipment request within 30 minutes.

Thirty minutes is only an example; yours should come from how long orders actually take. In practice the first number usually comes out of a meeting, where somebody picks a figure that sounds responsible, and it's too tight. It can be loosened once the job has run for a week and you can see what the spread really looks like.

Write the exceptions down alongside the rule as well. Preorders, orders on hold and cancelled orders are all table 18: they're waiting on purpose, and they shouldn't trigger a new dispatch.

Our refund monitoring has got both halves of this wrong at different times. At first it only counted refunds that were reported as failed, so if a reply never arrived, nothing registered at all. The check we added to catch those then counted some refunds that were meant to stay pending, and would have fired on perfectly healthy data.

Passing the deadline is a signal to go and look. A good floor manager asks the kitchen about a late meal before ordering a second one.

The reconciliation job

The job follows roughly the same routine as the floor manager:

List the orders that should have moved on by now.
Pick out the ones waiting longer than expected.
Ask the service that owns each fact what it knows.
Decide what to do next.
Check again later, until the case is resolved or handed to a person.

For step 1, the floor manager works from the seating plan. Watching plates come out of the kitchen would never turn up table 24, because nobody made their soup. A job that only looks at the messages a consumer received has the same blind spot: if an order's message never arrived, the job won't see that order at all.

We've had a real table 24. When an order came off a fraud hold, the event telling SAP never reached it. SAP had no way of knowing it had missed anything; as far as it was concerned, those orders were still on hold. A held order is supposed to wait, so nothing about it looked wrong. The only sign was that our system had released the order and SAP hadn't.

So the job has to start from the authoritative list of orders that should be in fulfilment:

// Read the order store itself, so a lost message can't hide an order
var overdue = await orders.ReadyForDispatchBefore(DateTime.UtcNow - DispatchDeadline);

foreach (var order in overdue)
{
    var booking = await fulfilment.FindBooking(order.BookingOperationId);
    await reconciler.Decide(order, booking); // record it, retry safely, or hand to a person
}

Keeping the query cheap

We nearly shipped a check that would have rescanned millions of order lines on our read replica every minute. It returned the right answer, just in the most expensive way available.

On a busy system, "find every overdue order" can turn into a scan of the whole table, every few minutes, on the same database that's taking new orders. Yet the job only cares about orders that are still waiting, which on a normal day is a small slice of it.

A partial index keeps a separate, smaller list of just those orders. As long as the query filters on the same status, PostgreSQL can search that list instead of scanning the full order history:

CREATE INDEX CONCURRENTLY orders_awaiting_dispatch
    ON orders (ready_for_dispatch_at)
    WHERE status = 'awaiting_dispatch';

That's how we fixed ours: a partial index on the waiting state, ordered by when each order line entered it, turned the scan into a narrow range read.

If noticing a problem a little later is acceptable, you can also read from a replica. Its list may be slightly out of date, so confirm the current state with the owning service before repairing anything.

What usually turns up

When the floor manager checks on a late table, the answer tends to be one of a few things:

What they find	In software	What to do
The meal is ready in the kitchen, but the order book still says "waiting"	The booking exists, but the order service never recorded it	Record the confirmed result and carry on
The docket is behind the printer	The work never started	Ask the owning service to do it, reusing the original idempotency key
The customer changed their mind	The order was cancelled while it waited	Check the current state before repairing anything
The meal was served after the customer cancelled	The booking went through even though the order was cancelled	Use a compensating action, such as cancelling the booking if that's still possible

The first row caught us out with refunds. A refund could be reported as failed and then go through on a retry, but our order system only listened to the first answer. It kept showing the refund as failed, which is exactly when someone is tempted to raise a second one.

Whatever the case, the repair should go through the service that owns the work, the same way a normal request would, and reuse the original request's idempotency key. If the first attempt landed after all, the service recognises the key and doesn't create a second booking. I covered idempotency in my last post, and reconciliation is one of the main reasons it's worth the effort. Don't recreate the work just because a search came back empty, either: the first attempt may still be in progress, or not visible yet.

Check the order again right before repairing it. The customer may have cancelled while the case sat in the queue, and a repair built from an old snapshot would ship something they no longer want.

Compensating actions can fail too. Cancelling a booking is a business operation in its own right, with its own rules: the carrier may already have collected the parcel. Microsoft's compensating transaction guidance covers this well.

A note on replay

Reprinting yesterday's dockets helps the accounts team piece the day back together. Putting them back on the rail cooks yesterday's meals again.

Event replay has the same split. Replaying events to rebuild a status view is fine. Replaying them to a consumer that books shipments is only safe if that consumer handles repeats. EventBridge lets you replay an archive to selected rules on the original event bus, which limits who receives it (EventBridge replay docs), but each receiver still needs to cope with events it has already seen.

Keeping track of open cases

Each open case needs a saved record of what was found, what was tried, who owns it and when to look at it again. Without one, a job that only scans recent orders lets older ones drift out of its window.

It also helps to work through open cases in small batches. If the cause is a struggling dependency, a flood of repairs will only make things worse.

What to measure

CPU, queue depth and error rates tell you whether the services are healthy. They can all look fine while one order has been stuck for two days. For an order workflow, these are more useful:

How many orders are past their deadline?
How long has the oldest unresolved order been waiting?
How many cases are waiting on a person?
Did the reconciliation job actually run? Zero open cases looks great on a dashboard, but not if the job hasn't run since Tuesday.

It's worth breaking things on purpose in a test environment too: drop an event before it reaches a consumer, delay a status update, crash the job halfway through a repair, or cancel an order while it's being investigated. Each time, the case should either be fixed safely or stay visible with a clear next step.

When an agent does the work

As more and more actions are carried out with the help of AI agents, the same rules apply. An agent that times out halfway through a task has the same problem as any other caller: it doesn't know whether the task finished, and retrying without the original idempotency key is how the same work gets done twice. An agent reporting that a task succeeded is also just another status page. The owning services still hold the facts, so the reconciliation job should check with them before trusting the agent's account of what happened.

Conclusion

None of this is especially clever. It's a scheduled query, a place to record open cases and a few numbers on a dashboard. Without it, though, we'd hear about a stuck order when the customer contacts us to ask where it is.

Read the cases as well as closing them. If the same kind keeps coming back, something in the normal flow is broken, and fixing it there leaves the reconciliation job with less to do.

Eventually Consistent: How Do We Know the Systems Eventually Agree?