Webhook Engineering: Payment and Booking Events at Scale

Webhooks seem simple. An external service sends an HTTP POST to your endpoint when something happens. Payment succeeded. Booking confirmed. Refund processed. Your server handles the event and updates state accordingly.

In practice, webhooks are one of the messiest parts of distributed system engineering. They arrive out of order. They get delivered multiple times. They come from multiple sources with different formats. They can arrive before the operation that triggered them has finished processing on your end. And when your endpoint is temporarily down, you miss events that never get retried.

For a travel booking platform that processes real payments, getting webhook handling right is not optional. A missed payment event means a user gets charged without their booking updating. A duplicate event means a refund gets processed twice. An out-of-order event means the system thinks a booking was cancelled before it was created.

Here's how we handle all of this.

The webhook flood

A single booking operation can trigger a cascade of webhook events from multiple sources.

From the payment processor: payment intent created, payment processing, payment succeeded, charge created. That's four events for one payment.

From the travel provider: booking request received, booking confirmed, ticket issued. Three more events.

From our own notification system: email sent, push notification delivered. Two more.

Nine events for one booking, from three different sources, arriving at unpredictable times in unpredictable order. If the user books two flights and a hotel, multiply by three.

Now consider that external services retry aggressively when they don't get a fast enough response. If your webhook endpoint takes more than a few seconds (maybe because it's doing a database write), the sender might retry, and now you're processing the same event twice.

Idempotent event processing

Every webhook handler we write is idempotent. The same event delivered ten times produces the same state as the same event delivered once.

We achieve this by storing a unique event ID for every webhook we receive. The first thing our handler does is check: have I seen this event ID before?

If yes, return a 200 OK immediately without processing. The sender is satisfied (they got their acknowledgment), and we don't duplicate side effects.

If no, store the event ID, process the event, and return 200.

This sounds simple but the timing matters. What if two deliveries of the same event arrive simultaneously? We use a database-level unique constraint on the event ID. The first insert succeeds. The second fails with a uniqueness violation, which we catch and treat as "already processed."

Payment webhook events must be processed idempotently to avoid duplicate actions. A duplicate "payment succeeded" event that triggers a second booking would be catastrophic. A duplicate "refund processed" event that refunds twice would lose money.

Ordering guarantees (or lack thereof)

Here's a fun problem: webhooks don't arrive in order.

A payment processor might send "payment_intent.succeeded" and "charge.created" as separate events. Logically, the charge is created before the payment succeeds. But the "succeeded" event might arrive first because it was processed faster on the sender's side.

Our booking provider might send "booking_confirmed" before "booking_request_acknowledged" because the confirmation webhook fires from a different internal service with lower latency.

We handle this with state machines rather than event sequences. Instead of assuming events arrive in a specific order and breaking when they don't, we model each entity (payment, booking) as a state machine with defined transitions.

A payment can be in states: created, processing, succeeded, failed, refunded. Any event that's valid for the current state is applied. Any event that represents a state the entity has already passed through is ignored. Any event that represents an impossible transition gets flagged for investigation.

If "payment_succeeded" arrives before "payment_processing," we skip the intermediate state and go directly to succeeded. If "payment_processing" arrives after "payment_succeeded," we ignore it because the entity is already in a later state.

This approach is tolerant of any arrival order while still maintaining a consistent state progression.

Failure handling and retries

What happens when our webhook endpoint is down? External services have their own retry policies, and they vary wildly.

Some payment processors retry up to a hundred times over several days with exponential backoff. Others retry five times and give up. Some travel providers don't retry at all: they send the event once, and if you miss it, too bad.

We can't control the sender's retry behavior. What we can control is our ability to recover from missed events.

Reconciliation jobs. We run periodic jobs that compare our internal state against the source of truth (payment processor state, booking provider state) and detect any drift. If our system shows a payment as "processing" but the processor shows it as "succeeded," we know we missed the webhook and can process the state change from the reconciliation.

Event polling. For providers that support it, we poll recent events on a schedule as a backup to webhook delivery. This catches any events that were lost in transit.

Health monitoring. We track the last-received timestamp for each webhook source. If we haven't received events from a source in longer than expected, we investigate. The absence of events is a signal that something might be wrong.

Real-time monitoring and alerting for all services means we catch delivery failures quickly. A missed webhook that isn't caught for hours can cascade into customer-visible problems.

Different sources, different challenges

Each external system we receive webhooks from presents unique engineering challenges.

Payment processor webhooks are the highest-stakes. They drive financial state transitions. They come with signature verification (we cryptographically verify each webhook to prevent spoofing). They have complex event types with nested data structures. And they're time-sensitive: a user is waiting for their booking confirmation.

Travel provider webhooks are the most unpredictable. Different providers have different webhook formats, different delivery guarantees, and different event schemas. We built a normalization layer that converts provider-specific events into a canonical internal event format before processing.

Internal webhooks (notification delivery receipts, document processing completion) are the most reliable because we control both ends. But they still need idempotency because internal services can retry too.

For each source, we maintain a dedicated handler with source-specific parsing, validation, and normalization. The canonical internal events then flow into a shared processing pipeline that handles state transitions and side effects.

Silent failures and how to catch them

The most insidious webhook problem is the silent failure. Everything looks fine: your endpoint is up, events are arriving, processing isn't throwing errors. But somewhere in the pipeline, an event got dropped or processed incorrectly, and no one noticed.

Maybe the event arrived but the state transition logic had a bug for a rare edge case. Maybe the event was marked as processed but the side effect (updating the booking record) silently failed. Maybe the event was from a newer API version with a field your code doesn't handle yet.

We catch silent failures through three mechanisms.

End-to-end consistency checks. After every booking, we verify that our internal state matches the payment processor and travel provider within a time window. Any mismatch triggers an alert.

Event processing metrics. We track not just whether events are received but whether they produce expected state changes. An event that arrives but doesn't change any state might be a duplicate (fine) or a processing failure (not fine).

User-facing consistency. When the user asks about their booking status, the agent checks live state from the sources of truth, not just our internal database. If there's a mismatch, it's surfaced and resolved in real time.

Multi-layer booking idempotency prevents duplicate bookings from webhook retries. But idempotency only prevents duplicates. It doesn't prevent missed events. The full solution requires idempotency plus reconciliation plus monitoring. All three together give us confidence that our system state stays accurate even when the webhook delivery is messy.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.