Webhook Reliability: Processing Payment Events Without Data Loss

A payment webhook arrives twice. Once from the original event, once because our acknowledgment was slow and the payment processor retried. If we process both, we might update the booking twice, send two confirmation emails, or worse, trigger duplicate downstream operations. If we process neither (because our handler choked on the first and rejected the second as duplicate), the booking state never updates and the traveler is stuck.

Processing payment webhooks exactly once, reliably, even when things go wrong, is one of the most important infrastructure problems in a booking platform. Get it wrong and you have financial discrepancies, confused travelers, and a reconciliation nightmare.

Signature verification

Before processing any webhook event, we verify its authenticity. The payment processor signs each webhook payload with a secret key. We compute the expected signature using the raw request body and our copy of the secret, then compare it to the signature in the request header.

If the signatures don't match, the event is rejected. No processing. No acknowledgment. This prevents spoofed events. An attacker who discovers our webhook URL can't send fake "payment succeeded" events to mark bookings as paid without actual payment.

The verification uses the raw request body, not parsed JSON. This is a common gotcha. If your web framework parses the JSON body before your webhook handler sees it, and the parsed-then-serialized body differs from the raw body (different key ordering, whitespace changes), the signature check fails on legitimate events. We capture the raw body before any parsing.

Idempotent processing

Every webhook event has a unique event ID assigned by the payment processor. When we receive an event, the first thing we do after signature verification is check whether we've already processed an event with this ID.

We maintain a record of processed event IDs with a TTL matching the processor's retry window (typically 72 hours). If the event ID exists in our records, we return a 200 acknowledgment without processing. The processor sees the 200, considers the webhook delivered, and stops retrying.

If the event ID is new, we record it and proceed with processing. The recording and processing happen in a transaction: if processing fails and the event ID record gets rolled back, the processor will retry and we'll process it again. If processing succeeds, the event ID is recorded and duplicates are rejected.

This approach handles the most common duplicate scenario: the processor sends an event, we process it, our acknowledgment response is slow, the processor retries, we receive the duplicate. The first processing completes, records the event ID, and the retry is rejected cleanly.

Background job processing

We don't process webhook events synchronously in the HTTP handler. The handler does three things: verify the signature, check for duplicates, and enqueue a background job. Then it returns 200 immediately.

This is important because webhook processing can be complex. A "payment succeeded" event might trigger: updating the booking status, sending a confirmation email, firing a push notification, updating the trip record, and recording analytics events. All of that takes time. If we did it synchronously, the handler might take 5-10 seconds. Many payment processors have a 5-second webhook timeout. If we exceed it, they retry, and we get the duplicate problem.

By enqueuing and returning immediately, we acknowledge within milliseconds. The actual processing happens asynchronously in a worker with its own retry logic.

The background job has the event payload and the event ID. The worker processes the event and records the result. If the worker fails (crashes, timeout, external service error), the job retries with exponential backoff. The idempotent handler on the webhook endpoint prevents duplicates from the processor's retries, and the idempotent job handler prevents duplicates from our own retries.

Event type routing

We receive multiple webhook event types. Payment succeeded, payment failed, charge disputed, refund completed, and several others. Each event type has different processing logic.

We route events to type-specific handlers. The "payment succeeded" handler updates the booking to confirmed. The "payment failed" handler marks the booking as failed and notifies the AI agent. The "charge disputed" handler flags the booking for review and triggers a support workflow.

Having separate handlers per event type makes each handler simpler and easier to test. It also means a bug in the "charge disputed" handler doesn't affect "payment succeeded" processing. Isolated failure domains at the handler level.

Monitoring the webhook pipeline

We track several metrics for webhook health.

Processing lag is the time between when the processor sent the webhook and when we finished processing it. This includes network transit, queue time, and processing time. Our target is under 30 seconds for payment events.

Failure rate is the percentage of webhook events that fail processing on first attempt. An increase indicates a bug in our handler or degradation in a downstream service.

Duplicate rate is the percentage of events that are duplicates. A low rate (1-5%) is normal and expected. A high rate indicates that our acknowledgment is too slow (the processor keeps retrying) or that our event ID deduplication has a gap.

Event type distribution tracks the mix of event types we receive. A sudden spike in "payment failed" events is a signal that something is wrong with our payment flow, not just the webhook infrastructure.

Building reliable webhook processing

For anyone implementing payment webhook handling, here's the pattern.

Verify signatures on every event. No exceptions. Use the raw request body for signature computation.

Check event IDs before processing. Maintain a deduplication store with a TTL matching the processor's retry window.

Acknowledge immediately, process asynchronously. Return 200 as fast as possible. Do the real work in a background job.

Make your event handlers idempotent at the business logic level. Even with deduplication, edge cases (race conditions between the webhook and the API-initiated booking flow) can cause the same event to be processed in different code paths. The business logic should handle this gracefully.

Monitor processing lag, not just success rate. A webhook pipeline that succeeds but takes 5 minutes is still a problem for the traveler experience. Confirmation emails, booking status updates, and trip synchronization all wait for webhook processing to complete.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.