--- title: "Webhooks at Scale: Lessons From Travel Event Delivery" description: "At-least-once delivery, request-signature signatures, exponential retry, and dead-letter queues — how we deliver millions of booking event webhooks without losing or duplicating events." canonical: https://nowah.xyz/blog/webhooks-at-scale-travel-event-delivery lastModified: "2026-08-07T08:12:48.130Z" --- # Webhooks at Scale: Lessons From Travel Event Delivery At-least-once delivery, request-signature signatures, exponential retry, and dead-letter queues — how we deliver millions of booking event webhooks without losing or duplicating events. A booking sat in limbo for six hours because the webhook never arrived\. The developer's system was waiting for a \`booking\.confirmed\` event to show the confirmation page to the traveler\. The event was generated on our side, serialized into a payload, and sent\. But the developer's server was briefly unreachable, our single delivery attempt failed, and the event was lost\. The traveler had paid, the ticket was issued, but they were staring at a "processing" screen. Support got involved. It was not a good experience for anyone. We rebuilt our webhook system after that incident. The guiding principle became: an event must be delivered, period. If it cannot be delivered now, try again. If it cannot be delivered after exhaustive retries, capture it for investigation and manual replay. Never silently drop an event. ## At-least-once delivery ![Illustration for this section](https://pics.nowah.xyz/website-media/developer-experience-012-img-1-retry-schedule.webp) We guarantee at-least-once delivery. This means every event will be delivered at least once, but it might be delivered more than once. We chose this over exactly-once delivery because exactly-once is effectively impossible in distributed systems without extremely complex two-phase commit protocols that introduce their own failure modes. At-least-once means the developer's webhook endpoint must be idempotent. Receiving the same event twice should produce the same result as receiving it once. We help with this by including a unique event ID in every payload. Developers can track event IDs they have processed and skip duplicates. We also include a sequence number per webhook subscription. Events are numbered sequentially, so a developer can detect gaps (missed events that might arrive later via retry) and out-of-order delivery (event 5 arriving before event 4 due to retry timing). ## Retry logic When a delivery attempt fails (timeout, non-2xx response, connection refused), we retry with exponential backoff over 24 hours. The retry schedule is: 1 second, 5 seconds, 30 seconds, 2 minutes, 10 minutes, 1 hour, 6 hours, 24 hours. That is eight attempts spread over a day. Each attempt includes random jitter (up to 20% of the backoff interval) to prevent thundering herds when many webhooks fail simultaneously and all retry at the same time. We consider a delivery successful when the endpoint returns a 2xx status code within 30 seconds. Anything else — 4xx, 5xx, timeout, connection error — triggers the next retry. The exception is 410 Gone, which we interpret as "this endpoint no longer exists" and we stop retrying immediately and mark the subscription for review. Between retries, the event sits in a durable queue. It cannot be lost to a server restart or deployment. We use a background job processing system with persistent storage for the queue, so even a full system restart picks up pending retries where they left off. ## request-signature signatures ![Supporting diagram](https://pics.nowah.xyz/website-media/developer-experience-012-img-2-signature-verify.webp) Every webhook payload includes a cryptographic signature that the receiver can use to verify the event came from us and was not tampered with in transit. We use request-signature-SHA256 with a per-subscription signing secret. The signature covers the full payload body and a timestamp: ``` X-Nowah-Signature: sha256=a1b2c3d4... X-Nowah-Timestamp: 1710500000 ``` The receiver recomputes the request-signature using their signing secret and the payload body, then compares it to the signature in the header. If they match, the payload is authentic. We include the timestamp so receivers can reject old payloads and prevent replay attacks — if the timestamp is more than five minutes old, the event should be rejected. We provide verification [code examples](/blog/documentation-as-product-test-code-examples) in our documentation for a single typed language across the stack, Python, and Go. The verification is a few lines of code, but getting it wrong is a security risk, so we make it as copy-paste-easy as possible. ## Dead-letter queues After eight retry attempts over 24 hours, if the event still has not been delivered successfully, it moves to a dead-letter queue. Dead-lettered events are not deleted. They sit in a separate queue where they can be investigated and replayed. The [developer dashboard](/blog/building-developer-dashboard-developers-use) shows dead-lettered events with the delivery history — every attempt, the response received (or lack thereof), and the timestamps. This lets developers diagnose why their endpoint was unreachable and replay the events once the issue is fixed. Replay is a first-class operation. From the dashboard or the CLI, developers can select dead-lettered events and re-deliver them. The replay sends the exact original payload with original headers and a fresh signature. It is indistinguishable from the original delivery from the receiver's perspective. ## Event ordering We do not guarantee strict event ordering. Event 3 might arrive before event 2 if event 2 failed its first delivery attempt and is in retry while event 3 succeeds on the first try. This is a deliberate design decision. Guaranteeing strict ordering requires holding back all subsequent events while a failed event is retrying, which means a single unreachable endpoint blocks all event delivery. The tradeoff is not worth it. Instead, we include sequence numbers and timestamps in every event\. Developers who need to process events in order can buffer them and sort by sequence number before processing\. Most developers do not need strict ordering — a \`booking\.confirmed\` event is meaningful regardless of whether it arrives before or after a \`booking\.payment\_collected\` event\. For the cases where ordering matters, we recommend using the booking [state machine](/blog/building-booking-state-machine) as the source of truth rather than webhook ordering. Query the booking's current status via the API rather than deriving it from the sequence of received webhooks. The API always returns the authoritative current state. ## Testing webhooks in development Testing webhooks during development requires getting events from our servers to a developer's local machine. We built this into the CLI. \`nowah webhooks listen\` opens a secure tunnel from our servers to the developer's localhost\. Webhook events route through the tunnel and arrive at the [local development](/blog/local-development-microservices) server as if it were a public endpoint. Events display in the terminal with syntax-highlighted JSON, and the developer can inspect headers, verify signatures, and see the full payload. Event filtering narrows the stream: \`\-\-event booking\.\*\` shows only booking events\. This is useful when a developer is working on a specific feature and does not want to see every event type\. The dashboard provides a complementary testing tool: a "Send Test Event" button for each event type. It generates a realistic test payload and delivers it to the configured endpoint. Combined with CLI forwarding, this gives developers a complete local testing setup for webhooks without deploying anything. We also support replaying historical events to the local tunnel\. A developer debugging a webhook processing bug can grab the event ID from their logs, run \`nowah webhooks replay \`, and receive the exact same payload their production system received\. Same data, same headers, same signature\. This makes reproduction trivial\. [Webhook reliability](/blog/webhook-reliability-payment-events) is infrastructure work that users never see when it works correctly. They only notice when it fails. Our goal is to make failures as rare as possible and recoverable when they happen. At-least-once delivery, exhaustive retries, dead-letter queues, and easy replay make that goal achievable. --- Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. [Plan your next trip](https://app.nowah.xyz).