--- title: Chaos Engineering for Travel Booking description: "How we intentionally break things to ensure the platform handles real failures gracefully — failure simulation, blast radius control, and confidence building." canonical: https://nowah.xyz/blog/chaos-engineering-travel-booking lastModified: "2026-08-07T03:51:30.639Z" --- # Chaos Engineering for Travel Booking How we intentionally break things to ensure the platform handles real failures gracefully — failure simulation, blast radius control, and confidence building. "We think the system handles payment failures." That's what we used to say. Now we say "We tested 200 payment failures last week and all recovered correctly." The difference between those two statements is chaos engineering. Believing your system is resilient and proving your system is resilient are fundamentally different activities. Belief comes from reading the code and deciding it looks correct. Proof comes from injecting failures and watching what actually happens. We've found that what actually happens frequently diverges from what the code suggests should happen. In a travel booking platform, failures aren't theoretical. Payment processors go down. Travel data providers time out. Database connections drop. Network partitions happen. The question isn't whether these failures will occur. It's whether your system handles them correctly when they do. ## The philosophy for payment-critical systems ![Illustration for this section](https://pics.nowah.xyz/website-media/infrastructure-056-img-1.webp) Chaos engineering in a system that charges credit cards and reserves airline seats requires more discipline than chaos engineering in a content delivery system. The consequences of a poorly controlled experiment are financial. A double charge, a lost booking, or a phantom reservation costs real money and real traveler trust. Our chaos engineering follows a strict protocol. Every experiment starts with a hypothesis: "If the payment processor times out during capture, the booking system should hold the booking in a pending state and retry after 30 seconds." The hypothesis is specific, testable, and has a clear success criterion. The experiment is designed to test exactly that hypothesis. We inject exactly one failure at a time. We don't inject a payment timeout AND a database failure simultaneously, because if something goes wrong, we can't attribute the cause. One variable per experiment. We measure the system's behavior against the hypothesis. Did the booking enter the pending state? Did the retry happen after 30 seconds? Did the traveler see an appropriate status message? Did the booking eventually complete or fail cleanly? If all answers match the hypothesis, we have evidence of resilience. If any answer diverges, we have a bug to fix. ## Types of failures we simulate Our failure injection covers four categories. **Network failures.** Partition between the API server and the database. Partition between the API server and the payment processor. Elevated latency on external API calls (adding 5, 10, 15 seconds to every response). Packet loss on the connection to the travel data provider. These simulate the most common real-world failures. **Dependency failures.** Payment processor returning 500 errors. Travel data provider returning malformed responses. Cache becoming unreachable. The AI model provider returning rate limit errors. Each external dependency has its own failure modes, and we simulate the ones we've seen in production plus the ones we fear. **Database failures.** Connection pool exhaustion (all connections in use, new queries wait). Slow queries (injecting 5-second delays on specific tables). Write failures (simulating disk full or replication lag). Transaction deadlocks. The database is the most critical internal dependency, and its failure modes are the most varied. **Queue failures.** Worker process crash during job processing. Queue overflow (more jobs produced than consumed). Message loss (job acknowledged but not completed). Duplicate delivery (same job processed twice). Queue failures are particularly insidious because they're silent. The system doesn't error. It just stops making progress. ## Blast radius control ![Supporting diagram](https://pics.nowah.xyz/website-media/infrastructure-056-img-2.webp) The most important rule of chaos engineering in a production-adjacent system: never break production for [real travelers](/blog/beta-testing-real-travelers-synthetic-data-misses). Every experiment has a blast radius that we define, control, and monitor. Most of our chaos experiments run in the staging environment, which mirrors production's configuration but doesn't serve real traffic. Staging experiments can be aggressive. We can crash workers, partition networks, and corrupt data without affecting anyone. When we run chaos experiments closer to production (which we do for certain scenarios that staging can't represent), we use isolation. The experiment affects a specific service instance behind a feature flag. Real traffic routes to healthy instances. The experiment instance receives synthetic traffic. If the experiment causes a crash or data inconsistency, only synthetic data is affected. We have a hard invariant that every chaos experiment verifies: zero double bookings. This is the one failure mode we consider existential. A traveler should never be charged twice for the same flight. Every experiment that touches the booking flow includes an assertion that checks for duplicate charges. If the assertion fails, the experiment stops immediately and we treat it as a P1 incident. ## Measuring behavior under failure "The system handled the failure" is not a useful measurement. We need specifics. How long did recovery take? What did the traveler see during the failure? Were there any data inconsistencies? Did related features degrade? Our chaos measurements include: time to detection (how quickly did our monitoring notice the injected failure), time to impact (how long before the failure affected traveler-facing behavior), time to recovery (how long after the failure was resolved did behavior return to normal), and data integrity (were all records consistent after recovery). For booking-specific experiments, we also check: payment state consistency (does the payment record match the booking record), [idempotency](/blog/idempotency-travel-booking) (did retry logic prevent duplicate actions), and notification delivery (did the traveler receive accurate status updates throughout the failure and recovery). We record all measurements and trend them over time. Our goal is that each measurement improves or holds steady. If time to recovery increases after a code change, we investigate before the code ships to production. ## The practice schedule Chaos engineering isn't something you do once and declare victory. Failure handling code rots like any other code. New features introduce new failure modes. Infrastructure changes alter the blast radius of existing failures. We run chaos experiments on a schedule. Weekly experiments in staging cover the core scenarios: payment failure, booking failure, database failure, queue failure. These are automated and run as part of our CI pipeline. If any experiment fails its assertions, the pipeline blocks. Monthly experiments are more creative. We pick a scenario we haven't tested before, design an experiment, and run it. These often reveal surprising failure modes. Last month we discovered that if the cache and the database disagree about a booking's status (due to a write that succeeded in the database but failed to update the cache), the traveler sees a stale status for up to 5 minutes. Not catastrophic, but confusing. Quarterly experiments involve the full [incident response](/blog/incident-response-travel-platform) team. We inject a failure without telling the on-call engineer what it is, and we observe how the team detects, diagnoses, and resolves it. These exercises test the people and the process, not just the code. ## Start chaos engineering for your own platform If you haven't started chaos engineering, here are the first three experiments to run. **Experiment one: external API timeout.** Pick your most critical external dependency. Add a 15-second delay to every response from that service. Watch what happens. Does the system degrade gracefully or cascade into a full outage? This experiment reveals whether you have [circuit breakers](/blog/circuit-breakers-external-apis), timeouts, and fallback behavior. **Experiment two: database connection exhaustion.** Reduce your database connection pool to 2 connections. Send normal traffic. Watch what happens when every connection is busy. This reveals whether your application queues gracefully, returns errors quickly, or hangs indefinitely. **Experiment three: worker crash during job processing.** Kill a worker process while it's processing a critical job (like a booking confirmation). Watch what happens. Does the job get retried? Does it retry correctly (idempotently)? Or does the job disappear and the traveler never gets their confirmation? These three experiments cover the most common failure categories: external dependency failure, internal resource exhaustion, and job processing failure. They'll reveal whether your system's resilience is real or theoretical. And they'll almost certainly surface at least one bug you didn't know about. --- Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. [Plan your next trip](https://app.nowah.xyz).