Our Deployment Pipeline: From Commit to Production

A one-line code change in the booking flow can ruin a thousand vacations. I'm not being dramatic. If a payment processing change introduces a bug that causes 3D Secure challenges to fail for a specific card issuer, every traveler using that issuer's card gets a failed booking. They retry. It fails again. They call their bank. They try a different platform. We've lost them.

Our deployment pipeline exists to prevent that. Not by moving slowly, but by validating thoroughly before anything reaches production. We deploy multiple times per day. Each deploy goes through the same gauntlet of checks.

Containerized builds

Every service builds into a container image. The same artifact that runs in testing runs in staging runs in production. There's no "it works on my machine" because we don't ship machines. We ship containers.

The container build is deterministic. Same code, same dependencies, same image. If the tests pass against a specific container image, that exact image is what gets deployed. We don't rebuild between environments.

The base images are standardized across services. Node.js runtime, common system dependencies, security patches. When a base image gets a security update, all services rebuild and redeploy on the updated base.

Testing gates

The pipeline has multiple testing gates, and each one must pass before the artifact moves to the next stage.

Unit tests run first. They're fast (seconds) and catch basic logic errors, type mismatches, and regression bugs. If unit tests fail, the pipeline stops immediately. No point running slower tests against broken code.

Integration tests run next. These test service interactions: API endpoints with real HTTP requests, database operations with real queries, and queue operations with real job processing. Our test configuration uses generous timeouts (60 seconds per test) and retries (2 per test) because some tests hit external APIs that are occasionally slow.

AI evaluation datasets are the most unusual gate in our pipeline. Before deploying a change to the AI agent, we run evaluation suites that test the agent against curated datasets. Flight-specific evaluations verify that the agent correctly handles booking requests, search queries, and edge cases. Safety evaluations verify that the agent resists prompt injection, doesn't leak system information, and stays within its authorized scope.

If any evaluation metric drops below its threshold, the deploy is blocked. This catches regressions in agent quality that unit tests can't detect, like a prompt change that accidentally makes the agent less helpful or a tool modification that breaks a specific query pattern.

Staging validation

After automated tests pass, the artifact deploys to staging. Our staging environment mirrors production infrastructure: same container orchestration, same database engine, same cache layer, same queue system. The only differences are credentials (staging uses sandbox payment APIs and test data providers) and scale (staging runs fewer instances).

In staging, we run a final round of smoke tests: key user flows tested end-to-end against the staging deployment. Can a user log in? Can they send a chat message? Can they search for flights? Can they initiate a booking?

We also check database migration compatibility. The migration runs in staging against a schema that matches production. If the migration fails or takes too long, we stop before it touches production.

Feature flags for the dangerous path

Any change that touches the booking or payment path goes behind a feature flag. No exceptions.

The feature flag system lets us deploy code to production without activating it. The new booking logic sits in production, behind a flag, with the old logic still active. We activate the flag gradually: 1% of traffic, then 10%, then 50%, then 100%. At each step, we monitor booking success rates, payment completion rates, and error rates.

If anything degrades, we kill the flag. Instant rollback. No deployment needed. No container swap. Just a configuration change that takes effect in seconds.

This approach means we can deploy frequently without fear. The dangerous code changes get extra protection. The safe changes (UI tweaks, copy changes, new non-critical features) deploy normally without flags.

Rollback procedures

Despite all the gates, sometimes a bad change reaches production. Maybe the bug only manifests under specific conditions that staging didn't cover. Maybe an external API changed behavior simultaneously. Things happen.

Our rollback procedure is fast and well-practiced:

1. Alert fires indicating a degradation (error rate, latency, booking failure rate).
2. On-call engineer confirms the issue and correlates with recent deploys.
3. If the deploy is the suspected cause, trigger rollback. The previous container image (the one that was running before the deploy) is always available. Rolling back is deploying the previous image, not reverting code.
4. Verify that the rollback resolved the issue.
5. Investigate the root cause, fix it, and re-deploy through the full pipeline.

The rollback can happen in minutes. The investigation might take hours. That's fine. Get production healthy first, then figure out what went wrong.

Building your own pipeline

If you're building a deployment pipeline for a high-stakes application:

Test at multiple levels. Unit tests catch code bugs. Integration tests catch interaction bugs. Evaluation datasets catch quality regressions. No single test level catches everything.

Make staging mirror production. If staging uses a different database engine or a different version of a dependency, bugs will slip through. Match the infrastructure as closely as possible.

Use feature flags for anything that touches money. The cost of maintaining a flag system is negligible compared to the cost of a broken payment flow in production.

Keep the previous artifact ready to deploy. Rollback speed depends on having the previous good state immediately available. Don't overwrite your previous container image when you deploy the new one.

Practice rollbacks. If the first time your team rolls back is during a real incident, it won't go smoothly. Practice in staging. Make sure everyone on the on-call rotation has done a rollback at least once.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.