Secrets Management for a Multi-Service AI Platform

Our backend has API keys for AI model providers, travel data providers, payment processors, authentication services, email delivery, push notifications, document storage, memory systems, and analytics platforms. Each of these has separate keys for development, staging, and production environments. Some have both public and secret keys. Some have webhook signing secrets.

When I counted, we manage over 40 distinct secrets across three environments. That is over 120 secret values. One leaked key could mean unauthorized bookings, stolen payment data, or a massive AI inference bill. Secrets management is not glamorous infrastructure work, but it is absolutely load-bearing.

The secrets sprawl problem

Traditional web apps might have a database connection string, an email API key, and a payment processor key. Maybe 5-10 secrets total. AI platforms have dramatically more because the product integrates with more external services.

An AI travel platform connects to at least: an AI model provider (inference API key), a travel data provider (search and booking keys), a payment processor (secret key, webhook secret, publishable key), an authentication service (secret key), an email delivery service (API key), a push notification service (credentials), a document storage service (access keys), a memory system (API key), and an analytics platform (API key).

That is 10+ services before you count internal secrets like signed session tokens signing keys, database passwords, and inter-service authentication tokens. And each service multiplied by three environments.

The sprawl itself is a risk. More secrets means more potential leak vectors, more rotation overhead, and more chances for a developer to accidentally commit a production key to version control.

Centralized management for distributed services

We manage secrets through environment-specific configuration files that are injected at deployment time. No secrets are stored in code repositories. No secrets are hardcoded. No secrets are committed, even encrypted, to version control.

Each environment has a secrets configuration that is separate from the application code. The deployment pipeline pulls secrets from the secret store and injects them as environment variables when the service starts. The application reads them from environment variables using a configuration module that validates all required secrets are present at startup. If a secret is missing, the service refuses to start and logs which secret is absent.

This validation step catches deployment errors before they become production incidents. A misconfigured deployment that forgets to inject the payment processor key will fail at startup rather than failing silently and then crashing when a user tries to pay.

Environment-specific configuration

Development, staging, and production use different keys for the same services. Development uses sandbox or test credentials that cannot make real bookings or charge real cards. Staging uses production-like credentials in sandboxed environments where available. Production uses real keys with real consequences.

The boundary between these environments is enforced at the infrastructure level, not just by convention. A development instance physically cannot access production secrets because they are stored in a different secret namespace. This prevents the classic failure mode where a development process accidentally runs against production data because someone copied the wrong .env file.

We also use different AI model configurations per environment. Development uses a cheaper, faster model. Staging uses the production model to catch quality regressions. Production uses the full configuration. The model API key is the same across staging and production (same provider), but the model selection and parameters differ.

Key rotation

API keys should be rotated regularly. We rotate keys on a schedule and immediately when a potential exposure is detected.

Rotation for most services follows this process: generate a new key in the provider's dashboard, add the new key to our secret store alongside the old key, deploy the new key to services, verify the new key works in production, remove the old key from the provider's dashboard.

The critical detail is that both keys are valid simultaneously during the transition. This enables zero-downtime rotation. Services using the old key continue working while we roll out the new key. Once all services are on the new key, we revoke the old one.

For payment processor keys, rotation is more sensitive. We schedule these during low-traffic windows and monitor payment success rates closely during and after the rotation.

Least-privilege access

Not every part of our system needs every secret. The AI agent needs the model provider API key and the travel data provider key, but it does not need the payment processor key. The payment processing worker needs the payment key but not the model provider key.

We implement least-privilege by scoping which secrets each service receives. The backend API gets the full set of secrets because it orchestrates all operations. Background workers get only the secrets relevant to their specific jobs. The frontend gets only publishable keys that are safe for client-side exposure.

Least-privilege limits the blast radius of a compromise. If an attacker gains access to a background worker, they get the email delivery key but not the payment key or the AI model key. The damage is contained to the compromised service's scope.

For the AI agent specifically, we are careful about which capabilities it can access. The agent can search and present options, but the actual booking and payment execution flows through a separate service with its own authentication. The agent cannot directly charge a credit card even if it wanted to.

This is a good security posture for AI systems in general. The AI agent should have the minimum permissions needed to do its job, just like any other service. The fact that it is "intelligent" does not mean it should be omnipotent.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.