A Shared In-Memory Layer: Caching, Queues, and Sessions

a shared in-memory data store is one of those technologies that keeps showing up in different parts of your architecture until you realize it has become load-bearing infrastructure. For us, a single an in-memory data store deployment powers four distinct subsystems: session caching, background job queues, travel data caching, and rate limiting. This post is about why we ended up here and how we manage the complexity of running a multi-purpose an in-memory data store system.

Why one tool for four jobs

The obvious question: why not use separate systems for each concern? A dedicated cache, a dedicated message queue, a dedicated session store?

The answer is operational simplicity. We are a small team. Every additional service in our infrastructure is another thing to monitor, upgrade, back up, and debug at 3 AM when it goes down. an in-memory data store is fast enough and capable enough to handle all four workloads, and running one an in-memory data store deployment well is easier than running four separate systems adequately.

This is a pragmatic choice, not a philosophical one. If any of these workloads outgrew an in-memory data store, we would split it out. So far, an in-memory data store handles the combined load without strain.

Session caching for conversation context

When the AI agent processes a user message, it needs context from the conversation history and user preferences. This context needs to arrive in milliseconds, not seconds. A 500ms delay in loading context means a 500ms delay before the agent even starts thinking about its response.

We cache active session data in an in-memory data store. When a user sends a message, the backend grabs their conversation context, preferences, and any in-progress booking state from an in-memory data store. If the cache is warm (which it is for active users), this takes under 5ms. If the cache is cold (user returning after a long break), we load from the database, cache it, and proceed. That cold-load path takes 50-100ms, which is acceptable because it only happens once per session.

Session data has a TTL of several hours. Active sessions refresh the TTL with each message, so a user in the middle of a conversation never hits a cache miss. Idle sessions expire naturally, freeing memory.

The data we cache per session includes the last 20 messages of conversation history (enough context for the agent to maintain coherence), user preferences, any active booking state, and tool results from recent searches. The total per-session footprint is typically 50-200KB, depending on how many search results are cached.

Job queue backbone

We run seven background worker queues for operations that should not block the main request path. Booking confirmation processing, email delivery, notification dispatch, analytics events, document generation, webhook processing, and scheduled maintenance tasks.

All of these run on a job queue system backed by an in-memory data store. Jobs are serialized into an in-memory data store lists. Workers consume from these lists, process the jobs, and report results. Failed jobs are retried with exponential backoff and eventually moved to a dead letter queue for manual inspection.

The booking confirmation queue is the most interesting from a reliability perspective. When a user completes payment, we need to confirm the booking with the travel provider, generate documents, send a confirmation email, and update the trip record. Each of these steps can fail independently, and partial failures need to be recoverable.

We implement this as a chain of jobs. Payment confirmed triggers booking confirmation. Booking confirmation triggers email and document generation. Each job is independently retryable. If the email fails but the booking succeeded, the user still has their booking and the email retries until it goes through.

an in-memory data store's durability guarantees matter here. We configure an in-memory data store with append-only file persistence so that queued jobs survive a an in-memory data store restart. A booking confirmation job that was enqueued but not yet processed will still be there after a restart.

Travel data caching

Flight and hotel data is volatile. A flight price from 10 minutes ago might be wrong. But a flight schedule from 10 minutes ago is probably still correct. And an airport's terminal map from 10 minutes ago is definitely still correct. Different data types need different cache strategies.

We use an in-memory data store for travel data caching with aggressive but type-appropriate TTLs:

Flight search results: 3-5 minute TTL. Prices change frequently, but within a few minutes, the same search usually returns the same results. This cache saves us from hammering the provider API when multiple users search similar routes.

Hotel availability: 10-15 minute TTL. Hotel prices change less frequently than flights, and availability is usually stable over short periods.

Reference data (airports, airlines, routes): 24-hour TTL. This data changes rarely. Caching it for a day is safe and eliminates thousands of redundant lookups.

Caching reduces our travel API costs by a large share. That is not a rounding error. At our scale, the difference between caching well and not caching at all is tens of thousands of dollars per month in API fees.

The tricky part is cache invalidation when data is known to be stale. If a user tries to book a cached flight option and the provider returns a "price changed" error, we invalidate that cache entry immediately so subsequent users see the updated price.

Sliding window rate limiting

Rate limiting is our fourth an in-memory data store use case. We use a shared rate-limit store to implement sliding window rate counters. Each request adds an entry to a sorted set keyed by user ID and route. We trim entries older than the window size and check the set cardinality against the limit.

This gives us precise, performant rate limiting. The sliding window avoids the burst problems of fixed-window counters. an in-memory data store's in-memory speed means the rate check adds negligible latency to each request (under 1ms).

We track rate limits across multiple dimensions: per-user, per-route, and per-provider. A user might have available chat capacity but be rate-limited on booking attempts. The provider rate limiter might throttle searches while letting other operations through.

High availability concerns

Running four subsystems on one an in-memory data store instance means an in-memory data store becoming unavailable affects everything simultaneously. This is the obvious risk of our approach.

We mitigate this with several strategies. an in-memory data store runs with replication so a replica can take over if the primary fails. We monitor an in-memory data store memory usage, connection count, and command latency continuously. We test failover scenarios regularly.

We also design each subsystem to degrade gracefully if an in-memory data store is temporarily unavailable. Session cache miss? Fall back to the database. Job queue unavailable? Hold jobs in application memory briefly and flush when an in-memory data store recovers. Rate limiter down? Apply conservative static limits until an in-memory data store returns. Travel cache miss? Go directly to the provider API.

These fallback paths are not as fast or efficient as the queue-backed paths, but they keep the product functional during a an in-memory data store hiccup. We have had exactly two an in-memory data store-related incidents in production. Both were resolved within minutes thanks to automatic failover, and neither resulted in user-visible errors thanks to the fallback paths.

an in-memory data store is infrastructure that you either get right or you feel constantly. We have invested in getting it right.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.