How We Stream AI Responses for a Fast First Token

A traveler types "Find me a flight to Tokyo next Thursday" and hits send. At that exact moment, a clock starts in their head. Not a literal clock, but a psychological one. If nothing happens for two seconds, they wonder if the app is broken. Five seconds and they're reaching for the back button. Ten seconds? They're gone.

We learned this the hard way. Our first prototype processed the entire AI response server-side, then shipped it to the client as one big payload. The experience was terrible. You'd ask a question and stare at a loading spinner for eight to twelve seconds while the AI searched flights, compared fares, checked seat availability, and composed a response. By the time the answer appeared, you'd forgotten what you asked.

Streaming changed everything. Instead of waiting for the full response, we push tokens to the client as they generate. The traveler sees the AI start to respond within a few hundred milliseconds. They watch it think. They see partial results materialize. The perceived wait goes from "is this thing broken" to "oh, it's working on it." Same total processing time, completely different experience.

Why we picked server streaming over bidirectional sockets for AI responses

This was one of our earliest architecture decisions, and it turned out to be one of the best.

unidirectional server streaming are a simple, HTTP-native protocol for pushing data from server to client. One direction. The server talks, the client listens. That is exactly what happens during an AI response. The client sends a message, then sits back and receives a stream of tokens, tool results, and status updates. There is no reason for the client to send data back mid-stream.

Bidirectional sockets allow both sides to send messages. They let both sides talk at any time. That sounds more capable, and it is, but that capability comes with real operational costs. bidirectional sockets require sticky sessions or connection-aware load balancing. They don't play well with many CDNs and reverse proxies without extra configuration. They maintain a persistent connection that consumes server resources even when idle.

Server streaming runs over plain HTTP. It works through every CDN and reverse proxy we've tested without special configuration. Load balancers distribute requests normally. When the stream ends, the connection closes cleanly. No persistent state to manage on the server.

We do use bidirectional sockets elsewhere in the platform for features that genuinely need bidirectional communication, like presence indicators and typing status. But for AI response streaming, server streaming is the right tool. Simpler to operate, easier to debug, and just as fast.

Structured event types

A naive server streaming implementation sends a stream of text tokens. That works for a basic chatbot, but travel queries produce more than text. A single "find me a flight" request triggers tool calls to search live inventory, returns structured flight data, and includes booking actions the traveler can take. We needed the stream to carry different types of data, each rendered differently on the client.

We settled on four event types:

Status updates tell the client what the AI agent is doing right now. "Searching flights to Tokyo..." or "Comparing 12 fare options..." These render as subtle status indicators in the chat UI. They're important because they keep the traveler informed during the three to eight tool calls a typical query triggers.

Content chunks are the actual text tokens of the AI response. These stream in token by token and render as the message being typed out in real time.

Tool results carry structured data, like flight offers with prices, times, airlines, and seat availability. The client receives these as typed JSON payloads and renders them as interactive cards inline in the conversation. A flight card appears mid-stream, fully interactive, before the AI has finished composing its text explanation.

Completion signals mark the end of the stream and carry the final session state. This is where the client knows the response is done and can enable the input field for the next message.

Each event type has a defined schema. The client knows exactly what shape of data to expect for each type, which means rendering logic is straightforward and errors are easy to catch.

Backpressure on slow clients

Here is a scenario that bit us early on. A traveler at an airport is on congested Wi-Fi. The AI agent generates tokens fast, but the network can't deliver them at that speed. Without backpressure handling, the server buffers tokens in memory for that connection. Multiply that by a few hundred slow connections, and you're looking at memory exhaustion on the server.

Our backpressure strategy has three parts.

First, we monitor the write buffer for each streaming connection. When the buffer exceeds a threshold, we know the client isn't consuming data fast enough.

Second, we slow down token delivery for that specific connection. The AI continues processing at full speed, but we batch tokens into larger chunks before writing them to the slow connection. Instead of sending individual tokens, we buffer a few hundred milliseconds worth and send them as a single write. This reduces the number of write operations without losing data.

Third, if the buffer grows beyond a hard limit, we close the connection gracefully and let the client reconnect. The client receives a partial response and can request completion from where it left off.

This approach means a slow client on airport Wi-Fi degrades their own experience slightly (slightly chunkier token delivery) without affecting any other connection on the server. That isolation is critical at scale.

Stream reconnection and partial recovery

Mobile networks drop connections. It's not a question of if, it's a question of how often. Airport Wi-Fi, cellular handoffs between towers, elevator rides, tunnel transitions. Our production data shows that streams get interrupted 5 to 15 percent of the time under poor network conditions.

When the client detects a dropped connection, it needs to recover without losing the partial response already received. Our reconnection protocol works like this:

The client tracks which events it has received by sequence number. On reconnection, it includes the last received sequence in the reconnection request. The server can then determine whether the response is still being generated (resume from that point) or has completed (send the remaining events from a short-lived buffer).

The reconnection target is under two seconds. In practice, most reconnections happen within one second on reasonable networks. The traveler sees a brief interruption indicator, then the stream resumes.

For cases where reconnection fails entirely, the client falls back to loading the complete response via a standard REST call once the AI finishes processing. The response is always available as a complete message after the stream ends. Streaming is the preferred delivery mechanism, not the only one.

Measuring streaming performance

We track four metrics for our streaming pipeline:

Time to first token measures how long between the client sending a message and receiving the first content event. Our target is under 500 milliseconds. This is the single most important metric for perceived responsiveness. In practice, we usually hit 200 to 400 milliseconds depending on the complexity of the query and whether the AI needs to reason before generating output.

Tokens per second measures throughput during active streaming. This is mostly determined by the language model inference speed, but our pipeline needs to stay out of the way. Any overhead from our server streaming infrastructure should be negligible compared to model latency.

Stream completion rate is the percentage of streams that deliver the complete response without requiring reconnection or fallback. This is our primary reliability metric. A high completion rate means the streaming infrastructure is working. A declining rate means something changed in client conditions or server behavior.

Time to interactive measures when the first actionable content appears, specifically the first flight card or hotel option the traveler can interact with. This matters more than total response time because the traveler can start evaluating options before the AI finishes talking.

A checklist for sub-500ms first-token delivery

If you're building your own server streaming streaming pipeline, here is what we learned matters most.

Keep the connection setup fast. streaming connections are just HTTP responses with a specific content type. Don't add heavy middleware to the server streaming endpoint. Authentication should happen before the stream opens, not as part of the stream setup.

Start streaming before tool calls complete. The AI can begin its text response while tool results are still pending. Send the initial status update and first tokens immediately, then interleave tool results as they arrive.

Buffer writes, not tokens. Sending individual tokens as individual network writes is wasteful. Batch a few milliseconds worth of tokens into a single write. The perceived experience is identical, but you reduce system call overhead by an order of magnitude.

Monitor from the client side, not just the server side. The server might show that it wrote the first token at T+100ms, but if the client doesn't receive it until T+800ms due to network latency, the traveler's experience is 800ms, not 100ms. Client-side instrumentation is the only metric that tells you the truth.

Treat reconnection as a first-class feature, not an edge case. If your server streaming pipeline doesn't handle reconnection gracefully, you're building for ideal conditions that don't exist in the real world. Travelers use your app in airports, on trains, on international roaming. The network will drop. Plan for it.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.