--- title: How We Stream AI Responses in Real Time description: "A 5-second search that streams progress feels faster than a 3-second spinner. Here is the streaming architecture behind Nowah's real-time AI chat." canonical: https://nowah.xyz/blog/streaming-ai-responses-real-time-chat lastModified: "2026-08-07T03:49:57.169Z" --- # How We Stream AI Responses in Real Time A 5-second search that streams progress feels faster than a 3-second spinner. Here is the streaming architecture behind Nowah's real-time AI chat. Here's a fact [that changed](/blog/launch-that-changed-our-roadmap) how we build our product: a five-second operation that streams progress feels faster to users than a three-second operation behind a loading spinner. This isn't intuitive. Three seconds IS faster than five seconds. But human perception of speed isn't about wall-clock time. It's about uncertainty. When you see a spinner, you don't know if the system is working, broken, or stuck. You don't know if it'll take two more seconds or twenty. When you see tokens appearing word by word with status updates like "Searching 4 airlines...", you know exactly what's happening. The wait feels productive instead of empty. Your attention is engaged by the progressive content rather than anxiously monitoring a spinning icon. This insight drove the entire architecture of our real-time AI chat. Everything streams. Text responses arrive token by token. Search progress is reported in real time. Booking status updates flow as they happen. The user is never staring at a blank screen wondering if something went wrong. Here's how we built it. ## Why batch request-response is dead for conversational AI ![Illustration for this section](https://pics.nowah.xyz/website-media/engineering-009-img-1.webp) Traditional web applications follow a simple pattern: client sends request, server processes, server sends complete response. This works fine when processing takes milliseconds. It falls apart when processing takes seconds, which is the reality for any AI that searches live data. When a user asks "Find me flights to Barcelona next week," the agent needs to: A typical turn goes through several stages: understand the request, load relevant traveler context, decide which tools to call, wait on provider APIs, rank options, and write a natural-language answer. Total wall-clock time is usually a few seconds, dominated by provider APIs. Total wall-clock time is usually a few seconds, depending on provider API latency. In a batch model, the user sees nothing for three to six seconds, then gets a wall of text. That feels slow, unresponsive, and a little unsettling when money is involved. Did it freeze? Should I tap again? Did my request actually go through? In a streaming model, the user sees "Let me search for flights to Barcelona..." almost immediately. Then "Searching across airlines..." moments later as the provider APIs are called. Then "Found 180 options, selecting the best matches for you..." while search completes as ranking completes. Then the options render one at a time with their scoring explanations. Same total time, radically different experience. Human conversation has natural timing. Pauses longer than about two seconds feel awkward. Exceed four seconds and the experience feels broken, like talking to someone who's ignoring you. Streaming keeps us within conversational rhythm even when the actual computation takes longer. ## Server streaming over bidirectional sockets: a deliberate choice We chose unidirectional server streaming (server streaming) for our streaming protocol. This was a deliberate choice over bidirectional sockets, and the reasoning is worth explaining because it comes up frequently in discussions about real-time AI architecture. Bidirectional sockets allow both sides to send messages. The client and server can both send messages at any time. Server streaming is unidirectional: the server streams to the client, and the client sends requests via regular HTTP. For our use case, unidirectional is enough. The user sends a message (regular HTTP POST), and the server streams back the response. We don't need the server to push unsolicited messages during a response stream. When we do need bidirectional push (live notifications, flight status updates between conversations), we use a separate a bidirectional socket connection for that purpose. Server streaming has several advantages over bidirectional sockets for AI streaming specifically. **HTTP-native.** server streaming works over standard HTTP connections. No protocol upgrade, no connection handshake complexity. This means it works through corporate proxies, CDNs, and load balancers without special configuration. bidirectional sockets often require specific infrastructure support that adds operational complexity. **Auto-reconnection.** The server streaming specification includes built\-in reconnection\. If the connection drops \(common on mobile networks\), the browser automatically reconnects and can resume from the last received event using the \`Last\-Event\-ID\` header\. We don't have to implement reconnection logic from scratch\. This is a significant advantage for mobile users who frequently switch between WiFi and cellular, pass through tunnels, or have spotty coverage\. **Simpler server implementation.** An server streaming endpoint is just an HTTP response with \`Content\-Type: text/event\-stream\` and a long\-lived connection\. No a bidirectional socket library, no connection [state management](/blog/state-management-ai-conversations), no ping-pong heartbeats to keep connections alive. The simplicity reduces our bug surface. Fewer moving parts means fewer things that can break. **Text-optimized.** server streaming transmits text, which is exactly what AI token streams are. We don't need the binary frame support that bidirectional sockets provide. Our payloads are JSON events, and server streaming handles them efficiently without the framing overhead of a bidirectional socket binary mode. The trade-off is that streaming connections are one-per-tab in older browsers and have a default limit of six connections per domain in some environments. Neither of these is a practical issue for us. Each conversation has one active stream at a time, and we're nowhere near the connection limit. On HTTP/2 (which all modern browsers use), the connection limit doesn't apply because server streaming streams are multiplexed over a single TCP connection. ## Backend streaming architecture: orchestrating the event stream ![Supporting diagram](https://pics.nowah.xyz/website-media/engineering-009-img-2.webp) On the backend, the streaming pipeline orchestrates multiple asynchronous operations into a single coherent event stream. This orchestration is where most of the complexity lives. When a user message arrives, the server opens an streaming connection and begins emitting events. The event types in our stream include: **Status events** report what the agent is doing. "Thinking...", "Searching flights...", "Ranking results...", "Checking hotel availability..." These give the user real-time feedback on the agent's progress through the processing pipeline. They're emitted at real pipeline checkpoints, not on a timer. When you see "Searching flights...", the API call is actually dispatching at that moment. **Token events** contain individual text tokens as the AI generates its response. The frontend renders these incrementally to create the word-by-word typing effect that makes the agent feel responsive and human-like. Tokens arrive as fast as the model generates them, which is typically faster than a person reads. **Tool result events** deliver structured data like flight cards and hotel options. These are rich JSON payloads that the frontend renders as interactive UI components rather than plain text. A flight card includes price, airline, times, layover info, and a "Book this" action. These events are larger than token events and represent meaningful data milestones in the stream. **Thinking events** communicate when the agent is reasoning about the next action but not yet generating visible output. These create a brief "thinking..." indicator that signals activity without the uncertainty of a blank screen. **Complete events** signal the end of the response and include metadata like message IDs, session state, and any offers the user can act on. The frontend uses this event to finalize the UI state: disable the streaming indicator, enable the input for the next message, and cache the response. **Error events** report failures with enough context for the frontend to display a useful message and offer recovery options. An error mid-stream is handled differently than an error at the start: the frontend preserves any content already rendered and appends the error context. The backend coordinates between the AI model (which generates tokens and tool calls), the tool execution layer (which calls [external APIs](/blog/circuit-breakers-external-apis)), and the event stream (which delivers everything to the client). These operations interleave: the model might generate some introductory text ("Let me search for flights to Barcelona..."), call a tool (flight search), wait for the result, generate more text using the result ("I found some great options..."), and emit tool result events with the flight cards. All of this gets serialized into the server streaming stream in real time. ## Frontend incremental rendering: harder than it looks Streaming changes [frontend architecture](/blog/chat-first-ui-frontend-architecture) in ways that aren't obvious until you build it. In a batch model, you fetch data and render it. State is simple: either you have the data or you don't. In a streaming model, you have partial data that grows over time. A text response is being built token by token. Flight results might arrive one at a time. The UI needs to update continuously without layout jank, scroll disruption, or flickering. We handle this with a streaming state machine on the frontend. The machine tracks: - Whether a response is in progress - The accumulated text tokens so far - Any structured results (flight cards, hotel cards) that have arrived - The current status message from the latest status event - Whether the stream is active, paused (waiting for a tool result), or complete The UI renders from this state. New tokens get appended to the text display. Rich cards render as their data arrives, sliding into the conversation flow. The scroll position follows the new content unless the user has manually scrolled up (in which case we show a "new content below" indicator instead of auto-scrolling, so we don't yank them away from what they're reading). Sub-second UI rendering for streaming responses required optimization. Rendering on every single token is too expensive; the DOM can't keep up at inference speed, especially on lower-powered mobile devices. We batch token renders on animation frames, accumulating a few tokens between frames and flushing them together. This gives smooth visual output without jank. The user sees a fluid typing effect. Under the hood, we're rendering batches of 3-5 tokens at 60fps rather than individual tokens at model speed. Layout stability is another challenge. When a flight card appears in the middle of a streaming text response, the card takes up vertical space that shifts content below it. Without careful handling, this causes a visible "jump" that's disorienting. We pre-allocate space for incoming cards based on status events ("about to show flight results" reserves space for three cards), and animate the cards into the reserved space smoothly. ## Streaming tool results: showing the work The most valuable part of streaming isn't the text tokens. It's the search progress and tool results. When the agent searches for flights, the user sees "Searching across airlines..." followed by "Found 180 options..." followed by "Ranking by your preferences..." This is live feedback from the actual tool execution, not canned loading messages. We emit these status events at real pipeline checkpoints. The "Searching..." event fires when the API call dispatches. The "Found N options..." event fires when results return. The "Ranking..." event fires when the ranking pipeline starts processing. This transparency builds trust. The user can see that the agent is actually searching real data, not generating plausible-sounding fiction. When the [three options](/blog/why-three-options-not-three-hundred) finally render as interactive cards, the user already knows they were selected from a large set based on real ranking criteria. They can trust the results because they watched the work happen. We also stream partial progress for long-running operations. If the agent is searching multiple providers in parallel, we show which providers have responded and which are still pending. "Found flights from 3 of 4 providers. Still checking one more..." This is honest about what's happening without making the user wait for the slowest provider. ## Error handling in streaming connections Streaming connections can fail in ways that batch connections can't. The connection might drop mid-response. The backend might crash while the stream is active. A tool call might fail after the agent has already started its response. We handle each of these carefully. **Connection drops.** server streaming auto-reconnects, but we need to handle the gap. When a connection drops and reconnects, the client sends the ID of the last event it received. The server can resume from that point if the response is still generating, or send the complete response if it finished during the gap. The user sees a brief "Reconnecting..." indicator and then the stream continues seamlessly. On mobile, this happens more often than you'd expect due to network transitions. **Backend failures.** If the backend crashes mid-stream, the streaming connection dies. The client detects this (no events for longer than our timeout threshold) and shows a recovery UI: "Connection lost. Tap to retry." The retry sends the original message again, and the server detects the retry via the idempotency key and either resumes or restarts. **Tool failures mid-stream.** The agent has already sent "Let me search for flights..." and then the flight search API returns an error. The agent needs to continue the stream with an error explanation: "I wasn't able to search that route right now. This might be a temporary issue with our flight data. Want me to try again, or should we look at different dates?" The stream stays alive; only the content changes from results to error recovery. The key principle: the stream should never just stop without explanation. If something goes wrong, the user should know what happened and what to do about it. Silence is the worst possible failure mode in a conversational interface. ## Where every millisecond goes: the latency budget Our latency budget for a typical search interaction: - Authentication and request parsing: milliseconds - Context retrieval: low tens to low hundreds of milliseconds - First token from AI inference (model starts generating): ~300-500ms - Status update streamed to user ("Searching..."): a short delay cumulative from message send - Tool call dispatch to travel provider APIs: a short delay after model decision - External API response: 1-4 seconds (provider-dependent, out of our control) - Result processing and ranking: ~200-300ms - AI generates response with results: begins immediately after ranking - Streamed response with flight cards: renders as generated The first token must arrive fast. That's the metric that determines perceived responsiveness. Everything after the first token is progressive enhancement that keeps the user engaged. Users are patient once they see the agent is working. They're not patient when staring at nothing. Our agent responds with streamed content within a couple of seconds. That number represents optimization at every stage: context retrieval is parallelized with model inference warmup (we start fetching memory before the model produces its first token), status events fire at pipeline checkpoints rather than waiting for completion, and result rendering begins as soon as the first option is ranked (not after all three are ranked). The metric we obsess over is time-to-first-meaningful-content (TTFMC), not time-to-complete-response. TTFMC for a flight search is about 500ms: the user sees the agent acknowledge their request and begin working. Time-to-complete might be 3-5 seconds. But the perceived experience is dominated by TTFMC, not total time. Streaming isn't just a UX pattern. It's an architectural choice that cascades through the entire system. Backend, frontend, [error handling](/blog/error-handling-conversational-systems), state management, reconnection logic, testing. All of it changes when you commit to progressive delivery. But the user experience gain is worth every bit of complexity. Because a five-second search that shows its work will always feel faster than a three-second search that doesn't. --- Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. [Plan your next trip](https://app.nowah.xyz).