--- title: Building a Searchable API Log Viewer description: "Millions of API requests need fast search and filtering. Here is how we built a log viewer with sub-second search, full request/response bodies, and retention tiers." canonical: https://nowah.xyz/blog/building-searchable-api-log-viewer lastModified: "2026-08-07T08:10:01.298Z" --- # Building a Searchable API Log Viewer Millions of API requests need fast search and filtering. Here is how we built a log viewer with sub-second search, full request/response bodies, and retention tiers. A developer came to us with a problem. Their booking flow was failing intermittently -- about one in fifty attempts -- and they could not reproduce it locally. They needed to find the failing requests in their API logs, compare them to successful ones, and figure out what was different. They had made roughly 500,000 API calls that month. We needed to help them find the handful that failed, show the full request and response bodies, and do it fast enough that the debugging session felt productive, not painful. That was the moment our log viewer went from "nice to have" to "non-negotiable infrastructure." It took 200 milliseconds to find the failing requests. The developer identified the issue in ten minutes -- a race condition in their retry logic that was sending duplicate booking confirmations. ## What to index for fast search ![Illustration for this section](https://pics.nowah.xyz/website-media/developer-experience-042-img-1-filter-bar.webp) The performance of a log viewer comes down to indexing strategy. Index too little and searches are slow. Index too much and writes are slow and storage costs explode. We index six fields: request ID, API key, HTTP method, endpoint path, status code, and timestamp. These six fields cover the vast majority of searches developers actually perform. Request ID \(\`req\_\` prefix\) is the primary lookup key\. When a developer gets an error, the error response includes a request ID\. They paste it into the log viewer and immediately see the full request and response\. This takes under 50 milliseconds for any request within the retention window\. Status code indexing supports range queries. A developer can search for all 4xx errors, all 5xx errors, or a specific code like 429. This is one of the most common search patterns: "show me everything that failed." Endpoint path indexing lets developers filter to a specific API route\. "Show me all calls to \`/flights/search\` that returned 500" narrows millions of records to a handful\. Timestamp indexing with millisecond precision supports time range queries. Combined with other filters, this enables investigations like "show me all booking errors in the last two hours." We do not full-text index request or response bodies. The cost is prohibitive at scale, and the need is rare. For the occasional case where a developer needs to search within bodies, we support downloading filtered log sets and searching locally. ## Search performance: the tradeoffs Sub-second search across millions of records requires compromises. Here are the ones we made. We partition logs by API key and time window. Each partition covers one key's traffic for one day. This means most searches only scan one partition, keeping query times low even as total log volume grows. Searches across multiple days or multiple keys fan out to multiple partitions in parallel. Performance degrades linearly with the number of partitions, not with the total log volume. A search across 7 days is roughly 7x slower than a search across 1 day, but still sub-second for most keys. We pre-aggregate common queries. The count of requests by status code per hour is materialized, not computed on every query. This makes the overview metrics dashboard fast without requiring full log scans. The trade-off is write amplification. Every incoming request gets written to the raw log and to several aggregate tables. For our write volumes, this is acceptable. For an API handling millions of requests per second, you would need a different architecture. ## The filtering UX ![Supporting diagram](https://pics.nowah.xyz/website-media/developer-experience-042-img-2-log-entry.webp) Fast search means nothing if the developer cannot express what they are looking for. The log viewer's filter bar supports five filter types that compose together. **Status filter.** Chips for "2xx," "4xx," "5xx," or specific codes. Multiple selections are OR'd together. **Endpoint filter.** A dropdown of all endpoints the developer has called, sorted by frequency. Also supports pattern matching for developers who know what they want. **Method filter.** GET, POST, PUT, DELETE. Simple but useful when combined with other filters. **Time range.** A date-time picker with presets (last hour, last 24 hours, last 7 days) and custom range support. The picker defaults to the developer's local timezone but shows UTC alongside. **Search box.** Free\-text search across request IDs and error codes\. Supports partial matches \-\- typing "req\_abc" finds "req\_abc123" and "req\_abcdef\." All filters compose. Selecting "5xx" + "/flights/book" + "last 24 hours" finds server errors on the booking endpoint in the past day. The result count updates in real time as filters change, so the developer knows if their query is too broad or too narrow before the results load. ## Request and response body rendering When a developer expands a log entry, they see the full picture: HTTP method, URL, status code, latency, request headers, request body, response headers, and response body. Bodies render with syntax highlighting and JSON folding. A flight search response can be large -- dozens of offers with nested fare details, routing segments, and metadata. JSON folding lets the developer collapse sections they are not interested in and focus on the specific fields they are debugging. Sensitive data gets masked automatically\. Authorization headers show \`Bearer \***\` instead of the actual token\. API keys in request parameters show the prefix and a mask: \`nwh\_live\_**\*\`\. This masking is applied at render time, not at storage time, so support engineers who need the full data for escalation can access it through internal tools\. ## Retention tiers Log storage is not free, and retention expectations vary by team size and compliance requirements. We offer two tiers. Free tier retains logs for 30 days. For individual developers and small teams, this covers the debugging use case well. Most issues are investigated within hours or days of occurrence, not weeks later. Paid tier retains logs for 90 days. For teams with compliance requirements or complex integrations that need longer investigation windows, the extended retention provides the data they need. Both tiers support export. Developers can download filtered log sets as JSON or CSV for archival in their own systems. For teams that need indefinite retention, we recommend setting up a webhook that streams log events to their own storage system in real time. ## Scaling for high-volume keys When a single API key generates over 100,000 requests per day, the log viewer needs to handle the volume without degrading search performance. We address this with automatic sampling for overview metrics. The sparklines and counters on the overview page use sampled data when total volume exceeds a threshold. Detailed search still scans all records, but the overview stays fast. Pagination in search results uses cursor-based tokens, not offset/limit. When a search returns thousands of results, the developer pages through them without missing or duplicating entries even as new logs arrive. For the highest-volume integrations, we partition more aggressively -- by hour instead of by day -- to keep per-partition sizes manageable. This is transparent to the developer. The [search interface](/blog/no-search-bar-booking-interface) works identically regardless of how the data is partitioned underneath. The log viewer is not the flashiest part of the [developer dashboard](/blog/building-developer-dashboard-developers-use). But when something goes wrong at 2 AM and a developer needs to understand what happened, it is the feature that matters most. Fast search, clear display, and enough retention to cover the investigation window. That is the job, and we take it seriously. --- Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. [Plan your next trip](https://app.nowah.xyz).