--- title: "Documentation as Product: Why We Test Our Code Examples in CI" description: "Dead documentation is worse than no documentation. We extract every code example and run it in CI. If the example fails, the build fails. Here is our doc-testing framework." canonical: https://nowah.xyz/blog/documentation-as-product-test-code-examples lastModified: "2026-08-07T08:11:05.554Z" --- # Documentation as Product: Why We Test Our Code Examples in CI Dead documentation is worse than no documentation. We extract every code example and run it in CI. If the example fails, the build fails. Here is our doc-testing framework. We discovered a code example in our docs that had not worked for six months. Not because someone reported it. Because we audited our examples and found it. The endpoint had changed, a required parameter had been added, and the example still showed the old version. During those six months, we estimate around 50 developers tried that example. Most of them probably thought they were doing something wrong and moved on. A few filed support tickets. One enterprising developer found the fix in our changelog and updated their code. The other 49 wasted time on a problem that was not theirs to solve. Dead documentation is worse than no documentation. When there is no documentation, developers know they need to figure things out. When there is documentation that looks authoritative but does not work, developers waste time trusting something that has silently rotted. ## The doc-testing framework ![Illustration for this section](https://pics.nowah.xyz/website-media/developer-experience-046-img-1-extraction.webp) Our solution is straightforward: every code example in our documentation is extracted, wrapped in a test harness, and executed in CI. If the example fails, the build fails. Nobody can ship a docs change or an API change that breaks an example without the CI pipeline catching it. The extraction step parses our Markdown documentation files and pulls out every code block tagged with a supported language (a single typed language across the stack, Python, Go, Ruby). Each extracted block gets wrapped in a minimal test harness that provides the necessary setup: an SDK client configured with sandbox credentials, [error handling](/blog/error-handling-recovery) to catch failures, and assertions that the response structure matches expectations. The harness is intentionally minimal. We do not want to test our SDK -- that has its own test suite. We want to test that the example a developer copies from our docs actually works when they paste it and run it. ## CI integration: testing on API changes, not just doc changes The subtle but important decision was when to run these tests. Running them only when documentation changes catches doc errors. Running them on every API change catches API drift. We run doc tests on every commit that touches API routes, middleware, or response schemas. If an engineer adds a required parameter to the flight search endpoint, the doc test for the flight search example fails immediately. The engineer who made the API change sees the failure and updates the example. This is better than the alternative, where the API change ships, the docs team gets around to updating examples a week later (maybe), and developers in between get broken examples. The [feedback loop](/blog/ai-feedback-loop) is tight: change the API, see the doc test fail, fix the example, ship both together. The alerts for doc test failures go to the engineering team, not the documentation team. The person who changed the API is the person best positioned to update the example, because they understand what changed and how the new version should be called. ## Freshness guarantees ![Supporting diagram](https://pics.nowah.xyz/website-media/developer-experience-046-img-2-freshness.webp) Every code example on our documentation site shows a "last verified" timestamp. This is not a manual annotation. It is generated from CI data -- the last time that specific example was extracted, executed, and passed. A developer looking at an example that says "Last verified: 2 hours ago" has strong confidence that it works. An example that says "Last verified: 3 months ago" signals potential staleness, though in practice our CI runs frequently enough that this rarely happens. The freshness badge does two things. It builds trust with developers, who have been burned by stale docs elsewhere. And it creates internal pressure to keep examples current, because a "last verified: 6 months ago" badge on your docs page is embarrassing. ## The cost-benefit of testable examples Maintaining testable examples takes real effort. Every example needs to be self-contained. Every example needs setup and teardown. When the API changes, examples need updating. When a new endpoint ships, new examples need writing and testing. Here is why it pays back at least 10x. Documentation satisfaction across the industry averages about 3.2 out of 5. When developers can trust that every example works, satisfaction jumps above 4.5 out of 5. That gap is enormous in terms of developer retention and word-of-mouth. Comprehensive, tested documentation reduces repeat support queries for the same issue by over 70%. Each [support ticket](/blog/every-support-ticket-dx-bug) costs engineering time to read, diagnose, and respond. Preventing those tickets through reliable examples frees that time for building features. Developers who complete a successful API call in their first session are roughly 4x more likely to reach production integration. Working examples directly drive that first successful call. Broken examples directly prevent it. The math is clear. The cost of maintaining testable examples is a fraction of the cost of the support tickets, developer churn, and reputation damage that broken examples cause. ## Setting this up for your own API If you want to add doc testing to your API, here is a reasonable starting point. First, standardize your documentation code blocks. Tag them with language identifiers and a test identifier so the extraction tool knows which blocks to test and which to skip (some blocks are intentionally illustrative and not meant to run). Second, build a simple extraction script that finds code blocks, wraps them in test harnesses, and writes them to a test directory. The harness provides SDK initialization, sandbox configuration, and a way to assert that the example did not throw an error. Third, add the test run to your CI pipeline. Run it on every commit that changes API code, not just documentation. This is the step that catches drift. Fourth, generate the "last verified" timestamp from CI results and inject it into the rendered documentation. The whole setup takes less than a day for a small API. The ongoing maintenance is minimal -- mostly updating examples when the API changes, which your CI now forces you to do immediately rather than letting it slide. We treat our documentation as a product with the same quality standards as our API. Tests, CI, freshness tracking, and failure alerts. It sounds like overkill until you consider the alternative: fifty developers trying a broken example over six months, slowly eroding trust in your platform. That is the cost we are unwilling to pay. --- Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. [Plan your next trip](https://app.nowah.xyz).