Publish the source next to the render
This is the strongest practice page available, because it is a mechanism rather than an intention. It is also the one this site implements on itself — this page has a markdown twin, generated in CI, and the build fails if it is stale.
The markdown twin at every URL
"every page is available as markdown at the same path with the extension swapped, and links inside the markdown point at markdown, so a traversing agent never has to parse HTML."
The second clause is the one that makes it work, and it is the one usually left out. Publishing a markdown version of a page is common. Publishing a markdown version whose internal links also point at markdown means an agent that arrives at one .md never has to return to the HTML surface — it can traverse the entire site without a parser, and without ever needing to guess that a twin exists for the next page.
Why it is load-bearing rather than decorative
An agent-access report run against this estate in August 2026 found the failure precisely, and it is not the failure people expect:
"many agents can only fetch URLs that a search engine has already returned to them… A link listed inside a fetched document did not count as having been seen. … It can read the map and cannot walk it."
"The markdown is exemplary and the discovery layer is what stops agents using it."
And the consequence, which should worry anyone publishing technical material:
"the package registry page, which is a summary written for a different purpose, becomes the authoritative source by default because it is the only one that ranks."
That is the real failure mode. Not that your documentation is bad — that a worse summary of it becomes canonical because it is the only version an agent can reach. Publishing excellent markdown that nothing can discover is a solved problem that stays unsolved.
The mitigation ladder, and what this site ships
| Rung | What it does | Status here |
|---|---|---|
| Get indexed | The real fix, and the only one that addresses the root cause: agents that can only fetch what a search engine returned need the search engine to have returned it. | In progress — the site is new. Not in our gift on any particular timescale, and on the build order rather than quietly hoped for. |
Make llms.txt self-sufficient | The cheap fix. If a link inside a document does not count as seen, then a file that is only links is the failure mode restated. It has to carry the substance. | Shipped. llms.txt states the thesis, the central claim and its mechanism in the file itself. |
Ship llms-full.txt | Removes link-following entirely. Every page, one file, one fetch. | Shipped. llms-full.txt, generated from the pages themselves. |
| The markdown twin at every URL | Keeps a traversing agent on the markdown surface once it arrives anywhere. | Shipped, and enforced. validate.js fails the build if any page lacks a twin, and CI regenerates and fails if a committed twin is stale. |
admin/build/gen_markdown.py converts each hand-written page, rewrites internal .html links to .md, and concatenates the result into llms-full.txt. It is about 400 lines with no dependencies, so it runs in CI with nothing installed. The build tooling is documented → — which is itself the practice this page is about.How the original was built — the part that is actually hard
"we added native support to the library for servingllms.txtand.mdfiles… The way it was done is important: it was done using edge functions. These are static files, the browser will not run any JavaScript on them, so we couldn't render encrypted data. What we did was add a little bit of JavaScript on the Lambda@Edge function that does the rendering of the file."
That detail is worth preserving because it names the genuine obstacle. The whole point of a markdown twin is that it is served as plain text to a client that will not execute anything — which is exactly what makes it useful to an agent, and exactly what rules out any client-side rendering strategy. If the content needs any transformation at all, the transformation has to happen before the bytes leave the server. An edge function is the smallest place that can be true.
This site takes the simpler road available to it: the twins are generated at build time and committed, so GitHub Pages serves static files and there is no runtime component at all. Same contract, no edge function — and CI is what keeps the two surfaces from diverging.
The ambition beyond markdown
for every page on the website, serve not just the content but the semantic knowledge graph of that content (and the references), alongside the LLM-friendly text.
Not implemented here, and named as an ambition rather than a feature. graphs.sgit.ai owns the how; what this site would own is the argument for why a publisher should do it — and that argument is not yet written.
And the version nobody has shipped
publish "not just the article text but the full body of evidence, the semantic graph, the source materials, the multilingual versions, the agentic workflow provenance, and every prompt and decision that went into the finished piece."
That is a genuinely radical openness claim and it is barely stated anywhere public. It is worth taking seriously because of what it would prove: in a world where a great deal of published text is machine-assisted, provenance is the only thing that distinguishes a considered document from a generated one — and provenance is checkable if you publish it and unfalsifiable if you do not.
This site ships part of it and should be precise about which part. The source materials are published verbatim — the briefs this site was built from are in the repository, in full, as the source of truth for anything the pages summarise. The evidence trail is partial: every number carries its source, date and caveat, and the unverifiable ones are listed as unpublished. The workflow provenance and the prompts are not published, and claiming otherwise would be exactly the kind of thing this page argues against.
One place the openness stops, and why
The practice is not openness for its own sake, and the boundary is instructive. A public vault's read key is published by definition, so cataloguing read keys adds convenience and no exposure. A central list of write keys would be "a single artefact whose compromise hands over write access to everything" — and the estate has already had a key exposure, which it also documents.
The sharper rule comes from the irreversibility: "Nothing published can be corrected. If a frozen sample vault turns out to contain something it should not, there is no edit available." Hence the discipline — escrow the write key before publishing, not after. Publishing is a one-way door, and the time to think about it is before you walk through, which is a general principle rather than a vault-specific one.