LAYERED AGENTIC RETRIEVAL: A PUBLISHING ARCHITECTURE FOR AGENT-NATIVE WEB
SURFACES
==============================================================================
Author: Francesco Marinoni Moretto
ORCID: 0009-0004-0096-4712 (https://orcid.org/0009-0004-0096-4712)
Contact: francesco.marinoni.moretto@gmail.com
Status: Working paper
Date: May 2026
------------------------------------------------------------------------------
ABSTRACT
As AI agents transition from read-only assistants to autonomous actors, the
human-readable web is failing them. Current retrofitting approaches, such as
schema.org-enhanced HTML or Retrieval-Augmented Generation (RAG), force agents
to parse unstructured or weakly-structured corpora or expend finite token
budgets on human-centric rendering pipelines. In response, a parallel
publishing layer is taking shape: major AI operators - including OpenAI,
Anthropic, Stripe, and Google - are increasingly exposing machine-legible
surfaces optimized for autonomous traversal, deterministic retrieval, and
reduced token consumption. This paper identifies and analyzes this emerging
publishing pattern as Layered Agentic Retrieval (LAR).
LAR is a publisher-side architectural commitment to surface deterministic,
machine-legible data designed explicitly for agent traversal. We articulate
LAR's three core commitments - plain-text substrates, progressive disclosure,
and workflow-organized hierarchies - and situate it within the lineages of
RESTful hypermedia and the Semantic Web. By delineating the limits of current
retrofitting approaches, this paper establishes LAR as a dedicated
institutional layer through which publishers can expose machine-legible
retrieval, policy, and operational surfaces to autonomous agents - the
structural artifact onto which cryptographic attestation attaches and on which
emerging legal provenance frameworks can operate.
Keywords: agentic commerce, agent-mediated retrieval, web architecture,
knowledge graph, semantic web, RESTful hypermedia
ABBREVIATIONS
ACP Agentic Commerce Protocol (OpenAI/Stripe).
ADK Agent Development Kit (Google).
A2A Agent2Agent (Linux Foundation, originally Google).
HATEOAS Hypermedia as the Engine of Application State (Fielding, 2000).
HAL Hypertext Application Language.
JSON-LD JSON for Linked Data (W3C).
LAR Layered Agentic Retrieval (this paper).
MCP Model Context Protocol (Anthropic, server-side).
RAG Retrieval-Augmented Generation.
RDF Resource Description Framework (W3C).
SPT Shared Payment Tokens (Stripe, used by ACP).
UCP Universal Commerce Protocol (Google + retail coalition).
WebMCP Browser-native tool exposure standard (W3C Web ML CG, distinct from
MCP).
1. INTRODUCTION
A small set of operators - OpenAI, Anthropic, Stripe, Apideck, Google's Agent
Development Kit team, WordLift, Microsoft's NLWeb group - are publishing
knowledge to AI agents in a shape that does not look like a website. The
pieces of this shape have local working names; the whole does not. This paper
proposes a name - Layered Agentic Retrieval (LAR) - and articulates the
architectural commitments that distinguish what these operators ship from
conventional web publishing or crawler-friendly retrofit.
Each of these operators publishes a separate agent-facing layer, in order to
expose content optimized for agent traversal rather than human rendering.
While this pattern is already emerging in production, the vocabulary that lets
merchants, researchers, and policy actors discuss it as a single architectural
object is missing. The paper's contribution is naming and framing, not formal
specification or benchmarking. It situates LAR in its theoretical lineage,
distinguishes it from adjacent approaches (notably Retrieval-Augmented
Generation and schema.org-enhanced HTML), and identifies the empirical and
legal questions that remain open.
2. THE PATTERN EMERGING IN PRODUCTION
OpenAI's developer documentation site exposes /codex/llms-full.txt: a single
markdown file containing the complete documentation, regenerated on every doc
update.1 (#note1) Google's Agent Development Kit ships an
equivalent.2 (#note2) Apideck publishes a three-tier disclosure
pattern: /llms.txt as a summary index, /llms-full.txt as an enumerated index
of every doc page, individual .md files at each leaf, and a programmatic
md/index.json for machine traversal.3 (#note3) Other developer-tool
vendors - Meteor, FastMCP, assistant-ui, Chkk, Mintlify - ship the same
shape.4 (#note4) Anthropic's published skills are organised
similarly: each skill exposes its name and description in YAML frontmatter
(the manifest), with SKILL.md as the entry point and additional resources -
markdown documentation, executable scripts, and bundled assets - in
subdirectories the agent loads only when its task requires them.5
(#note5)
The surfaces6 (#note6) examined here share observable structural
properties: the markdown and JSON files appear to be regenerated alongside doc
updates rather than scraped from rendered HTML. This pattern is consistent
with shared upstream source material with the human-facing artifacts, though
it does not prove it.
These concrete forms differ in their surface organisation, but three
properties recur across all of them:
Root declaration. A root surface (whether a single file or a manifest)
declares what is published and links to deeper material.
Format separation. Narrative content is held in markdown while structured
content requiring deterministic parsing is held in JSON, with the two kinds
separated rather than interleaved.
Explicit machine-resolvable links. Links between files are explicit and
machine-resolvable, allowing an agent to fetch the root, decide which subtree
its task requires, and walk down only as far as it needs - without rendering,
scraping, or inferring structure from HTML.
The protocol layer is taking shape in parallel. Microsoft's NLWeb defines how
a website exposes structured data to agents through a knowledge-graph-backed
natural language query interface.7 (#note7) Google's Universal
Commerce Protocol (UCP), launched at NRF 2026, standardises commerce
capabilities across AI surfaces.8 (#note8) OpenAI and Stripe's
Agentic Commerce Protocol (ACP), open-sourced in September 2025, addresses the
same transactional layer through agent-mediated execution within
conversational surfaces.9 (#note9) WebMCP, a W3C Community Group
Report (latest draft 10 February 2026), exposed in early 2026 Chrome 146
preview builds behind the "Experimental Web Platform Features" flag, lets a
website declare itself as a tool surface for in-browser agents.10
(#note10)
These initiatives address different layers - site-level discovery, commerce
capability negotiation, agent-mediated transaction, browser-native tool
exposure - but share an underlying assumption: agents need surfaces designed
for them, and those surfaces will be declared, not inferred.
External infrastructure providers are operationalising this layer distinction.
Google's Search Central guidance, published 15 May 2026, explicitly states
that publishers "don't need to create new machine readable files, AI text
files, markup, or Markdown to appear in generative AI search," while a
parallel section recognising autonomous agents notes that "protocols like
Universal Commerce Protocol (UCP) are emerging that will allow Search agents
to do more."11 (#note11) Cloudflare's Agent Readiness Score,
launched 17 April 2026, operationalises assessment of publisher-side agent
infrastructure across four dimensions (discoverability, content, bot access
control, capabilities) plus commerce protocols, with adoption tracked across
108,595 successfully scanned domains as of May 2026.12 (#note12)
The asymmetry is informative: Google scopes its guidance to Search
optimisation while signalling that agentic experiences require different
infrastructure; Cloudflare operationalises that infrastructure with empirical
adoption data. The publisher/agent distinction LAR articulates is becoming
externally recognised in the ecosystem the paper describes.
3. THREE ARCHITECTURAL COMMITMENTS
The operators building these surfaces deliberately, rather than as accidental
byproducts of documentation tooling, share three commitments that distinguish
their output from retrofit.
Plain text instead of pages. Markdown for narrative, JSON for facts, YAML for
configuration. No DOM, no JavaScript, no rendering pipeline. An agent parses
these formats deterministically in milliseconds, with no model call required
for extraction. The llms.txt specification states the rationale directly:
context windows are too small to handle most websites in their entirety, and
converting complex HTML pages (with navigation, ads, and JavaScript) into
LLM-friendly plain text is difficult and imprecise.4 (#note4)
Progressive disclosure instead of flat exposure. A manifest at the root, topic
indexes one layer down, detail files at the leaves. The agent walks the tree
to the depth its task requires and stops. Token budget is controlled by the
agent, not by the publisher's page boundaries. This is the structural inverse
of a sitemap. Anthropic's Claude Skill Design has codified this pattern as
progressive knowledge disclosure for tool packaging, where agents see only
skill names and short descriptions initially and load detailed files only when
needed.5 (#note5) MajesticLabs's hierarchical-agents pattern
applies the same principle to codebase documentation.13 (#note13)
Organised by workflow instead of by topic. A merchant's human site is
organised by what the merchant wants to say: about us, our story, our
products, contact. A LAR surface is organised by what the agent was tasked to
do, and the discretion the principal delegated for doing it: availability,
inventory, policies, pricing, the endpoint that commits the transaction. The
structure mirrors the agent's mandate, not the merchant's marketing.
Each commitment is explicitly stated in the documentation of the operators
cited above; what is proposed here is the framing that holds the three
together as a single architectural pattern.
The theoretical justification is the economics of the LLM context window: an
agent operating under finite token budget pays a cost proportional to every
byte loaded, and topic-organised structures force the agent to traverse
material organised around the publisher's narrative concerns before reaching
the data its task actually requires.
Workflow organisation collapses this asymmetry by aligning the structure of
the surface with the structure of agent decisions, allowing token usage to
scale with task complexity rather than with publisher verbosity.14
(#note14)
The discipline of building a good LAR surface is therefore not to design the
path the agent will follow, but to remove obstacles from every path the agent
might take. The work is to expose relevant data efficiently, without forcing
the agent through structure that reflects the publisher's mental model rather
than the agent's task.
These three commitments are what make the architecture layered rather than
retrofit, agentic rather than crawler-friendly, retrieval-shaped rather than
render-shaped.
LAR is not Retrieval-Augmented Generation. RAG is a consumer-side pattern that
retrieves probabilistically over unstructured or weakly-structured corpora to
augment generation; LAR is a publisher-side commitment to structure the source
so that retrieval can be deterministic when the consuming agent respects the
layered structure.15 (#note15)
Two terms are used in this paper with deliberate distinction, doing different
work even when they describe the same artifact.
Machine-readable describes formats a machine can deterministically parse - the
standard sense established by RDF, HATEOAS, and the broader Semantic Web
tradition.
Machine-legible describes content a machine can meaningfully act upon. JSON
exposing five generic product attributes is machine-readable; JSON exposing
twenty content-rich attributes that capture provenance, occasion-fit, and
operational constraints is machine-legible.
The conceptual distinction between format and meaning has surfaced in adjacent
literature on digital product labels and Agent Experience design; the two-term
vocabulary proposed here is intended to make the distinction usable in
operator and policy discourse.
LAR commits to both: machine-readable formats (plain text, JSON) carrying
machine-legible content (workflow-organised, identity-bound,
attestation-ready).
4. THEORETICAL LINEAGE AND RECENT CONVERGENCE
Although the vocabulary proposed here is new, the pattern it names is not.
Manifest-driven traversal of linked, machine-readable resources has scholarly
lineage.
The hypermedia-driven REST architectures articulated by Fielding (2000) -
commonly known as HATEOAS - describe this shape: a client fetches a root
resource, discovers links at runtime, and traverses progressively to fulfil
its task.16 (#note16) The early Semantic Web vision of Berners-Lee,
Hendler and Lassila (2001) extended the pattern to machine-readable linked
data, anticipating agents that would compose actions across distributed
semantic resources.17 (#note17)
LAR is best understood as an instantiation of this lineage adapted to the
LLM-agent consumer: plain text and JSON calibrated for context-window
absorption rather than HAL or RDF, organisation by task-workflow rather than
resource-entity, consumer that performs semantic parsing on prose rather than
following typed links. The architectural innovation, where one exists, lies in
the calibration to the consumer rather than in the traversal pattern itself.
Recent practitioner thinking has converged on a parallel architectural pattern
from the consumer side. Andrej Karpathy (OpenAI founding member, former Tesla
AI Director) published his LLM Wiki sketch (2026) via a post on X and a gist
on GitHub - the genre by which influential AI thinking now first reaches the
field - describing an LLM agent ingesting a user's raw sources, compiling them
into a persistent markdown knowledge base, and maintaining it across
queries.18 (#note18)
The sketch arrives at architectural decisions that overlap with LAR's
publisher-side commitments: markdown substrate, three-layer separation,
rejection of RAG as default. Karpathy's wiki is what an agent maintains for
one user's consumption; LAR is what a publisher exposes for any agent's
traversal. The convergent design choices on opposite ends of agent-mediated
information flow are the more interesting observation than the architectural
overlap itself.
A strict architectural critique might observe that the systems discussed above
operate at different layers of the software stack: llms.txt addresses
documentation discovery, MCP standardizes tool invocation, and ACP governs
transactional execution. LAR does not claim these protocols are equivalent,
nor that they require a shared implementation layer.
The claim is narrower and structural: finite-context autonomous agents create
common economic pressures across discovery, retrieval, and execution. In
response, operators are increasingly exposing machine-legible institutional
surfaces designed to reduce ambiguity, traversal cost, and rendering overhead
for autonomous systems. LAR is proposed as a publisher-side conceptual
vocabulary for understanding this convergence.
5. THE LIMITS OF RETROFIT
The dominant publisher-side approach to making content readable by agents is
structured data embedded in HTML using schema.org vocabulary, typically
expressed as JSON-LD inside