Get Discovered by AI. Steal the Playbook.
I asked Claude to recommend a tool in a category I had spent six weeks building a product for. It named three companies. The one I had built was not one of them, even though the homepage said exactly what the buyer was searching for, in plain English.
The product was not the problem. The problem was that the agent doing the searching had never been allowed to read the site, and the crawlers that had read it were the ones I should have been blocking. I had the SEO backwards. Not wrong by a little. Backwards.
This is the build log for fixing that: how I made a production Next.js app discoverable and usable by AI agents, what I learned about the two completely different populations of bot crawling your site right now, and the four-layer stack I shipped to handle them. The whole thing is free this week, no paywall: the concepts, the architecture, the exact crawler lists, and the working code. I was out for three weeks, so consider it a welcome-back.
The thing nobody set up their site for
For fifteen years, “being found” meant one thing: rank on Google, get a human to click a blue link, land on your page. Every SEO playbook optimizes for that single motion.
That motion is no longer the only one, and for a growing slice of buyers it is not even the main one. When someone wants to know “what is the best tool for X,” a real fraction of them now ask ChatGPT, Claude, or Perplexity instead of Google. The agent fans out, reads a handful of pages, and returns a synthesized answer with a few citation links.
If your site is not in that answer, you do not get a second-page ranking. You get nothing. There is no page two in an agent’s reply. There are three citations, and you are either one of them or you are invisible.
This shift already has names, and they are worth knowing because each one optimizes for a different thing:
SEO, Search Engine Optimization. Rank a page so a human clicks a blue link. Twenty years of playbooks, all optimizing for a position in a list.
AEO, Answer Engine Optimization. Be the answer instead of the link. The buyer asks and gets a direct response (a featured snippet, a voice answer, an AI overview), often with no click at all.
GEO, Generative Engine Optimization. Be one of the few sources a model cites when it composes an answer inside ChatGPT, Claude, or Perplexity.
The through-line: SEO assumes a human eventually reads your page. AEO and GEO assume a machine reads it first and a human may never see the page at all. That is why a site with flawless SEO can still be invisible to an agent. It was built for a reader with eyes, and the new reader does not have any. Nobody set their site up for that reader, including me.
So the question stopped being “how do I rank?” and became “when an agent reads my site on behalf of a buyer, can it actually read it, and does it know what it is looking at?”
For this app, the honest answer was no on both counts.
Here is what that means if you run a business and not just a website: the buyers you lose this way never show up in your analytics. A human who never finds you at least never registers as a bounced session, because there is no event to measure. You are not underperforming on a dashboard. You are absent from a shortlist you cannot see. That is the part that should worry a VP of Marketing more than any ranking drop: the failure mode is silent, and the first time you notice it is when a prospect says “I asked Claude and it recommended three other vendors.”
The failed first move: I almost blocked everyone
My first instinct was the one most teams have right now, and it is wrong.
The discourse in 2025 was all about AI companies scraping your content to train their models without permission or payment. The defensive reflex that came out of that was simple: block the AI bots. Drop GPTBot, ClaudeBot, CCBot, and Google-Extended into robots.txt with Disallow: / and feel good about protecting your IP.
I started down exactly this path. And if I had finished it, I would have made the invisibility problem worse, because that reflex collapses two completely different jobs into one rule.
Here is the distinction that reorganized the whole project:
There are two populations of bot crawling the web, and they do opposite things for your business.
Training bots (
GPTBot,ClaudeBot,anthropic-ai,Google-Extended,CCBot,Bytespider, and friends) crawl your pages now to bake into the next model’s frozen weights. They do not cite. They do not link back. They do not preserve attribution. Their contribution to you is, at best, vague brand recall in some future offline answer.Retrieval bots (
OAI-SearchBot,ChatGPT-User,Claude-Web,PerplexityBot,Perplexity-User,Mistralai-User) fetch your pages in response to a live user query and return an answer with a citation link. They drive clicks. They drive traffic. They drive conversion.
If you blanket-block “AI bots,” you are usually blocking both. You keep your content out of next year’s model weights, sure. But you also make yourself uncitable today, which is the exact opposite of what you want. You have protected the asset by making it impossible to sell.
If you are making the case for an AI-crawler policy internally, here is the framing that works: “Blocking training bots protects our content. Blocking retrieval bots makes us invisible to the buyers using AI to shortlist vendors. Those are two different bots, and right now our
robots.txtcannot tell them apart.”
That reframe is the entire point of the build. Once you separate the two populations, the rest of the architecture follows.
The Agent-Readable Stack
SEO makes you findable by search engines. What I shipped makes you readable by AI agents. I am calling it The Agent-Readable Stack, and it has four layers. Each one answers a question an agent asks when it lands on your site.
Layer 1, Permission: “Am I allowed to read this, and for what?” A robots.txt that does two things at once: it allows the retrieval bots, disallows the training-only bots, and declares a machine-readable content signal on every block stating your preference (ai-train=no, search=yes, ai-input=yes). The Allow/Disallow rules are the enforcement. The Content-Signal is the declared intent. You want both, because enforcement is what bots obey today and the signal is what the emerging standards will read tomorrow.
Layer 2, Inventory: “What pages exist here?” A sitemap.xml that lists your canonical public surfaces and nothing else. The discipline that matters: the sitemap must stay in sync with the Allow rules in Layer 1. If robots.txt allows it and the sitemap omits it, you have a blind spot. If the sitemap lists it and robots.txt blocks it, you are sending mixed signals to the crawler.
Layer 3, Format: “Can you give me this without the HTML soup?” Content negotiation that detects when an agent asks for text/markdown and serves clean markdown instead of a JavaScript-rendered page. Agents reason far better over structured markdown than over a wall of <div>s. This is the single highest-leverage layer and the one almost nobody ships, because it means treating “agent” as a first-class client alongside “browser.”
Layer 4, Catalog: “What can I actually do here?” Machine-readable description of your endpoints, via RFC 8288 Link headers and a /.well-known/api-catalog linkset (RFC 9264 / RFC 9727). This is how you tell an agent “the human-facing /chat page is backed by a conversational endpoint, and here is its documentation.” It turns your site from a set of pages into a described service.
Read top to bottom, the stack is a conversation: Am I allowed in? What do you have? Can you give it to me clean? And what can I do with it? Most sites cannot answer question one coherently. Almost none can answer all four.
What follows is the full build, no paywall: the exact robots.txt with the retrieval-vs-training crawler lists, the markdown content-negotiation middleware, the api-catalog linkset, and the canonical-host bug that quietly broke everything for a day.
Layer 1, in full: the two-population robots.txt
The core move is to write robots.txt in three blocks: a default policy, an explicitly-allowed retrieval tier, and an explicitly-disallowed training tier. Every block carries a Content-Signal line declaring intent even where the Allow/Disallow already enforces it.
# Default policy: allow public surfaces, protect app internals
User-agent: *
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /
Allow: /chat
Disallow: /api/
Disallow: /auth/
Disallow: /dashboard/
Disallow: /login
# RETRIEVAL / USER-INITIATED BOTS (ALLOWED)
# They fetch in response to a live user query and return cited answers.
User-agent: OAI-SearchBot
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /
Disallow: /api/
User-agent: ChatGPT-User
Allow: /
Disallow: /api/
User-agent: Claude-Web
Allow: /
Disallow: /api/
User-agent: PerplexityBot
Allow: /
Disallow: /api/
User-agent: Perplexity-User
Allow: /
Disallow: /api/
User-agent: Mistralai-User
Allow: /
Disallow: /api/
# TRAINING-ONLY BOTS (DISALLOWED)
# They crawl to feed frozen model weights with no citation back.
User-agent: GPTBot
Content-Signal: ai-train=no, search=yes, ai-input=yes
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: cohere-ai
Disallow: /
# Discovery
Sitemap: https://example.com/sitemap.xml
Two non-obvious decisions worth stealing:
The
Content-Signalline on every block, including the disallowed ones. Enforcement (Disallow) is what makes a bot stop today. TheContent-Signal(ai-train=no, search=yes, ai-input=yes, per the emerging Content Signals draft) is a declared, machine-readable preference that survives even where you are not hard-blocking. You are stating policy, not just setting a tripwire.Why not just let the training bots absorb your terminology? There is a real strategy called the “Kleenex play”: let your wording soak into the model so it answers with your term by default. It only works when your term is genuinely novel, you out-publish everyone, and you do not need the click. For a commercial funnel it fails on the click: a model describing your category without linking to you generates zero signups. And it is irreversible. Flipping
ai-train=notoyeslater is trivial. You cannot extract content already frozen into weights. So you capture category language through channels that carry attribution into training data anyway (bylined posts, podcasts, press) and keep the app itself retrieval-only.
Layer 2: a sitemap that cannot drift
The sitemap is deliberately tiny and generated, not hand-maintained, so it cannot fall out of sync with reality.
import type { MetadataRoute } from "next";
// Canonical public surfaces. Must stay in sync with robots.txt Allow rules.
const ROUTES = ["", "/chat"] as const;
export default function sitemap(): MetadataRoute.Sitemap {
const base = (process.env.NEXT_PUBLIC_SITE_URL ?? "https://example.com").replace(/\/$/, "");
const lastModified = new Date();
return ROUTES.map((path) => ({
url: `${base}${path || "/"}`,
lastModified,
changeFrequency: path === "" ? "weekly" : "monthly",
priority: path === "" ? 1.0 : 0.8,
}));
}
The one-line comment is load-bearing: ROUTES is the single source of truth, and it is the same list as the Allow rules in Layer 1. Drift between those two lists is the most common silent failure in this whole stack.
The canonical-host bug. For about a day, everything looked fine and nothing worked, because the canonical host pointed at a domain we had retired while the live site ran on the current one. Agents following the canonical URL bounced to a redirect and gave up. The fix was one line and a separate PR. The lesson: in agent-readable infrastructure, a wrong-but-plausible canonical host is worse than a missing one, because it sends the agent confidently in the wrong direction.
Layer 3: serving markdown to agents
This is the layer that separates “I added a robots.txt“ from “I built for agents.” When a client sends Accept: text/markdown, the middleware rewrites the request to a route that returns clean markdown instead of the rendered React page.
const MARKDOWN_DOCS = new Set(["/", "/chat"]);
export async function middleware(request: NextRequest) {
const { pathname } = request.nextUrl;
if (
request.method === "GET" &&
MARKDOWN_DOCS.has(pathname) &&
!request.headers.has("x-md-bypass") &&
/\btext\/markdown\b/i.test(request.headers.get("accept") || "")
) {
const rewriteUrl = request.nextUrl.clone();
rewriteUrl.pathname = "/api/md";
rewriteUrl.searchParams.set("path", pathname);
return NextResponse.rewrite(rewriteUrl);
}
// ... auth/session handling continues below
}
The x-md-bypass header is an escape hatch so internal tooling can force the HTML path. The MARKDOWN_DOCS set keeps the negotiation scoped to pages where a clean-text version actually makes sense. An agent reasoning over three paragraphs of structured markdown will out-perform the same agent trying to parse a hydrated SPA every time.
Layer 4: telling agents what your endpoints do
Two pieces. First, Link headers (RFC 8288) advertise related resources on the response itself. Second, a /.well-known/api-catalog linkset (RFC 9264, described in RFC 9727) gives a single machine-readable map of your service.
{
"$schema": "https://www.rfc-editor.org/rfc/rfc9727",
"linkset": [
{
"anchor": "https://example.com/",
"describedby": [
{ "href": "https://example.com/sitemap.xml", "type": "application/xml", "title": "Site map of public pages" }
],
"author": [
{ "href": "https://example.com/robots.txt", "type": "text/plain", "title": "Crawl policy (RFC 9309)" }
]
},
{
"anchor": "https://example.com/api/chat",
"service-doc": [
{ "href": "https://example.com/chat", "type": "text/html", "title": "Chat: conversational endpoint backing the public /chat experience" }
]
}
]
}
This is the layer that turns a website into a described service. The anchor for /api/chat with a service-doc link tells an agent: there is a conversational endpoint here, and this is what it does. That is the difference between an agent knowing you have a chat page and an agent knowing it can use it.
What I would do differently
Ship Layer 3 first, not last. Markdown content negotiation is the highest-leverage piece and I built it third. If I were starting over, I would prove the agent-readable path end-to-end with markdown before touching the crawler lists.
Add a synthetic agent check to CI. I verified the stack by hand. The right move is a test that sends
Accept: text/markdown, asserts a markdown response, and fails the build if an agent would get HTML soup. Drift betweenROUTES, the sitemap, and the Allow rules should be caught by a test, not by a reader noticing you are missing from a Perplexity answer.Treat the crawler list as a living file. New retrieval bots appear constantly. This is a maintained dependency, not a one-time config.
Template repo (optional): I am packaging this as a drop-in Next.js starter, agent-readable-stack-template, with all four layers, the full crawler lists, the markdown route, and the CI check. [LINK if ready by send; otherwise cut this line, the code above stands on its own.]
Steal this: apply it to your own stack
The story above is one Next.js app, but the stack is framework-agnostic. Here is the order to ship it on your own site, cheapest win first.
Start with Layer 1. The robots.txt above is pure text, identical on every stack, and takes twenty minutes: copy the three blocks, swap example.com for your domain, and make the one real decision (block training bots if you run a commercial funnel; keep them if you are running the Kleenex play). If you only have an afternoon, Layer 1 plus the Layer 2 sitemap is the 80/20. It moves you from “blanket-blocking everything” or “wide open” to “citable by the bots that send traffic, closed to the ones that do not.”
Layer 3 is the real differentiator, and it is the same idea on every framework: when a request carries Accept: text/markdown, return clean markdown instead of your rendered page.
// Express / Node: the same content-negotiation move as the Next.js middleware above
const MARKDOWN_DOCS = new Set(["/", "/chat"]);
app.get("*", (req, res, next) => {
const wantsMarkdown = /\btext\/markdown\b/i.test(req.get("accept") || "");
if (req.method === "GET" && MARKDOWN_DOCS.has(req.path) && wantsMarkdown) {
return res.type("text/markdown").send(renderMarkdown(req.path));
}
next();
});
Rails / Django / Laravel: do it in the controller with content negotiation (
respond_to/ aformatbranch). Same condition, same two outcomes.Static sites (Astro, Hugo, Eleventy): generate a
.mdtwin of each page at build time and serve it with an edge rule, or just publish/page.mdURLs.No code at all: if you are on Cloudflare, “Markdown for Agents” (in the Signal Stack below) does this at the edge. You may get Layer 3 for free.
Then verify every layer from your terminal before you trust it:
curl -s -H "Accept: text/markdown" https://yoursite.com/ | head # Layer 3: markdown back, not HTML soup
curl -s https://yoursite.com/robots.txt | grep -i "user-agent" # Layer 1: confirm your crawler tiers
curl -s https://yoursite.com/.well-known/api-catalog # Layer 4: the linkset JSON, if you ship endpoints
Layer 4 only matters if an agent could actually call something on your site (a chat, search, or booking endpoint). Pure content site? Skip it. Everyone else: ship Layers 1 through 3, run the three curl checks, and you are readable by the agents your buyers are already asking.
The Agent-Readable Stack, one screen: Permission (who crawls, train vs retrieve), Inventory (what exists), Format (serve clean markdown), Catalog (describe your endpoints). Block the bots that take. Welcome the bots that cite.
The next era of being found is not about out-ranking a competitor. Be legible to the reader with no eyes, and the human it answers for shows up too.
What is the messiest part of your current AI stack, the thing nobody talks about publicly? Hit reply, I read every one.
Next issue: I built a benchmark that measures the real cost of an AI model, not cost per token, but total cost to actually finish the job. The cheapest model per token is usually the most expensive per result, and I have the numbers.
Signal Stack
↑ Cloudflare’s Content Signals Policy now ships on 3.8M+ managed-robots.txt domains, defaulting to search=yes, ai-train=no (Cloudflare, Sept 2025). The train-vs-retrieve split in this issue is not a hot take. It is quietly becoming the web’s default setting.
↑ Cloudflare shipped “Markdown for Agents”: crawlers send Accept: text/markdown, and a 16,180-token HTML post comes back as roughly 3,150 tokens of clean markdown (InfoQ, Mar 2026). Layer 3 of the stack is not polish. The edge layer is standardizing it because the token math is five to one.
👀 Stack Overflow and Cloudflare turned on pay-per-crawl: a 402 Payment Required for training crawlers, with search and community reads still free (Stack Overflow, Feb 2026). The two-population split is hardening from a preference into a price. Charge the takers, welcome the citers.
👀 Only 7.4% of the Fortune 500 ship an llms.txt, and just 11% name a single AI user-agent in robots.txt (ProGEO.ai, Mar 2026). The agent-readable web is wide-open greenfield. The cost of being early is one config file.
↓ ChatGPT’s share of AI referral traffic fell to 64.5%, down from 86.7% a year earlier, with Gemini now at 21.5% (Similarweb via The Digital Bloom, Feb 2026). A GPTBot-only allowlist already ignores a third of the market. Allow every retrieval bot, not just OpenAI’s.
Know a builder who would actually run this? Forward it to them, it takes 5 seconds.
More Signal. Less Noise.