llms.txt Explained: The New robots.txt for AI Crawlers

Everyone knows robots.txt. Most developers have at least glanced at a sitemap.xml. But llms.txt? When I added it to Sharry a few weeks ago, most people I mentioned it to had never heard of it.

That surprised me, because the reason I bothered is already visible in my analytics: chatgpt.com has been one of Sharry's top 3 referrers for weeks now, right behind AlternativeTo and Google sign-ins. People are asking ChatGPT (and Claude, and Perplexity) for screen-sharing tools instead of Googling — and I have no control over what those models say about Sharry unless I make it easy for them to find out.

That's what llms.txt is for. Here's how it fits next to the two files you already know, with real code from Sharry's production Bun server.

robots.txt: the original "can I come in?"

robots.txt is the oldest of the three — a plain-text convention from 1994. It sits at the root of your domain and tells crawlers which paths they're allowed to request. It's not a security boundary (nothing stops a bad actor from ignoring it), just a polite instruction that well-behaved crawlers — Googlebot, Bingbot, and most AI crawlers — actually follow.

Sharry's version is generated per-request, so staging locks itself down automatically:

export function buildRobotsTxt(baseUrl: string): string {
  const isProduction = baseUrl === "https://sharry.live";
 
  if (!isProduction) {
    return `User-agent: *
Disallow: /`;
  }
 
  const disallows = ROBOTS_DISALLOW_PATHS.map(
    (path) => `Disallow: ${path}`,
  ).join("\n");
 
  return `User-agent: *
Allow: /
${disallows}
 
Sitemap: ${baseUrl}/sitemap.xml`;
}

ROBOTS_DISALLOW_PATHS is just /host, /join, /login, /auth/ — the actual app screens, which have nothing to rank for and shouldn't show up in search results. Everything marketing-facing (/, /pricing, /faq, the use-case pages) stays allowed.

sitemap.xml: the map of what's actually there

robots.txt says what crawlers can't touch. sitemap.xml says what's actually worth looking at — an explicit, machine-readable list of URLs, plus hints about how often each one changes and how important it is relative to the rest of the site.

// excerpt — the real list has a few more localized/use-case entries
export const SITEMAP_PAGES: readonly SitemapPage[] = [
  { path: "/", priority: "1.0", changefreq: "weekly" },
  { path: "/faq", priority: "0.9", changefreq: "monthly" },
  { path: "/changelog", priority: "0.6", changefreq: "weekly" },
  {
    path: "/use-cases/screen-sharing-to-tv",
    priority: "0.85",
    changefreq: "monthly",
  },
  {
    path: "/use-cases/remote-support",
    priority: "0.85",
    changefreq: "monthly",
  },
  {
    path: "/use-cases/discord-alternative",
    priority: "0.8",
    changefreq: "monthly",
  },
  { path: "/pricing", priority: "0.8", changefreq: "monthly" },
  { path: "/contact", priority: "0.5", changefreq: "monthly" },
  { path: "/imprint", priority: "0.3", changefreq: "yearly" },
  { path: "/privacy", priority: "0.3", changefreq: "yearly" },
] as const;

This list is deliberately curated, not auto-generated from the router. Sharry's actual routes include /host and /join too, but those are per-session app state, not content — there's nothing there for a search engine to index, so they're left out entirely rather than merely disallowed.

Both of these files exist for the same audience: crawlers that build an index of pages, one URL at a time. That model works fine for search engines. It doesn't really work for a chatbot that's expected to answer "what is Sharry and how much does it cost" in a single response, without crawling five separate pages and synthesizing them itself.

llms.txt: the new file for a new kind of visitor

llms.txt is a proposed convention (via llmstxt.org, not an official web standard — no W3C, no IETF RFC) for giving language models a single, curated, plain-Markdown summary of a site: what it does, how it works, what it costs, and which pages matter most. Instead of a machine parsing rendered HTML and guessing at what's marketing copy versus navigation chrome, it just gets handed the facts directly.

Here's the (trimmed) function that generates Sharry's:

/**
 * Generates /llms.txt — a structured Markdown summary for LLM crawlers.
 * See https://llmstxt.org/ for the convention.
 */
export function buildLlmsTxt(baseUrl: string): string {
  return `# Sharry
 
> Instant, encrypted screen sharing in your browser — no download, no account required.
 
Sharry is a WebRTC-based screen sharing web application. A host shares their
screen from any modern browser; viewers join using a Session ID and password,
also without installing anything. Connections are peer-to-peer and end-to-end
encrypted via WebRTC/DTLS — no media data ever passes through the server.
 
## Pricing
 
| Plan | Price | Session limit | Viewer limit |
|------|-------|---------------|--------------|
| Free | €0/mo (forever free) | 5 minutes | 1 viewer |
| Pro  | €0/mo during Early Access (€9/mo after) | Unlimited | Unlimited |
 
## Pages
 
- [Home](${baseUrl}/): Start sharing or join a session
- [Pricing](${baseUrl}/pricing): Plan details and Early Access info
- [FAQ](${baseUrl}/faq): Frequently asked questions
// ...more pages
`;
}

And it's served exactly like the other two — plain text, at a well-known path:

// llms.txt - Structured Markdown summary for LLM crawlers (https://llmstxt.org/)
if (url.pathname === "/llms.txt") {
  const baseUrl = process.env.BASE_URL || "https://sharry.live";
  return new Response(buildLlmsTxt(baseUrl), {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

The full version also lists concrete use cases (remote support, TV casting, pair programming) and links every marketing page, so a model answering "what's a good Discord alternative for screen sharing" has the actual, current answer instead of whatever it last saw in training data — which, for a product that ships changes weekly, is stale within days.

I'll be honest about the catch: unlike robots.txt, there's no formal spec anyone has to follow, and unlike sitemap.xml, no search engine requires it. Some AI crawlers reportedly read it, others ignore it, and there's no public list of which is which. It's speculative. But it's a single Markdown file that took under an hour to write, costs nothing to serve, and — worst case — nothing reads it and nothing changes.

All three, side by side

Each answers a narrower question than the last, and each targets a slightly different visitor. If your site only has the first two, you're covering crawlers that build an index. If people are already asking ChatGPT about products like yours — check your referrers, you might be surprised — the third one is worth an hour of your time too.

Adding these to your own site

None of this needs a specific framework. The pattern is the same everywhere:

1. robots.txt — one file at your domain root. Disallow app/auth routes, allow everything marketing-facing, and link your sitemap.
2. sitemap.xml — a curated, hand-maintained list of public pages with priority/changefreq hints. Resist the temptation to auto-generate it from your router; app routes and account pages don't belong in it.
3. llms.txt — a Markdown file, served as text/plain, summarizing what your product is, what it costs, and which pages explain what. Write it like you're briefing a smart colleague who's never seen your site, not like ad copy.

Wrapping up

I don't have hard evidence yet that llms.txt moves the needle — it's been live for a few weeks, and Sharry's ChatGPT referrals predate it, so I can't claim credit. What I can say is that robots.txt and sitemap.xml used to be the whole conversation about "how does my site talk to machines," and that conversation now has a third participant. Cheap insurance, in other words. I'll report back if the referrer data shows a real shift.

You can see Sharry's live files yourself: robots.txt, sitemap.xml, llms.txt.

Questions? Find me on X @pr0gstar.