AI Crawlers (llms.txt)

As people increasingly discover sites through AI assistants, being readable by AI crawlers matters as much as classic SEO. SpiderPublish makes your content available to ChatGPT, Claude, Perplexity, Gemini, and others in clean, machine-friendly form — no scraping, no JavaScript rendering, no plugins.

llms.txt — the index

Your site automatically serves an /llms.txt file following the llmstxt.org convention. It is an index of your site: the titles and links of your published pages, posts, and docs, each with its description. AI crawlers use it to quickly understand what your site contains and where to look.

llms.txt honors your indexing settings — pages you have set to noindex are left out, just like your sitemap.

llms-full.txt — the full content

Where llms.txt is an index, /llms-full.txt is the whole thing: the full content of your site rendered as clean Markdown, in one fetch. An AI crawler can ingest your entire published site at once, with no scraping and no JavaScript to execute.

This surface is opt-in — you enable it in your site settings. When on, it:

  • includes your published pages, posts, and docs as Markdown,

  • respects noindex (those pages are excluded),

  • is size-governed, so very large sites are capped cleanly with a clear marker rather than silently cut off.

Per-page Markdown

Every page is also available as Markdown by adding .md to its URL — for example /about.md or /blog/my-post.md. This is a clean, structure-preserving version of the page that AI tools and scripts can consume directly. Text inside interactive components is included too, so a page built entirely from components still produces a readable Markdown version (and feeds llms-full.txt).

Why enable it

If you want your product, your docs, and your writing to show up accurately when someone asks an AI assistant about your space, these surfaces are how. They give the crawlers exactly what you want them to read, in the form they read best.

A note on component-built pages

Pages built entirely from a custom interactive component are not serialized into the AI surfaces — those surfaces render content authored as blocks, posts, and docs. If you need a page in the AI corpus, build it with content blocks (or include the key text as block content).

Tips

  • Enable llms-full.txt if AI discoverability matters to you — it is the single biggest lever.

  • Keep descriptions filled in. They feed llms.txt and help crawlers route to the right page.

  • Noindex still means hidden. A noindexed page stays out of the AI surfaces too — exactly what you want for utility pages.

Next steps