Control what search engines (and AI crawlers) see on your site

Control what search engines and AI crawlers see

Not every page on your site belongs in Google. Login screens, password-reset pages, checkout confirmations, internal thank-you pages — these add noise to your search presence and, increasingly, to the AI crawlers that read your site through llms.txt.

SpiderPublish now gives you a one-click control for exactly this.

The toggle

Open any page in the editor, go to Page Settings → SEO, and you will see "Allow search engines to index this page." Leave it on (the default) and the page is listed normally. Turn it off and the page is immediately removed from your site's sitemap.xml and your llms.txt — so neither traditional search engines nor AI crawlers are pointed at it.

Sensible defaults

System pages your site needs but shouldn't advertise — login, sign-up, forgot-password, and the 404 page — are set to not indexed out of the box. You don't have to think about them.

For the AI era

llms.txt is the emerging standard that tells AI crawlers what your site is about. SpiderPublish keeps it in sync with the same toggle, and now lists each page with its SEO description — so the pages you do want surfaced are described clearly, and the ones you don't simply aren't there.

Good to know

After you change the setting, your sitemap.xml and llms.txt update right away, but search engines re-crawl on their own schedule — it can take time for the change to reflect in Google.