LLM.txt & AI Crawler Setup Guide for Substack writers

Create a text file at the root of your publication's domain (if custom domain is used) or ensure it's accessible via your Substack settings for AI crawlers.

Include markdown-style links to your most important 'pillar' posts, series, and key essays.

Add a 'FAQ' section within the file to directly answer common AI training bot queries about your niche and content focus.

Implement directives via your robots.txt (if using a custom domain) or leverage Substack's specific settings for crawler permissions, e.g., User-agent: GPTBot\nAllow: /premium-essays/\nAllow: /deep-dives/\nDisallow: /comments/

Verify your crawler permissions and access using tools that simulate AI bot behavior or by checking server logs for successful fetches.

Monitor crawl frequency and targeted URLs in your analytics to ensure AI bots are accessing your most valuable content nodes.

Wrap your core narrative and analysis within `<article>` tags to clearly delineate individual posts.

Use `<section>` with descriptive `aria-label` attributes for distinct thematic parts within a single long-form essay.

Ensure any data tables or lists use proper `<thead>`, `<tbody>`, `<ul>`, `<ol>`, and `<li>` tags for structured data extraction by AI.

Keep conceptually related arguments and supporting evidence within distinct sections, ideally under 500 words each.

Avoid relying on implicit context; reiterate the primary subject or thesis at the beginning of each major section.

Eliminate ambiguous pronouns (e.g., 'it,' 'this,' 'they') and replace them with explicit references to the topic, argument, or named entity being discussed.