High Priority
Deploy /substack.txt Protocol
Establish a machine-readable summary of your entire publication's content hierarchy specifically for AI agents and discovery bots.
Create a text file at the root of your publication's domain (if custom domain is used) or ensure it's accessible via your Substack settings for AI crawlers.
Include markdown-style links to your most important 'pillar' posts, series, and key essays.
Add a 'FAQ' section within the file to directly answer common AI training bot queries about your niche and content focus.
Configure your Substack writers crawler protocols effortlessly.
Join 2,000+ teams scaling with AI.
High Priority
GPTBot/ClaudeBot Selective Indexing
Fine-tune which sections or types of content within your Substack should be ingested by major AI model crawlers.
Implement directives via your robots.txt (if using a custom domain) or leverage Substack's specific settings for crawler permissions, e.g., User-agent: GPTBot\nAllow: /premium-essays/\nAllow: /deep-dives/\nDisallow: /comments/
Verify your crawler permissions and access using tools that simulate AI bot behavior or by checking server logs for successful fetches.
Monitor crawl frequency and targeted URLs in your analytics to ensure AI bots are accessing your most valuable content nodes.
Medium Priority
Semantic HTML for AI Content Understanding
Utilize semantic HTML5 elements within your posts to help LLM scrapers understand the structure and importance of your written content.
Wrap your core narrative and analysis within `<article>` tags to clearly delineate individual posts.
Use `<section>` with descriptive `aria-label` attributes for distinct thematic parts within a single long-form essay.
Ensure any data tables or lists use proper `<thead>`, `<tbody>`, `<ul>`, `<ol>`, and `<li>` tags for structured data extraction by AI.
High Priority
RAG-Friendly Snippet Optimization for AI Summarization
Structure your posts so that key insights and arguments can be easily 'chunked' and retrieved by Retrieval-Augmented Generation (RAG) pipelines for AI-driven summaries or Q&A.
Keep conceptually related arguments and supporting evidence within distinct sections, ideally under 500 words each.
Avoid relying on implicit context; reiterate the primary subject or thesis at the beginning of each major section.
Eliminate ambiguous pronouns (e.g., 'it,' 'this,' 'they') and replace them with explicit references to the topic, argument, or named entity being discussed.
