High Priority
Deploy /community.txt Protocol
Establish a machine-readable summary of your entire forum hierarchy, key categories, and critical discussion threads specifically for AI agents and knowledge extraction bots.
Create a text file at /community.txt with a brief introduction to your forum's purpose and primary topics.
Include markdown-style links to your most important categories, sticky threads, and official announcements.
Add a 'FAQ' section within the file to address common queries about forum rules, user roles, or technical support for AI agents.
Configure your Forums crawler protocols effortlessly.
Join 2,000+ teams scaling with AI.
High Priority
LLM Bot Selective Indexing
Fine-tune which sections of your forum (e.g., specific categories, user-generated content types) should be ingested by AI crawlers for knowledge graph construction or sentiment analysis.
Implement `User-agent: LLM-Bot Allow: /category/general-discussion/ Allow: /category/technical-support/ Disallow: /private-messages/ Disallow: /user/settings/
Verify your crawler permissions and access patterns using tools like `crawl-test.com` or by monitoring server access logs for the specified user-agent.
Monitor crawl frequency and depth in your server logs to ensure LLM bots are accessing relevant discussion threads and not overwhelming user-specific or administrative sections.
Medium Priority
Semantic Thread Structure & Ingestion
Utilize semantic HTML5 elements and structured data to help LLM scrapers understand the hierarchy and context of forum posts, replies, and user profiles.
Wrap individual discussion threads or 'topics' within `<article>` tags to signal their primary content.
Use `<section>` with descriptive `aria-label` attributes for distinct forum categories or sub-forums (e.g., `aria-label='Software Development Discussions'`).
Ensure all user-generated content, especially structured data within posts (e.g., code snippets, bug reports), uses appropriate semantic tags like `<code>`, `<pre>`, and adheres to Schema.org markup for `DiscussionForumPosting`.
High Priority
RAG-Ready Snippet Optimization for Discussions
Structure forum content to be easily 'chunked' and retrieved by Retrieval-Augmented Generation (RAG) pipelines for AI-powered Q&A or knowledge base generation.
Keep related posts, replies, and their context within a logical container of approximately 500-750 words to facilitate effective chunking.
Avoid 'floating' context by ensuring each post or reply explicitly references the main topic or preceding post it's replying to, using clear identifiers.
Eliminate ambiguous pronouns (e.g., 'It,' 'They,' 'This') and replace them with explicit references to the product, feature, or user being discussed to improve RAG accuracy.
