High Priority
Deploy Community /ai.txt Protocol
Establish a machine-readable summary of your entire community hierarchy specifically for AI agents and knowledge graph builders.
Create a text file at /ai.txt with a brief introduction to your community's purpose and scope.
Include markdown-style links to your most important community spaces (e.g., key forums, onboarding guides, popular discussion categories).
Add a 'FAQ' section within the file to directly answer common queries from AI training bots about community governance, moderation policies, and user demographics.
Configure your Online communities crawler protocols effortlessly.
Join 2,000+ teams scaling with AI.
High Priority
Community Bot Selective Indexing
Fine-tune which sections of your online community should be ingested by AI crawlers (e.g., for building knowledge bases or analyzing discussion trends).
User-agent: CommunityBot Allow: /discussions/ Allow: /knowledge-base/ Disallow: /user-profiles/
Verify your crawler permissions using a simulated bot tester or by monitoring initial crawl requests.
Monitor crawl frequency in your server logs to ensure community bots are accessing designated content areas and not sensitive user data.
Medium Priority
Semantic HTML for Discourse Structure
Utilize HTML5 landmarks and semantic tags to help LLM scrapers understand the hierarchical structure and topical relevance of your community content.
Wrap core discussion threads within <article> tags to signal their individual importance and topic.
Use <section> tags with descriptive 'aria-label' attributes for distinct community categories or sub-forums (e.g., 'aria-label="Technical Support Discussions"').
Ensure all data presented in tables (e.g., user leaderboards, feature comparison charts) uses proper <thead> and <tbody> tags for structured data extraction.
High Priority
Knowledge Graph-Friendly Snippet Optimization
Structure community content and answers so they can be easily extracted and integrated into Retrieval Augmented Generation (RAG) pipelines or knowledge graphs.
Keep logically related discussion points or knowledge base articles within cohesive, digestible segments (e.g., under 750 words).
Avoid 'floating' context; explicitly state the primary subject or question being addressed at the beginning of each segment.
Eliminate ambiguous pronouns (e.g., 'it', 'they', 'this') and replace them with the specific community topic, feature, or user role being discussed.
