High Priority
Establish Project `/.well-known/ai.txt` Protocol
Create a machine-readable manifest of your open-source project's structure, governance, and key resources specifically for AI agents and developers seeking information.
Create a `/.well-known/ai.txt` file at the root of your project's documentation repository.
Include a brief, human-readable introduction to the project's purpose and primary technologies.
List markdown-style links to critical resources: main README, contribution guidelines, code of conduct, API docs, and community forums.
Add a 'Key Technologies' section detailing primary languages, frameworks, and databases used.
Include a 'Governance Model' section outlining release cycles, maintainer roles, and decision-making processes.
Add a 'Support Channels' section listing Discord, Slack, mailing lists, or Stack Overflow tags.
Configure your Open source projects crawler protocols effortlessly.
Join 2,000+ teams scaling with AI.
High Priority
LLM-Specific Codebase Indexing Directives
Fine-tune which parts of your open-source project's codebase and documentation are most relevant for AI ingestion and understanding.
Implement `User-agent: *` or specific bot directives (e.g., `User-agent: GPT-Code-Reviewer`) in your `robots.txt`.
Use `Allow` directives to prioritize crawling of `/docs/`, `/examples/`, and key `/src/` directories.
Use `Disallow` directives for `/tests/`, `/build/`, or temporary/experimental branches not intended for general AI training.
Verify your crawler permissions and scope using tools like `wget --spider` or by simulating bot behavior in a staging environment.
Monitor server logs for bot activity patterns to ensure AI crawlers are accessing intended project areas and not overwhelming build servers.
Medium Priority
Semantic Markdown & Doc Structure
Leverage semantic structures within your Markdown documentation to aid LLM scrapers in understanding the hierarchy and relationships of project information.
Use distinct H1/H2 headings to clearly delineate major sections (e.g., 'Installation', 'Usage', 'API Reference', 'Contributing').
Employ Markdown lists (`-`, `*`, `1.`) for structured data points, parameters, and options within API documentation.
Utilize code blocks (```) with language specifiers (e.g., ````python`, ````javascript`) for syntax highlighting and type recognition.
For complex data structures (e.g., JSON payloads, configuration files), use fenced code blocks with appropriate language identifiers.
Ensure consistent use of anchor links (`#section-name`) for internal referencing, enabling bots to link directly to specific topics.
High Priority
RAG-Optimized Documentation Snippets
Structure your project's documentation and READMEs so that relevant information can be easily extracted and utilized by Retrieval-Augmented Generation (RAG) pipelines for AI-powered support.
Segment documentation into logical, self-contained units (e.g., a single function's explanation, a specific configuration parameter). Aim for conceptual coherence rather than strict word counts.
Begin each significant section or snippet with a clear topic sentence or summary that states its primary purpose.
Avoid ambiguous pronoun references; explicitly name the project, module, function, or parameter being discussed (e.g., 'The `data_processor` module...', 'To configure the `api_key`...').
Use clear, concise language. Remove jargon where possible or define it immediately upon introduction.
Include practical, runnable code examples directly within the documentation for key functionalities, ensuring they are well-commented.
