High Priority
Deploy API-Spec /llm.txt Protocol
Establish a machine-readable inventory of your API product's endpoints, schemas, and key documentation sections specifically for AI agents and LLM web crawlers.
Create a text file at /llm.txt with a concise overview of your API product's core functionality and target use cases.
Include markdown-style links to critical documentation pages: API reference, authentication guides, SDKs, and common integration patterns.
Add a 'API Documentation FAQ' section within the file to directly address common queries from AI models regarding API usage, rate limits, and error codes.
Configure your API products crawler protocols effortlessly.
Join 2,000+ teams scaling with AI.
High Priority
LLM Crawler Selective Endpoint Indexing
Fine-tune which API documentation sections and reference pages should be ingested by specific LLM crawlers (e.g., for training AI assistants or knowledge graph construction).
Implement `User-agent: LLM-Crawler-Name Allow: /docs/api-reference/ Allow: /docs/guides/ Disallow: /docs/examples/internal/ Disallow: /account/
Medium Priority
Semantic OpenAPI Spec & HTML Ingestion
Leverage structured data within your OpenAPI specifications (Swagger/YAML) and semantic HTML in your developer portal to aid LLM scrapers in understanding API resource hierarchy and data relationships.
Ensure your OpenAPI spec uses descriptive `summary` and `description` fields for all endpoints, parameters, and response schemas.
Use semantic HTML5 landmarks (`<article>`, `<section>`, `<aside>`) within your developer portal pages, particularly for detailing individual API endpoints and their associated request/response examples.
Structure API reference pages with clear headings (H2, H3) that map directly to API operations and resource types, facilitating hierarchical understanding by LLMs.
High Priority
RAG-Optimized API Response Snippet Generation
Structure your API documentation and example responses so they can be easily parsed and utilized by Retrieval-Augmented Generation (RAG) pipelines for chatbots and AI assistants.
Present API request/response pairs in discrete, self-contained blocks (e.g., JSON snippets) that are easily isolatable.
Ensure code examples are contextually relevant and include explanations of parameters, return values, and potential error states.
Eliminate ambiguous references within documentation; explicitly name API endpoints, parameters, and data fields instead of using pronouns.
