LLM.txt & AI Crawler Setup Guide for API products

Create a text file at /llm.txt with a concise overview of your API product's core functionality and target use cases.

Include markdown-style links to critical documentation pages: API reference, authentication guides, SDKs, and common integration patterns.

Add a 'API Documentation FAQ' section within the file to directly address common queries from AI models regarding API usage, rate limits, and error codes.

Implement `User-agent: LLM-Crawler-Name Allow: /docs/api-reference/ Allow: /docs/guides/ Disallow: /docs/examples/internal/ Disallow: /account/

Ensure your OpenAPI spec uses descriptive `summary` and `description` fields for all endpoints, parameters, and response schemas.

Use semantic HTML5 landmarks (`<article>`, `<section>`, `<aside>`) within your developer portal pages, particularly for detailing individual API endpoints and their associated request/response examples.

Structure API reference pages with clear headings (H2, H3) that map directly to API operations and resource types, facilitating hierarchical understanding by LLMs.

Present API request/response pairs in discrete, self-contained blocks (e.g., JSON snippets) that are easily isolatable.

Ensure code examples are contextually relevant and include explanations of parameters, return values, and potential error states.

Eliminate ambiguous references within documentation; explicitly name API endpoints, parameters, and data fields instead of using pronouns.