LLM.txt & AI Crawler Setup Guide for Dev Tools

Create a JSON file at `/.well-known/ai-manifest.json` with a concise overview of your dev-tool's purpose.

Include structured JSON objects mapping key documentation sections (e.g., 'API Reference', 'SDK Guides', 'Tutorials') to their respective URLs.

Add a 'capabilities' array to specify supported languages, frameworks, and common integration patterns relevant to developers.

Define specific User-Agent rules in your `robots.txt` or use HTTP headers for targeted access (e.g., `User-agent: Copilot-Bot Allow: /docs/api/ Allow: /sdk/python/ Disallow: /internal-staging/`).

Utilize API gateway configurations or middleware to enforce access policies based on AI agent identifiers.

Monitor server logs for unusual access patterns or excessive requests from known AI bot user agents to detect potential scraping abuse.

Wrap code blocks within `<pre><code>` tags and assign appropriate `class` attributes for syntax highlighting and language identification (e.g., `class="language-javascript"`).

Use `<h2>`, `<h3>` for clear section hierarchies within documentation pages, mirroring logical code organization (e.g., 'Initialization', 'Methods', 'Event Handlers').

Implement JSON-LD schema markup for code examples, detailing parameters, return types, and usage context to enhance programmatic understanding.

Isolate distinct code functions or API endpoints within self-contained blocks, ideally under 1000 tokens, with clear headings.

Ensure code examples are complete and runnable within their context, avoiding reliance on external, undefined variables within the snippet.

Replace ambiguous placeholders (e.g., `YOUR_API_KEY`, `[variable_name]`) with explicit examples or clear instructions on how to obtain/define them.