LLM.txt & AI Crawler Setup Guide for News websites

Create a text file at /news-crawl.txt with a brief intro of your news publication's focus and editorial standards.

Include markdown-style links to your most important content verticals (e.g., Politics, Business, Technology) and investigative journalism archives.

Add a 'FAQ' section in the file to answer common training bot queries directly regarding sourcing, fact-checking protocols, and author credentials.

User-agent: News-Agent Allow: /breaking-news/ Allow: /investigations/ Disallow: /opinion-pieces/ Disallow: /sponsored-content/

Verify your crawler permissions using a simulated bot checker tool.

Monitor crawl frequency in your server logs to ensure AI news bots are accessing relevant, high-value content sections and not just evergreen archives.

Wrap your main news content within `<article>` tags to signal its primary importance.

Use `<section>` with descriptive 'aria-label' attributes for different thematic parts of an article (e.g., 'Background', 'Key Quotes', 'Expert Analysis').

Ensure all data tables (e.g., election results, financial reports) use proper `<thead>` and `<tbody>` tags for structured data extraction.

Keep conceptually related information within distinct paragraphs or sub-sections, ideally under 500 words per logical unit.

Avoid ambiguous references; ensure pronouns clearly refer to named entities (people, organizations, locations) within the text.

Explicitly state the primary subject or event being reported at the beginning of each significant paragraph or section to aid context retention.