High Priority
Implement `/robots.txt` for AI Agents
Establish a machine-readable directive file outlining your Framer site's content structure for AI agents, akin to a sitemap for bots.
Create a `robots.txt` file at the root of your Framer project (accessible via `yourdomain.com/robots.txt`).
Include a brief overview of your site's purpose and the types of content available for AI ingestion.
Use `Sitemap:` directives to point to your Framer XML sitemap, facilitating discovery of all pages.
Leverage `User-agent:` directives to specify rules for specific AI bots (e.g., `GPTBot`, `Google-Extended`) and `Disallow:` or `Allow:` rules for specific paths relevant to Framer component libraries or templates.
Configure your Framer users crawler protocols effortlessly.
Join 2,000+ teams scaling with AI.
High Priority
Framer Component-Level Indexing Control
Fine-tune which specific sections or Framer components on your site should be accessible and ingested by AI crawlers.
Within your Framer project settings or via meta tags on specific pages, define `noindex` directives for low-value or duplicate content sections (e.g., staging environments, internal admin pages).
Use `Allow:` directives in `robots.txt` for key areas like your public component showcases, design tutorials, or case studies.
Monitor server logs for AI crawler activity (e.g., `GPTBot`, `Google-Extended`) to ensure they are accessing intended Framer content paths and not resource-intensive areas.
Medium Priority
Semantic Structure for Framer Elements
Utilize HTML5 semantic elements within Framer to improve AI scrapers' understanding of content hierarchy and relationships between design elements.
Wrap core content areas within Framer pages using `<main>` and `<article>` tags to denote primary content.
Employ `<aside>` for supplementary Framer design tips or related component links.
Structure navigation using `<nav>` and ensure headings (`<h1>` to `<h6>`) are used logically to represent the hierarchy of your Framer design system or template documentation.
High Priority
Framer Content Chunking for RAG
Optimize your Framer site's content structure for efficient 'chunking' and retrieval by Retrieval-Augmented Generation (RAG) pipelines.
Break down lengthy Framer tutorials or component documentation into distinct, self-contained sections, ideally under 700 words each.
Ensure each section begins with a clear topic sentence or heading that summarizes its core content, reducing the need for external context lookup.
Replace ambiguous references (e.g., 'this feature', 'the component') with specific Framer element names or function descriptions (e.g., 'the Accordion component', 'the `useMediaQuery` hook').
