High Priority
Implement 'robots.txt' for AI Agents
Establish a machine-readable directive file for AI crawlers, outlining your store's structure and content accessibility for training and information retrieval.
Create or modify your '/robots.txt' file to include directives for AI bots (e.g., User-agent: GPTBot, User-agent: CCBot).
Include explicit 'Allow' and 'Disallow' rules for key WooCommerce sections like product pages, category archives, cart, and checkout to guide AI crawlers.
Add a brief, plain-text description of your store's primary product categories or services to assist AI in understanding your business context.
Configure your WooCommerce users crawler protocols effortlessly.
Join 2,000+ teams scaling with AI.
High Priority
Selective Indexing for AI Models
Fine-tune which parts of your WooCommerce store are accessible for ingestion by specific AI crawlers, such as those used for knowledge base creation or product data aggregation.
Define specific user-agent rules in 'robots.txt' for known AI crawlers (e.g., User-agent: PerplexityBot, Allow: /products/, Disallow: /checkout/).
Utilize meta robots tags on pages (e.g., <meta name='robots' content='noai'>) to prevent AI ingestion of sensitive or non-product-related content.
Monitor server logs for AI crawler activity to confirm they are adhering to your directives and accessing intended store sections.
Medium Priority
Semantic HTML for Product Data Hierarchy
Leverage HTML5 semantic elements to clearly define the structure and relationships of product information, aiding AI scrapers in accurate data extraction.
Enclose individual product listings within `<article>` tags to signify distinct content units.
Use `<section>` tags with descriptive labels (e.g., `<section aria-label='Product Specifications'>`) for different product attribute groups.
Ensure product tables (e.g., for variations, specifications) use `<thead>`, `<tbody>`, and `<th>` tags correctly for structured data parsing.
High Priority
Optimized Content Chunks for RAG
Structure product descriptions, FAQs, and support articles to be easily segmented ('chunked') and utilized by Retrieval-Augmented Generation (RAG) AI models.
Keep distinct product features or support topics within self-contained blocks of approximately 500 words.
Reiterate the core product name or issue in summaries of sections to provide clear context for RAG pipelines.
Eliminate ambiguous pronouns; explicitly state product names, feature names, or error codes instead of using 'it' or 'they'.
