WEO LLMS & AI Crawler Guide
Prepare your WordPress site for the AI Search Era. As search engines evolve into Answer Engines (like ChatGPT Search, Perplexity, and Google Gemini), they rely less on HTML and more on structured, raw text data. The llms.txt standard is the new “robots.txt” for AI—a file specifically designed to tell Large Language Models (LLMs) what your site is about and where to find your best content. WEO LLMS & AI Crawler Guide automatically generates a perfect, dynamic llms.txt file for your site. It creates a categorized “Resume” of your website that helps AI bots understand your structure instantly. 🚀 Key Features Categorized Content Output: Automatically groups your content by type (e.g., ## Post, ## Page, ## Products) so AI understands your site structure. Automated Sitemap Detection: Automatically detects and lists your sitemap.xml at the top of the file for deeper crawling. Markdown Optimization: Converts your content titles, links, and excerpts into clean Markdown (- [Title](Link) - Excerpt) optimized for LLM token limits. Robots.txt Integration: Automatically adds a link to your llms.txt inside your site’s robots.txt file so crawlers can find it. Custom Control: Choose exactly which Post Types (Pages, Posts, Products, Custom Types) to include. Custom Intro/Outro: Add specific instructions for AI bots (e.g., “This site is the authority on Technical SEO…”). Performance Caching: Built-in caching ensures the file loads instantly without slowing down your server. 💡 Why do you need this? If you want your content to appear in AI answers (ChatGPT, Claude, Perplexity), you need to make it easy for them to read. A standard HTML page is full of noise (menus, sidebars, popups). An llms.txt file provides a clean, noise-free map of your content. This plugin handles the technical heavy lifting, ensuring your site is “AI Ready” in seconds.
Zicy AI Access Monitor
Zicy AI Access Monitor helps site owners understand when known AI crawlers and AI assistant retrieval agents access their WordPress content. The plugin logs self-identifying AI-related User-Agents, summarizes raw visits into daily totals, and provides an admin dashboard for reviewing AI access patterns across your site. It is designed for AI visibility, AI search optimization, and content monitoring workflows. What it tracks: Search crawlers — AI systems that may crawl content for search or answer retrieval workflows, such as OAI-SearchBot and PerplexityBot Training crawlers — AI-related crawlers that may access content for model training or dataset-building workflows, such as GPTBot, ClaudeBot, CCBot, Amazonbot, Bytespider, CohereBot, Mistral-ai, and Meta-WebIndexer User proxies — AI assistant fetchers that retrieve pages when responding to user requests, such as ChatGPT-User, Claude-User, Perplexity-User, Gemini-Deep-Research, Google-Agent, and BingPreview AI control agents — AI-related User-Agents such as Google-Extended that may represent AI data-use control behavior rather than ordinary search crawling How it works: Matching visits are recorded as temporary raw logs. URLs are normalized to reduce duplicates caused by common tracking parameters. IP addresses are anonymized by default before being stored. Each night at 02:15 site time, raw logs older than today are summarized into daily totals. Raw logs that have been rolled up are deleted. Daily summary records are retained so you can review historical AI access activity. In your WordPress admin dashboard: View daily summaries of known AI bot and AI assistant access See today’s raw logs before the next rollup Filter by date, bot, type, or signature Review top pages accessed by AI assistant user proxies Compare week-over-week activity for top pages See vendor diversity indicators for accessed pages Export daily summary data as CSV Receive weekly email summaries with top accessed pages Generate copy-ready AI Tracking Briefs for selected pages Why it matters: See which known AI systems are accessing your content Identify which pages AI assistants retrieve most often Support AI search optimization and AI visibility reporting Understand differences between crawler, training, control, and user-triggered AI access Build structured page briefs that can be used in AI visibility workflows Bot allowlist includes: OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User Anthropic: ClaudeBot, Claude-User, Claude-SearchBot Perplexity: PerplexityBot, Perplexity-User Google: Google-Extended, Gemini-Deep-Research, Google-Agent Microsoft: BingPreview Amazon: Amazonbot Common Crawl: CCBot ByteDance: Bytespider Cohere: CohereBot Mistral: Mistral-ai Meta: Meta-WebIndexer Developers can extend or modify the allowlist with the gp_aibal_ai_bots filter. Important detection note: This plugin detects only self-identifying User-Agents. It does not guarantee that every visit is from the claimed company or system, because User-Agent headers can be spoofed. The plugin should be used as a visibility and reporting aid, not as a security or access-control tool. Privacy Zicy AI Access Monitor is designed to keep raw request-level data temporary. The plugin stores matching AI-related access events in your WordPress database, anonymizes IP addresses by default, rolls raw logs into daily summaries, and deletes rolled-up raw logs. Daily summaries are retained for reporting. The plugin does not send logged access data to Zicy or any third-party service. Site owners are responsible for ensuring that their use of access logging complies with applicable privacy laws and their own privacy policy.
