Better Robots.txt – AI-Ready Crawl Control & Bot Governance vs LLMagnet – AI Visibility & AI SEO for Claude, ChatGPT & More

Better Robots.txt replaces the default WordPress robots.txt workflow with a smarter, structured version you can configure and preview before publishing. Instead of a blank textarea, you get a guided wizard with presets, plain-language explanations, and a final Review & Save step so you can inspect the generated robots.txt before it goes live. Built for beginners and advanced users alike, Better Robots.txt helps you control how search engines, AI crawlers, SEO tools, archive bots, bad bots, social preview bots, and other automated agents interact with your site. Trusted by thousands of WordPress sites, Better Robots.txt is designed for the AI era without resorting to hype, vague promises, or hidden rules. Better Robots.txt is available in Free, Pro, and Premium editions. The free plugin covers the guided workflow and essential crawl control features, while Pro and Premium unlock additional governance, protection, and AI-ready modules. Some screenshots on the plugin page show features from all three editions. A quick overview Why Better Robots.txt is different Most robots.txt plugins fall into one of three categories: Simple text editor Virtual robots.txt manager Single-purpose AI or policy add-on Better Robots.txt goes further. It gives you a complete, guided crawl control workflow so you can: Choose a preset that matches your goals Control major crawler categories without writing everything by hand Keep core WordPress protection rules visible and editable Clean up low-value crawl paths that waste crawl budget Generate a cleaner robots.txt output Preview the final result before saving What you can control Better Robots.txt helps you manage: Search engine visibility AI and LLM crawler behavior AI usage signals such as search, ai-input, and ai-train preferences SEO tool crawlers Bad bots and abusive crawlers Archive and Wayback access Feed crawlers and crawl traps WooCommerce crawl cleanup CSS, JavaScript, and image crawling rules Social media preview crawlers ads.txt and app-ads.txt allowance llms.txt generation Advanced directives such as crawl-delay and custom rules Final review before publishing Editions Better Robots.txt is available in three editions: Free – Includes the guided setup, the Essential preset, core crawl control features, and the final Review & Save workflow. Pro – Adds more advanced governance and protection modules, including additional AI, crawler, and cleanup controls. Premium – Unlocks the most restrictive and advanced protection options, including the Fortress preset and additional high-control modules. Some options shown in the interface are marked Free, Pro, or Premium so users can immediately understand which modules belong to each edition. Presets Setup starts with four modes: Essential – A clean, practical configuration for most websites that want a better robots.txt without complexity. AI-First – For publishers and content sites that want AI-ready governance without shutting down discovery. Fortress – For websites that want stronger protection against scraping, archive capture, and unnecessary crawl activity. Custom – For users who prefer to configure each module manually. For many sites, one preset plus a quick review is enough. Built for beginners and experts Beginners get: A guided setup instead of a raw robots.txt box Preset-based configuration Plain-language explanations for important choices A safer workflow with a final preview step Advanced users get: Editable core WordPress protection rules Fine-grained crawler controls by category WooCommerce-oriented cleanup options Consolidated output options Advanced directives and custom rules A final output they can inspect before publishing AI-ready, without hype Better Robots.txt includes features for modern AI-related crawl governance, including: AI crawler handling Optional llms.txt support AI usage signals for compliant systems Optional machine-readable governance signals for advanced use cases These features help you express how you want automated systems to use your content. However, Better Robots.txt does not claim to control AI by force. Like robots.txt itself, these signals are most useful with compliant systems and good-faith crawlers. What Better Robots.txt is Better Robots.txt is: A robots.txt governance plugin for WordPress A guided configuration workflow instead of a raw text editor A crawl control layer to reduce wasteful crawling A practical bridge between SEO, crawl hygiene, and AI-era policy signaling A way to keep your crawl policy clearer for humans and machines Technical reference for advanced users: Better Robots.txt also maintains a public GitHub repository with product definition, governance notes, and machine-readable artefacts. What Better Robots.txt is not Better Robots.txt is not: A firewall or Web Application Firewall (WAF) An anti-scraping enforcement engine A legal compliance engine A guarantee that every bot will obey your rules A replacement for server-level security or access control It helps you publish a clearer crawl policy. It does not replace infrastructure-level protection. Typical use cases Use Better Robots.txt if you want to: Clean up a weak or noisy default robots.txt Reduce crawl waste on WordPress or WooCommerce Keep major search engines allowed while restricting other bots Control whether archive bots can snapshot your site Publish AI usage preferences more clearly Keep social preview bots allowed while limiting scrapers Review the final file before making it live Key Features Guided step-by-step wizard Preset-based setup: Essential, AI-First, Fortress, Custom Search engine visibility controls AI and LLM crawler governance AI usage signals support SEO tool crawler controls Bad bot and abusive crawler options Archive and Wayback access controls Spam, feed, and crawl trap cleanup WooCommerce crawl cleanup options CSS, JavaScript, and image crawling rules Social media preview crawler controls ads.txt and app-ads.txt allowance Optional llms.txt generation Consolidated output option Core WordPress protection rules remain visible and editable Final Review & Save preview screen About the publisher Better Robots.txt is developed and maintained by Pagup, a digital readability firm based in Quebec, Canada. Pagup helps organizations become correctly understood by search engines, generative AI systems, and autonomous agents. The robots.txt file is the first surface that AI crawlers read when they discover a site. A well-structured robots.txt that references governance files such as llms.txt, ai-manifest.json, and interpretation policies helps AI systems understand your site faster and more accurately. Better Robots.txt is one component of a broader digital readability practice that includes semantic content architecture, AI governance and machine readability, and interpretive SEO. Part of the Pagup ecosystem pagup.com — Digital readability firm. Diagnostic, semantic architecture, AI governance. gautierdorval.com — Doctrine, canonical definitions, interpretive governance research. interpretive-governance.org — Formal versioned standard for interpretive governance. better-robots.com — Documentation and resources for Better Robots.txt.

Open playground Free Demo LLMagnet is the complete AI visibility solution for WordPress. As AI assistants like ChatGPT, Claude, and Perplexity become the new discovery layer, your site needs to be optimized for them-not just Google. With full support for both llms.txt and llms-full.txt formats, seamless integration with RankMath and Yoast SEO, and an intelligent recommendations system, LLMagnet ensures your content is discovered and properly represented by AI crawlers. Schema.org JSON-LD is built in as well: scan your site for existing structured data, review validation and recommendations, generate compliant markup, and publish it from the dashboard. Output is injected as a single JSON-LD block in the page head so it complements-rather than replaces-schema from your SEO plugin, giving search engines and AI assistants clearer signals about your organization, FAQs, and (on supported plans) WooCommerce products. Connect your site to any AI client and work directly from there. Link LLMagnet to Claude, ChatGPT, Cursor, or any MCP-compatible AI assistant and manage your AI visibility through natural conversation. Ask your AI to pull live reports, surface your weakest pages and products, build a concrete action plan to improve your visibility score, and help you execute it-without leaving your chat. It turns your favorite AI assistant into a hands-on AI-SEO co-pilot for your business. Powered by the WordPress Abilities API (6.9+) and the official MCP Adapter, with MCP/OAuth kept opt-in and disabled by default for security. Why You Need LLMagnet AI bots are already visiting your site. The question is: are they understanding your content? LLMagnet ensures your WordPress site speaks the language of AI crawlers through: llms.txt & llms-full.txt Protocol – Full support for AI-readable site maps with extended metadata Real-time Bot Analytics – See which AI models visit your content Smart Recommendations – Get actionable suggestions to improve AI visibility SEO Plugin Compatibility – Works seamlessly with RankMath and Yoast SEO WooCommerce Integration – Track AI-driven product discovery and revenue Schema.org JSON-LD – Scan, validate, generate, and publish structured data so machines can read your site reliably WordPress Abilities API support – Your AI assistant (Claude, ChatGPT, Cursor) can read your AI visibility score, bot analytics, and llms.txt status natively via the WordPress 6.9+ Abilities API and the official MCP Adapter Agent-Ready Site (Agentic Web) – WebMCP browser tools, markdown endpoints, well-known agent discovery files, and AI crawler controls that make your site easy for autonomous agents to read and act on Elementor Integration – Check AI visibility scores and add per-page schema right inside the Elementor editor Key Features Work Directly From Claude, ChatGPT & Cursor Connect your site to Claude, ChatGPT, Cursor, or any MCP-compatible AI client and manage AI visibility from your chat Ask your AI assistant to generate live reports on bot traffic, visibility scores, and llms.txt status Have your AI surface your weakest pages and products, then build a concrete action plan to improve them Create and execute a visibility improvement plan together-without leaving your AI assistant Built on the WordPress Abilities API (6.9+) and the official MCP Adapter; MCP/OAuth are opt-in and disabled by default for security Agent-Ready Site (Agentic Web) Make your site readable and actionable for autonomous AI agents and AI crawlers. Every feature ships off by default — enable only what you want agents to use. * WebMCP browser tools – Register search and content tools on navigator.modelContext so in-browser agents can interact with your site on every front-end page (built on the emerging W3C Web Model Context API) * Markdown endpoints – Serve any page as clean markdown at {permalink}.md, plus content negotiation that returns markdown on the normal URL when an agent sends Accept: text/markdown * Well-known agent discovery – Serve agent-card.json, agent-skills, mcp.json, security.txt, and the change-password redirect so agents can discover what your site offers * AI crawler controls in robots.txt – Managed User-agent groups for AI crawlers plus a Content-Signal block to declare how your content may be used * Link discovery headers – HTTP Link headers advertising llms.txt, the agent card, and schemamap.xml for automatic discovery * IndexNow – Ping IndexNow-backed engines automatically when content is published or updated for faster (re)indexing * Site hygiene for agents – Optional security headers (nosniff, Referrer-Policy, Permissions-Policy, HSTS, CSP report-only), Open Graph verify-or-fill (only when no SEO plugin owns the tags), and restored RSS feed tags * Live WebMCP status – See the real-time status of the W3C Web Model Context API in your own browser and the tools this site registers for agents AI-Ready Content Auto-generates /llms.txt and /llms-full.txt at your site root for AI crawler discovery Full support for llms-full.txt format with extended metadata Creates clean Markdown exports of your content in /llms-docs/ Supports posts, pages, products, and custom post types Smart auto-updates on content changes + daily sync Automatic robots.txt integration for proper AI crawler guidance Schema.org JSON-LD Scan pages for existing JSON-LD and catch invalid or missing structured data Actionable recommendations (for example Organization, LocalBusiness, FAQPage, and related types) Guided wizard to generate markup and publish a single JSON-LD block to the front end without overwriting schema from other plugins WooCommerce-aware recommendations and commerce schema options on supported plans Powerful Analytics Dashboard Track visits from ChatGPT, Claude, Perplexity, Gemini, Grok, and more Interactive charts showing bot activity over time Click tracking via UTM parameters to measure real engagement Site-wide Visibility Score with actionable insights Smart recommendations system for improving AI visibility Real-time suggestions for meeting AI crawler standards Pages & Posts Content Analysis AI-Readiness Score – Every page and blog post gets a score based on how well AI models can understand and cite it Content Quality Audit – Checks headings structure, meta descriptions, word count, and image ALT coverage Actionable Insights – Pinpoint exactly what’s holding each piece of content back from AI visibility Content Optimization Drawer – Fix issues and improve scores without leaving the dashboard Bulk Content Overview – Identify your weakest content at a glance and prioritize improvements Elementor Integration AI Visibility score in the editor – An “AI Visibility” button in the Elementor editor top bar (with a floating fallback) shows the current score for the page or product you’re editing Optimization drawer over the editor – Open the same Page/Product optimization drawer right inside Elementor and fix issues without switching screens Per-page JSON-LD schema field – Add custom structured data from Elementor’s document settings; it’s merged into LLMagnet’s single JSON-LD block so it never conflicts with other schema Zero overhead when unused – Loads only inside the Elementor editor and stays completely inactive on sites without Elementor WooCommerce Commerce Analytics (PRO) Product Visibility Scores – AI-readiness rating for each product AI Revenue Funnel – Track bot visits → add-to-cart → purchases Product of the Week – Highlight top AI-discovered products Content Quality Analysis – Descriptions, tags, categories, image ALT coverage Product Optimization Drawer – Edit and improve products without leaving the dashboard AI-Powered Commerce For WooCommerce stores, LLMagnet brings the future of AI commerce today. Aligned with Google’s emerging UCP (Universal Commerce Protocol) vision, we help your products get discovered, recommended, and purchased through AI assistants. Track how AI bots browse your catalog, measure the AI-to-revenue funnel, and optimize product content for maximum visibility in conversational commerce. Email Reports Automated weekly/monthly reports delivered to your inbox Track AI engagement trends over time Share insights with your team How It Works Install & Activate – LLMagnet starts working immediately with guided onboarding Configure Content – Choose which post types to include Auto-Generation – llms.txt, llms-full.txt and Markdown files are created and maintained Track AI Visits – See which bots discover your content Follow Recommendations – Get actionable suggestions to improve AI visibility Optimize Products – Improve visibility scores for better AI recommendations Measure Revenue – Connect AI traffic to actual conversions (WooCommerce) SEO Plugin Compatibility LLMagnet works seamlessly alongside your existing SEO setup: * Full RankMath Integration – Respects noindex settings and SEO configurations * Complete Yoast SEO Support – Works with all Yoast meta settings * Smart robots.txt Management – Automatically registers llms.txt and llms-full.txt * Elementor Editor Integration – AI visibility scores and per-page schema available directly inside the Elementor editor * No conflicts with your current SEO workflow AIO / GEO / AEO Benefits AIO (AI Optimization) – Structured content that AI models can parse accurately GEO (Generative Engine Optimization) – Better representation in AI-generated answers AEO (Answer Engine Optimization) – Increased chances of being cited as a source Data & Privacy LLMagnet is designed to be privacy-first. No data is sent to any external service unless you explicitly opt in. This section describes exactly what data can leave your site, when, and how to control it. Optional telemetry (opt-in, off by default) During onboarding you are asked whether to share product data with us. If you choose “Continue without sharing” (or never answer), nothing is sent to the services below. Your choice is stored in the llmagnet_telemetry_consent option and can be changed at any time (re-run the onboarding wizard, update the option, or POST to the llm-analytics/v1/privacy/settings REST endpoint as an administrator). Brevo (https://www.brevo.com – email/CRM service). Only after opt-in, the plugin syncs the site owner as a contact in our Brevo account and sends a small number of lifecycle emails (a getting-started reminder, a notification when the first AI bot visit is detected, and a trial reminder). Data sent: WordPress admin email address, site name, site URL/domain, plugin version, onboarding completion status, the site’s AI visibility score, and free-trial status. Sync happens shortly after opt-in and again when the site identity or plugin version changes. See Brevo’s privacy policy: https://www.brevo.com/legal/privacypolicy/ Mixpanel (https://mixpanel.com – product analytics). Only after opt-in, the plugin tracks how the wp-admin dashboard of this plugin is used (page views, button clicks such as “Generate Now”, settings saves, onboarding progress, errors). Events run only on LLMagnet admin pages – never in your site visitors’ browsers – and are identified by your site domain. Properties include: admin email, site URL, plugin and WordPress versions, site language, WooCommerce active yes/no, and plan/trial status. Data is sent to Mixpanel’s EU endpoint (api-eu.mixpanel.com). See Mixpanel’s privacy policy: https://mixpanel.com/legal/privacy-policy/ Visitor analytics stay on your server AI bot visit logs (bot name, user-agent string, page path, timestamp) are stored locally in the wp_llm_bot_visits and wp_llm_bot_page_clicks database tables and are never sent to any external service. These tables log AI crawlers – not human visitors. Data retention: logs older than the configured retention window are pruned automatically by a daily background task. The window is controlled by the llmagnet_data_retention_days option (allowed values: 30, 90, 180, 365, or 0 to keep forever; default 365). Attribution cookie (for your cookie policy) To attribute conversions to AI platforms, the plugin sets one first-party cookie on site visitors – but only when a visitor lands with a utm_source URL parameter matching a known AI platform (ChatGPT/OpenAI, Gemini, Copilot, Perplexity, Claude, DeepSeek, Grok, Bing, Llama/Meta AI, Mistral). Regular visitors never receive it. Name: llmagnet_llm_attribution Purpose: ties an AI-referred visit to a later purchase (WooCommerce AI revenue funnel) Contents: a random session ID (UUID), the detected AI platform name, the landing page path, utm_medium, utm_campaign, and a first-touch timestamp – no IP address and no personal identifiers Lifetime: 7 days (filterable via llmagnet_attribution_ttl_days) Flags: HttpOnly, SameSite=Lax, Secure on HTTPS sites A matching session row is stored locally in the wp_llm_attribution_sessions table; unconverted sessions are deleted automatically after twice the cookie lifetime. Attribution tracking can be disabled entirely by setting the llmagnet_attribution_tracking option to 0 (also available via the llm-analytics/v1/privacy/settings REST endpoint) or via the llmagnet_attribution_tracking_enabled filter. WordPress privacy tools LLMagnet registers a personal data exporter and eraser (Tools → Export/Erase Personal Data) for attribution session data. Sessions are matched to a person through the billing email of their WooCommerce orders; erasure deletes the session rows and detaches the session from order analytics.