Block AI Crawlers vs WP Robots Txt

Protect Your Content from AI Scraping This plugin helps you prevent AI crawlers from using your content as training data for their products. By updating your site’s robots.txt, it blocks common AI crawlers and scrapers, aiming to protect your content from being used in the training of Large Language Models (LLMs). Features Blocks AI Crawlers Includes: OpenAI – Blocks crawlers used for ChatGPT Google – Blocks crawlers used by Google’s Gemini AI products Facebook / Meta – Used for Facebook’s AI training Anthropic AI – Blocks crawlers used by Anthropic Perplexity – Block crawlers used by Perplexity Applebot – Blocks crawlers used by Apple … and more! Experimental Meta Tags The plugin adds the “noai, noimageai” directive to your site’s meta tags, instructing AI bots not to use your content in their datasets. Please note that these tags are experimental and have not been standardized. Custom robots.txt Rules Have custom entries for your robots.txt file? You can now add them directly through the plugin! Usage After activation, the plugin will automatically update your robots.txt and add the necessary meta tags. No further configuration is required, but you can check the settings page for a full list of blocked crawlers. Limitations While this plugin aims to block specified crawlers, it cannot guarantee complete protection against all forms of scraping, as some bots may disregard robots.txt directives. Support For questions or support, please post on the forums or on GitHub.

WordPress, by default, includes a simple robots.txt file that’s dynamically generated from within the WP application. This is great, but how do you easily change the content? Enter WP Robots Txt, a plugin that adds an additional field to the “Reading” admin page where you can do just that. No manual coding or file editing required! Simply visit https://your-site.com/wp-admin/options-reading.php and you can control the contents of your https://your-site.com/robots.txt Changelog