Better Robots.txt – AI-Ready Crawl Control & Bot Governance vs UpdraftPlus

Better Robots.txt replaces the default WordPress robots.txt workflow with a smarter, structured version you can configure and preview before publishing. Instead of a blank textarea, you get a guided wizard with presets, plain-language explanations, and a final Review & Save step so you can inspect the generated robots.txt before it goes live. Built for beginners and advanced users alike, Better Robots.txt helps you control how search engines, AI crawlers, SEO tools, archive bots, bad bots, social preview bots, and other automated agents interact with your site. Trusted by thousands of WordPress sites, Better Robots.txt is designed for the AI era without resorting to hype, vague promises, or hidden rules. Better Robots.txt is available in Free, Pro, and Premium editions. The free plugin covers the guided workflow and essential crawl control features, while Pro and Premium unlock additional governance, protection, and AI-ready modules. Some screenshots on the plugin page show features from all three editions. A quick overview Why Better Robots.txt is different Most robots.txt plugins fall into one of three categories: Simple text editor Virtual robots.txt manager Single-purpose AI or policy add-on Better Robots.txt goes further. It gives you a complete, guided crawl control workflow so you can: Choose a preset that matches your goals Control major crawler categories without writing everything by hand Keep core WordPress protection rules visible and editable Clean up low-value crawl paths that waste crawl budget Generate a cleaner robots.txt output Preview the final result before saving What you can control Better Robots.txt helps you manage: Search engine visibility AI and LLM crawler behavior AI usage signals such as search, ai-input, and ai-train preferences SEO tool crawlers Bad bots and abusive crawlers Archive and Wayback access Feed crawlers and crawl traps WooCommerce crawl cleanup CSS, JavaScript, and image crawling rules Social media preview crawlers ads.txt and app-ads.txt allowance llms.txt generation Advanced directives such as crawl-delay and custom rules Final review before publishing Editions Better Robots.txt is available in three editions: Free – Includes the guided setup, the Essential preset, core crawl control features, and the final Review & Save workflow. Pro – Adds more advanced governance and protection modules, including additional AI, crawler, and cleanup controls. Premium – Unlocks the most restrictive and advanced protection options, including the Fortress preset and additional high-control modules. Some options shown in the interface are marked Free, Pro, or Premium so users can immediately understand which modules belong to each edition. AI crawler controls by edition The Free edition lets site owners choose a basic global policy for AI search and discovery crawlers: allow or block. AI training crawlers remain separated from AI search crawlers, so a site can block training use while still allowing AI-powered search and answer discovery. The Pro and Premium editions add advanced controls, including per-bot AI crawler rules, custom user agents, AI usage signals, llms.txt, and additional machine-readable governance files. Better Robots.txt separates AI-related agents into three practical families: AI training crawlers, such as GPTBot, Google-Extended, and ClaudeBot. AI search and discovery crawlers, such as OAI-SearchBot, PerplexityBot, and Claude-SearchBot. User-action fetchers, such as ChatGPT-User, Perplexity-User, and Claude-User. User-action fetchers are triggered by user requests. Depending on the provider, robots.txt may not apply to those fetches in the same way as it applies to automated crawlers. For stronger enforcement, combine robots.txt with server logs, WAF rules, provider verification, or IP-based controls where available. Presets Setup starts with four modes: Essential – A clean, practical configuration for most websites that want a better robots.txt without complexity. AI-First – For publishers and content sites that want AI-ready governance without shutting down discovery. Fortress – For websites that want stronger protection against scraping, archive capture, and unnecessary crawl activity. Custom – For users who prefer to configure each module manually. For many sites, one preset plus a quick review is enough. Built for beginners and experts Beginners get: A guided setup instead of a raw robots.txt box Preset-based configuration Plain-language explanations for important choices A safer workflow with a final preview step Advanced users get: Editable core WordPress protection rules Fine-grained crawler controls by category WooCommerce-oriented cleanup options Consolidated output options Advanced directives and custom rules A final output they can inspect before publishing AI-ready, without hype Better Robots.txt includes features for modern AI-related crawl governance, including: AI crawler handling Optional llms.txt support AI usage signals for compliant systems Optional machine-readable governance signals for advanced use cases These features help you express how you want automated systems to use your content. However, Better Robots.txt does not claim to control AI by force. Like robots.txt itself, these signals are most useful with compliant systems and good-faith crawlers. What Better Robots.txt is Better Robots.txt is: A robots.txt governance plugin for WordPress A guided configuration workflow instead of a raw text editor A crawl control layer to reduce wasteful crawling A practical bridge between SEO, crawl hygiene, and AI-era policy signaling A way to keep your crawl policy clearer for humans and machines Technical reference for advanced users: Better Robots.txt also maintains a public GitHub repository with product definition, governance notes, and machine-readable artefacts. What Better Robots.txt is not Better Robots.txt is not: A firewall or Web Application Firewall (WAF) An anti-scraping enforcement engine A legal compliance engine A guarantee that every bot will obey your rules A replacement for server-level security or access control It helps you publish a clearer crawl policy. It does not replace infrastructure-level protection. Typical use cases Use Better Robots.txt if you want to: Clean up a weak or noisy default robots.txt Reduce crawl waste on WordPress or WooCommerce Keep major search engines allowed while restricting other bots Control whether archive bots can snapshot your site Publish AI usage preferences more clearly Keep social preview bots allowed while limiting scrapers Review the final file before making it live Key Features Guided step-by-step wizard Preset-based setup: Essential, AI-First, Fortress, Custom Search engine visibility controls AI and LLM crawler governance AI usage signals support SEO tool crawler controls Bad bot and abusive crawler options Archive and Wayback access controls Spam, feed, and crawl trap cleanup WooCommerce crawl cleanup options CSS, JavaScript, and image crawling rules Social media preview crawler controls ads.txt and app-ads.txt allowance Optional llms.txt generation Consolidated output option Core WordPress protection rules remain visible and editable Final Review & Save preview screen About the publisher Better Robots.txt is developed and maintained by Pagup, a digital readability firm based in Quebec, Canada. Pagup helps organizations become correctly understood by search engines, generative AI systems, and autonomous agents. The robots.txt file is the first surface that AI crawlers read when they discover a site. A well-structured robots.txt that references governance files such as llms.txt, ai-manifest.json, and interpretation policies helps AI systems understand your site faster and more accurately. Better Robots.txt is one component of a broader digital readability practice that includes semantic content architecture, AI governance and machine readability, and interpretive SEO. Part of the Pagup ecosystem pagup.com — Digital readability firm. Diagnostic, semantic architecture, AI governance. gautierdorval.com — Doctrine, canonical definitions, interpretive governance research. interpretive-governance.org — Formal versioned standard for interpretive governance. better-robots.com — Documentation and resources for Better Robots.txt.

The UpdraftPlus Backup & Migration Plugin is trusted by the WordPress community to backup, restore and migrate their WordPress websites. UpdraftPlus is actively installed on more than 3 million websites around the world. Backup with UpdraftPlus UpdraftPlus is the world’s highest ranking and most popular scheduled backup and migration plugin. Backup to your preferred storage location and restore in just three clicks. Backup to Dropbox, Google Drive, Amazon S3 (or compatible), Rackspace Cloud, FTP, DreamObjects, Openstack Swift or email. The paid version also allows you to backup to Microsoft OneDrive, Microsoft Azure, Google Cloud, Backblaze B2, SFTP, SCP, pCloud, WebDAV or UpdraftVault, our integrated storage option for UpdraftPlus. Backup manually or schedule to run every 2, 4, 8 or 12 hours, daily, weekly, monthly or fortnightly. Restore with UpdraftPlus Your WordPress website is vulnerable. Your site may get hacked. Something could go wrong with an update or your server could crash. You might need to reverse a change made or a mistake caused by human error. Whatever the reason, you can restore your WordPress website in just a few clicks with UpdraftPlus. Choose which components to restore (e.g. plugins, themes, database etc.), click again to restore, then click again to return to the main screen. Migrate with UpdraftPlus It’s easy to migrate your WordPress website to another web host, server or domain with this free version of UpdraftPlus. To migrate, simply download your database, plugins, themes etc. from the source site, then upload them into your destination site. When migrating, a built-in search-and-replace engine identifies old strings and replaces these to reflect the new location. UpdraftPlus saves you time and reduces the risk of broken links or missing files associated with manual migrations. Why UpdraftPlus? Writing a reliable backup and migration plugin that ‘just works’ consistently across millions of different WordPress deployments is hard. UpdraftPlus is relied upon to do exactly that. We’re trusted and actively deployed on more websites around the world than any other WordPress backup and migration plugin. UpdraftPlus: Backs up, migrates and restores Offers a huge number of remote storage locations Allows you to schedule backups, so you can ‘set and forget’ Is comprehensive and easy to use Is proven to work on over 3 million sites. UpdraftPlus Premium The free version of UpdraftPlus will back up and migrate your website perfectly well. However if you need more features and options you can purchase our Premium version. Backup and Migrate with UpdraftPlus Premium, and: Get automatic backups before updates. Restore the very latest version if a WordPress or plugin update breaks your site. Backup Incrementally. Changes are added to the master, saving server resources vs repeatedly backing up in full. Get more remote storage options including Microsoft OneDrive, SFTP, Microsoft Azure, WebDAV, Google Cloud, SCP, Backblaze and pCloud. Get 1GB of integrated storage with UpdraftVault for 99.999% reliability, redundancy and scalability. Get more control over how many backups are stored in a specified interval. Restore from other backup plugins including BackWPup, BackupWordPress, Simple Backups and more. Run backups at set times e.g. during periods of low traffic. Backup to more than one location for added protection. Get detailed reports. Includes cryptographic checksums so you can verify the integrity of the backup files and more. Get Premium migration. Migrating to the destination site is more direct and can be done from the source site. Get Multisite / multi-network compatibility. Backup non-WP files and databases e.g. tables belonging to your e-commerce store or customisations to WordPress core. Get database encryption. Manage your backups and migrations from the WP-CLI. Get Premium Support. Find out more on our comparison page. UpdraftPlus Premium is available for purchase here. Managing multiple websites? UpdraftCentral is a powerful remote control dashboard for WordPress that allows you to manage your backups, as well as updates, users, pages, posts, plugins and themes from one central location. Choose between: UpdraftCentral (free, self-hosted) UpdraftCentral Premium (paid-for, self-hosted) UpdraftCentral Cloud (paid-for, fully-hosted) Optimize your sites centrally by combining the power of WP-Optimize and UpdraftCentral or manage your backups centrally by combining the power of UpdraftPlus with UpdraftCentral. Need to create a temporary clone of your site? UpdraftClone makes it quick and easy to create a temporary sandbox for changes you want to try out. Simply select the versions of WordPress and PHP that you want, and we’ll do the rest. More about UpdraftClone The full suite of plugins from Team Updraft and friends UpdraftPlus. Backup, migrate and restore. Rated 5* on the plugin directory and trusted by more than 3 million WordPress website owner WP-Optimize. Clean the database, compress images and caching. Optimize your WordPress website. 5* user rating. 1 million + active installs. All-In-One Security (AIOS). Secure your WordPress website. Comprehensive, feature rich and easy to use. 5* user rating and 1 million + active installs. WP Overnight. Quality Extensions for your WooCommerce store. 5* rated invoicing, order and product management, customer management solutions and more. Easy Updates Manager. Take control of updates. Lots of features for free, and a Premium version with even more – over 300,000 users. Internal Link Juicer.Boost your SEO. Automate internal links within your WordPress website. Save time and rank higher in the search engines. For other useful, free plugins see our lead developer’s profile, here. Are you multilingual? Can you translate? Do you want to help people that speak your language to backup, migrate and restore their WordPress websites? The UpdraftPlus backup, migration and restoration plugin is ready and waiting. The translation process is easy, and web-based – please go here for instructions: https://updraftplus.com/translate/. Or, if you’re already an expert WordPress translator, just pick out the .pot file from the wp-content/plugins/updraftplus/languages/ directory – if you scan for translatable strings manually, then you need to get these functions: _x(), __(), _e(), _ex(), log_e(). Many thanks to our existing translators. License Copyright 2011-24 David Anderson This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA Non-English translators are provided by volunteers, and wordpress.org does not give plugin developers control over their translations; thus relying on them is at your own risk; UpdraftPlus cannot guarantee that translations from the original English are accurate. We recognise and thank those mentioned at https://updraftplus.com/acknowledgements/ for code and/or libraries used and/or modified under the terms of their open source licences.