EDH Bad Bots vs Magic robots.txt

EDH Bad Bots is an intelligent bot detection and blocking system that protects your WordPress site from unwanted crawlers and malicious bots. Unlike traditional blocking methods that rely on user agent strings (which can be easily spoofed), this plugin uses a honeypot technique to identify and block bots that don’t respect your site’s robots.txt directives. Key Features Automatic Bot Detection: Identifies bad bots using a hidden trap URL technique Smart Blocking System: Blocks misbehaving bots with configurable duration (default 30 days) Advanced DNS Resolution: PTR record lookups with DNS over HTTPS (DoH) support for hostname identification Dual-Level Blocking: Server-level .htaccess blocking AND PHP-level blocking for maximum effectiveness Configurable Blocking Methods: Choose between .htaccess blocking (Apache) or PHP-only blocking (Nginx compatible) IP Whitelist Management: Protect trusted IPs from ever being blocked Enhanced Admin Interface: Clean dashboard with hostname display, manual hostname updates, and debug tools Background Processing: Automated hostname resolution via WordPress cron jobs Legitimate Crawler Protection: Synchronous FCrDNS verification prevents Googlebot and similar crawlers from being blocked, even if robots.txt is misconfigured robots.txt Health Check: Daily cron monitors for a physical robots.txt missing the trap Disallow rule and alerts the admin Database Optimization: Automatic cleanup of expired blocks to maintain performance Security-First Design: All forms include proper nonce verification and user capability checks How It Works The plugin implements a sophisticated honeypot system: Trap URL Generation: Creates a unique, hidden URL specific to your domain Robots.txt Integration: Automatically adds a Disallow rule for the trap URL Hidden Link Placement: Places an invisible link to the trap URL in your site’s footer Bot Detection: When bad bots ignore robots.txt and follow the hidden link, they’re identified Automatic Blocking: Detected bot IPs are blocked with configurable duration and immediate effect Hostname Resolution: PTR record lookups identify the hostname/organization behind blocked IPs Legitimate Crawler Guard: If a request to the trap URL carries a Googlebot, Bingbot, or similar User-Agent, FCrDNS is run synchronously before any block decision. Verified crawlers are whitelisted on the spot; spoofed UAs are blocked normally. robots.txt Health Check: A daily cron fetches and validates the site’s robots.txt. If a physical file is missing the Disallow rule, an admin notice is shown until the issue is resolved. Configuration Admin Dashboard Access the plugin dashboard at Tools > Bad Bots in your WordPress admin: Whitelisted IPs Tab Add IP addresses that should never be blocked Remove IPs from the whitelist View all currently whitelisted addresses with timestamps Blocked Bots Tab View all currently blocked IP addresses with hostnames See when each IP was blocked and when the block expires Manually update missing hostnames for better identification Force refresh all hostnames to clear cache and re-resolve Debug hostname resolution issues (when WP_DEBUG is enabled) Manually unblock IPs if needed Options Tab .htaccess Blocking: Enable/disable server-level IP blocking via .htaccess file Block Duration: Configure how many days to block detected bots Configure blocking method based on your server setup (Apache vs Nginx) Server-level blocking bypasses caching for immediate effect Help Tab Detailed explanation of how the plugin works Best practices for managing IPs Information about .htaccess blocking options Unique trap URL for caching plugin exclusion Requirements WordPress 6.2 or higher PHP 7.4 or higher MySQL 5.6 or higher Apache server (for .htaccess blocking) or Nginx (PHP-only blocking) Writable .htaccess file (if using Apache server-level blocking) Technical Details Database Tables The plugin creates two custom database tables: wp_edhbb_blocked_bots: Stores blocked IP addresses with expiration dates and hostnames wp_edhbb_whitelisted_ips: Stores permanently whitelisted IP addresses DNS Resolution System The plugin includes an advanced DNS lookup system: DNS over HTTPS (DoH) Support Primary providers: Cloudflare DNS, Google DNS Secure queries: HTTPS-encrypted DNS requests for enhanced privacy Fallback system: Automatic fallback to traditional DNS methods PTR Record Lookups Reverse DNS: Converts IP addresses to hostnames for better identification IPv4 and IPv6 support: Full support for both IP versions Caching: Results cached for 1 hour to improve performance Background processing: Automated hostname resolution via WordPress cron Blocking Methods The plugin offers two blocking approaches: 1. Server-Level Blocking (.htaccess) Default method for Apache servers Blocks IPs at the server level before WordPress loads Bypasses caching plugins for immediate effect More efficient and faster blocking Automatically manages .htaccess file with unique markers Safe cleanup on plugin deactivation 2. PHP-Level Blocking Alternative method for Nginx or when .htaccess is unavailable Blocks IPs during WordPress initialization Compatible with all web servers May be affected by caching plugins No server configuration files modified Security Features Nonce Verification: All forms use WordPress nonces for CSRF protection Capability Checks: Only users with manage_options capability can access admin features Input Sanitization: All user inputs are properly sanitized and validated SQL Injection Protection: All database queries use prepared statements Safe .htaccess Management: Uses unique markers and automatic cleanup Performance Optimization Automatic Cleanup: Expired blocks are automatically removed from the database Efficient Queries: Database operations are optimized for minimal performance impact Smart Loading: Admin assets only load on the plugin’s admin page Server-Level Blocking: .htaccess blocking prevents blocked requests from reaching PHP Whitelist Filtering: Whitelisted IPs are excluded from .htaccess rules automatically DNS Caching: Hostname lookups cached to reduce DNS query overhead Background Processing: Hostname resolution runs in background to avoid delays API Hooks Actions plugins_loaded: Plugin initialization init: Early request blocking check template_redirect: Bot trap detection wp_footer: Hidden link injection admin_menu: Admin page registration admin_notices: robots.txt misconfiguration warning edhbb_update_hostnames_cron: Background hostname resolution (hourly) edhbb_check_robots_txt_cron: robots.txt Disallow validation (daily) Filters robots_txt: Adds disallow rule to robots.txt edhbb_trusted_crawler_domains: Customise the FCrDNS hostname suffixes used to identify legitimate crawlers (e.g. .googlebot.com) edhbb_trusted_crawler_ua_patterns: Customise the User-Agent tokens that trigger a synchronous FCrDNS check on trap hits (e.g. googlebot, bingbot) File Structure ` edh-bad-bots/ ├── admin/ │ └── views/ │ └── admin-display.php # Admin interface HTML ├── assets/ │ ├── css/ │ │ └── admin-style.css # Admin page styling │ └── js/ │ └── admin-script.js # Admin page JavaScript ├── includes/ │ ├── class-edhbb-admin.php # Admin functionality │ ├── class-edhbb-blocker.php # Bot detection and blocking │ ├── class-edhbb-database.php # Database operations │ └── class-edhbb-dnslookup.php # DNS/PTR lookup system ├── edh-bad-bots.php # Main plugin file ├── LICENSE └── readme.txt ` Contributing Contributions are welcome! Please feel free to submit a Pull Request. Development Setup Clone the repository to your WordPress plugins directory Ensure you have a WordPress development environment running Activate the plugin and test your changes License This project is licensed under the GPL v3 or later. Author EncodeDotHost – Website: https://encode.host – GitHub: @EncodeDotHost Support For support, please visit the support forum: https://wordpress.org/support/plugin/edh-bad-bots/

In a nutshell: You don’t want robots wasting your resources and slowing down your site if they don’t give you some kind of direct or indirect benefit. An optimal configuration will maximize your SEO results and reduce your operating costs. You’ll need less than a minute to set your site characteristics and get a customized robots.txt configuration installed, matching your needs and easily updated without headaches. Let us to apply our experience in maintaining and optimizing sites. A clear knowledge of what we want to do and why, moved us to change the decission method from human created robots.txt to an automated system, things are simpler as they seems. Here comes to help this plugin, doing all the magic for you. Some of your site specs will be automatically detected to reduce the manual config needs to the minimum. A few simple questions about your site will be enough to vast majority of sites. Update procedure Any change required to the robots.txt is applied in real time when operating in virtual file mode. If you are working in real file mode, after updating the plugin version you should click the Save Changes button in the configuration screen to regenerate the robots.txt file.