Bisteinoff SEO Robots.txt vs noindex SEO

Have you encountered an obstacle while creating and editing robots.txt file on your website? Bisteinoff SEO Robots.txt is an easy-to-use plugin that helps you generate and configure a correct robots.txt file, which is essential for search engine optimization (SEO). This file defines crawling rules for search engine bots such as Google, Bing, Yahoo!, Yandex, and others. The plugin works perfectly both if the file robots.txt has never been created or if it already exists. Once installed the plugin makes an optimized robots.txt file that includes special rules common for WordPress websites. After that you can proceed further customization specific for your own website if needed. If the plugin detects one or several Sitemap XML files it will include them into robots.txt file. No FTP access, manual coding or file editing is required that makes managing settings easy and convenient! Key Features Automatic generation of optimized robots.txt with WordPress-specific rules Special rules for Google and Yandex search engines Custom rules support for any search engine bot Automatic sitemap detection and inclusion WooCommerce compatibility with specific rules Multisite support Easy-to-use admin interface Modern PHP architecture with namespaces for conflict-free operation

Fine-grained control over how search engines index and display your WordPress content. Apply 5 independent robots directives to 25 different page contexts with flexible implementation methods. 5 Robots Directives: noindex: Prevent search engines from indexing the page nofollow: Prevent search engines from following links on the page noarchive: Prevent search engines from showing cached versions nosnippet: Prevent search engines from showing text snippets in results noimageindex: Prevent search engines from indexing images on the page Implementation Methods: HTML Meta Tags: Traditional method, easy to verify in page source (default) HTTP Headers: More robust, works with all content types including PDFs and images Both: Maximum compatibility for all scenarios Control Levels: Global Settings: Apply directives to 25 different page contexts (posts, pages, archives, etc.) Granular Control (Optional): Override global settings for individual posts, pages, and custom post types via meta boxes in the editor Perfect for: Blocking indexing of attachment pages while allowing link following Preventing duplicate content issues with flexible directive combinations Controlling archive page indexing with granular control Managing pagination SEO with independent settings Protecting private content from search engine caching Preventing snippet display while still indexing content Main pages Front Page: Block the indexing of the site’s front page. Home: Block the indexing of the site’s home page. Pages and Posts Page: Block the indexing of the site’s pages. Privacy Policy: Block the indexing of the site’s privacy policy page. Single: Block the indexing of a post on the site. Singular: Block the indexing of a post or a page of the site. Taxonomies Category: Block the indexing of the site categories. The lists where the posts appear. Tag: Block the indexing of the site’s tags. The lists where the posts appear. Dates Date: Block the indexing when any date-based archive page (i.e. a monthly, yearly, daily or time-based archive) of the site. The lists where the posts appear. Day: Block the indexing when a daily archive of the site. The lists where the posts appear. Month: Block the indexing when a monthly archive of the site. The lists where the posts appear. Time: Block the indexing when an hourly, “minutely”, or “secondly” archive of the site. The lists where the posts appear. Year: Block the indexing when a yearly archive of the site. The lists where the posts appear. Archives Archive: Block the indexing of any type of Archive page. Category, Tag, Author and Date based pages are all types of Archives. The lists where the posts appear. Author: Block the indexing of the author’s page, where the author’s publications appear. Post Type Archive: Block the indexing of any post type page. Pagination Pagination: Block the indexing of the pagination, i.e. all pages other than the main page of an archive. Search Search: Block the indexing of the internal search result pages. Attachments Attachment: Block the indexing of an attachment document to a post or page. An attachment is an image or other file uploaded through the post editor’s upload utility. Attachments can be displayed on their own “page” or template. This will not cause the indexing of the image or file to be blocked. Previews Customize Preview: Block the indexing when a content is being displayed in customize mode. Preview: Block the indexing when a single post is being displayed in draft mode. Error Page Error 404: This will cause an error page to be blocked from being indexed. As it is an error page, it should not be indexed per se, but just in case. Important note: if you have any doubt about any of the following items it is best not to activate the option as you could lose results in the search engines. Compatibility WordPress: 5.7 – 7.1 PHP: 7.2 – 8.5 Security This plugin adheres to the following security measures and review protocols for each version: WordPress Plugin Handbook WordPress Plugin Security WordPress APIs Security WordPress Coding Standards Plugin Check (PCP) Privacy This plugin does not collect any information about your site, your identity, the plugins, themes or content the site has. Vulnerabilities No vulnerabilities have been published up to version 2.0.1. Version 2.0.0 includes proactive security hardening based on comprehensive security audit (see docs/SECURITY-2026-01-20.md). Found a security vulnerability? Please report it to us privately at the noindex SEO GitHub repository.