Multipart robots.txt editor vs ReCrawler

This plugin needs more documentation! You can edit your robots.txt and add remote content to it. E.g. you have several sites and want to use a centralized robots.txt. Features Include or exclude WordPress’ own robots.txt (core function) Include or exclude plugins – e.g. sitemap plugins – output to robots.txt (filter output) Include or exclude a remote text file (the common part) Include or exclude custom records from the settings page (the site specific part) Where is robot.txt? WordPress handles robots.txt as a virtual URL – just the same way as posts and pages. So when you browse to https://example.com/robots.txt WordPress generates robots.txt on the fly. TODO add more description here add a video too add an admin notice for subdir installs (robots.txt is useless in a subdir) ‘At least one “Disallow” field must be present in the robots.txt file.’ – check for that Links Development of this plugin goes on on GitHub.

ReCrawler is a small WordPress Plugin for quickly notifying search engines whenever their website content is created, updated, or deleted. Improve your rankings by taking control of the crawling and indexing process, so search engines know what to focus on! Once installed, it detects pages/terms creation/update/deletion in WordPress and automatically submits the URLs in the background via ReCrawler, Google API, Bing API, Yandex API, and WebSub (formerly PubSubHubbub) protocols. It ensures that search engines invariably have the latest updates about your site. 🤖 What is ReCrawler? ReCrawler is an easy way for websites owners to instantly inform search engines about latest content changes on their website. In its simplest form, ReCrawler is a simple ping so that search engines know that a URL and its content has been added, updated, or deleted, allowing search engines to quickly reflect this change in their search results. Without ReCrawler, it can take days to weeks for search engines to discover that the content has changed, as search engines don’t crawl every URL often. With ReCrawler, search engines know immediately the “URLs that have changed, helping them prioritize crawl for these URLs and thereby limiting organic crawling to discover new content.” ReCrawler is offered under the terms of the Attribution-ShareAlike Creative Commons License and has support from Microsoft Bing, Yandex. ✅ Requirement for search engines Search Engines adopting the ReCrawler protocol agree that submitted URLs will be automatically shared with all other participating Search Engines. To participate, search engines must have a noticeable presence in at least one market. ⛑️ Documentation and support If you have some questions or suggestions, welcome to our GitHub repository. 💙 Love ReCrawler for WordPress? If the plugin was useful, rate it with a 5 star rating and write a few nice words. 🏳️ Translations Help translate ReCrawler 🇺🇸 English (en_US) – Mikhail kobzarev 🇷🇺 Русский (ru_RU) – Mikhail kobzarev You could be next… Can you help with plugin translation? Please feel free to contribute! External services This plugin uses external services, which are documented below with links to the service’s Privacy Policy. These services are integral to the functionality and features offered by the plugin. However, we acknowledge the importance of transparency regarding the use of such services. Yandex Webmaster – https://webmaster.yandex.ru Yandex IndexNow – https://yandex.com/indexnow Bing IndexNow – https://www.bing.com/indexnow Bing Webmaster – https://ssl.bing.com/webmaster/ Google Indexing API – https://indexing.googleapis.com/ Naver IndexNow – https://searchadvisor.naver.com/indexnow Seznam IndexNow – https://search.seznam.cz/indexnow IndexNow – https://api.indexnow.org Google Developers Console – https://console.developers.google.com/ Yandex oauth – https://oauth.yandex.ru/ WebSub Hub (Google) – https://pubsubhubbub.appspot.com/ WebSub Hub (Superfeedr) – https://pubsubhubbub.superfeedr.com/