Rank Math SEO – AI SEO Tools to Dominate SEO Rankings vs Crawler Record

Crawler Record shows when Google, ChatGPT, Claude, and other top bots last visited pages on your website, so you can be confident your pages are being crawled. This is the first step in ensuring these platforms index your website so they can serve it to visitors. The data covers several top user-agents, including: Google Bing ChatGPT (OpenAI) Claude (Anthropic) Perplexity Meta Apple Amazon DuckDuckGo Common Crawl Administrators with manage_options can view this information: Admin page: a grouped list of all crawlers, with last seen time, last page, and robots.txt status for the crawler. Recent Pages: From the admin page, you can select a crawler and see its latest visited pages, the response status WordPress observed, and a destination when WordPress exposes one for a retained 3xx response. Crawler Activity Overview: Review 28-day visit totals, response-code health, crawler mix, a daily activity timeline, and crawler staleness. URL reports use up to the 50 most recent recorded visits per crawler. Crawler Status Overview: Filter the main dashboard and crawler table by crawlers that visited within 28 days, have become stale, or have never visited. The unfiltered dashboard includes the stale-crawler age distribution alongside current activity; the Stale view focuses on that distribution, latest-visit details, and Recent Pages links instead of empty or potentially misleading historical metrics. Header Codes for User-Agents: Did a user-agent successfully visit a page (2xx), receive a 3xx response, or land on a page that was missing or inaccessible (4xx)? Crawler Record will show you. The 3xx report includes every retained 3xx status, including 304 Not Modified; only responses with an observable redirect destination show where the request was sent. Admin Bar (front-end): quick-glance status for all user-agent visits. Per Page/Post: From the edit screen on any page or post, you can see the last time each crawler visited that page. Administrators can also open the main dashboard or an individual crawler’s Recent Pages report from this box. Robots-aware: The plugin checks your robots.txt and evaluates Allow/Disallow rules for a given path. WordPress’s “Discourage search engines” preference is reported separately because it asks compliant search engines to avoid indexing. Response-aware: Recent visits include the final HTTP status WordPress can observe, including ordinary WordPress 404 pages and 3xx responses that occur after tracking begins. The 3xx report groups all retained 3xx statuses together; 304 Not Modified does not redirect and therefore has no destination. Other 3xx rows show a destination only when WordPress/PHP exposes one. Responses served by a CDN, web server, static cache, or security layer before WordPress runs cannot be recorded. Identification note: Crawler identification is based on the supplied user-agent string. User-agent strings can be spoofed, and this plugin does not verify IPs. Learn how to use this plugin. Highlights Supports common user-agent variants for Google, Bing, OpenAI (ChatGPT), Anthropic (Claude), Perplexity, Meta, Apple, Amazon, DuckDuckGo, and Common Crawl. robots.txt status computed from a local physical or WordPress virtual source without outbound requests. Known limitations Can only track crawlers from the time the plugin is installed—it cannot reconstruct earlier visits. Per-post “last seen” timestamp and user-agent metadata update at most once every 10 minutes by default. Separately, rapid same-crawler, same-URL, same-status visits are suppressed for 30 seconds, and a high emergency ceiling retains up to 50 visits per crawler per minute. Concurrent requests matching the same crawler are not coordinated. One database write may replace another, causing a visit to be absent from recent history or daily totals, or causing the latest-visit fields to reflect whichever write finished last. Dashboard totals are therefore approximate retained activity rather than an exact raw access log because of throttling, retention limits, and possible concurrent writes. Response codes are available only for requests that reach WordPress. Responses served earlier by a CDN, web server, static cache, or security layer are outside the plugin’s visibility. The 3xx report includes all retained 3xx statuses, including 304 Not Modified. Redirect destinations are available only when WordPress/PHP exposes a Location header. Redirects issued before WordPress runs cannot be recorded, and a followed destination is retained as a separate visit if it reaches WordPress. Stored source and redirect URLs include scheme, validated host, optional port, and path. Query strings are omitted to avoid retaining sensitive tokens, personal data, and high-cardinality tracking parameters. Reverse-proxy configuration can affect the scheme WordPress observes. URLs up to 114 characters are stored normally. URLs from 115 through 1,028 characters retain the visit with a 100-character prefix and an excessive-length marker; they are displayed with an ellipsis and are not linked. URLs longer than 1,028 characters are omitted from URL-level reports. Sitewide tracking for such requests is subject to the 30-second suppression window, while applicable per-post metadata uses its separate 10-minute throttle. Privacy This plugin stores: – Timestamps of crawler visits (float, with microseconds) – Exact matched crawler user-agent strings, bounded to 512 characters – Last URL and a bounded 50-visit recent history per crawler – Final HTTP response status observable by WordPress for recent crawler visits – Observable redirect destination, when PHP exposes one for a retained 3xx response other than 304 Not Modified – Last post ID per crawler (for admin performance) Stored source and redirect URLs omit query strings. Incoming hosts are validated and used only for reporting; they are never used for redirects, filesystem access, or outbound requests. Existing URL length limits apply independently to redirect sources and destinations. The plugin stores requests matching configured crawler patterns, but user-agent strings can be spoofed. It performs no IP verification and transmits no data to third parties. Crawler reports, the front-end admin-bar menu, and the post/page meta box require manage_options. Source URLs with hosts outside the configured WordPress home/site hosts are displayed as non-clickable diagnostic text. Uninstall removes all current and legacy Crawler Record options, transients, and post metadata from each site in a multisite network; WordPress removes the plugin files themselves. License GPLv2 or later. See LICENSE file.

Rank Math SEO – WordPress AI SEO Plugin to Rank Higher 📈 ★★★★★ Rank Math SEO helps you stay ahead of competitors with AI-powered SEO tools built for modern search engines, AI assistants, and evolving search experiences. The plugin brings powerful AI SEO tools right into WordPress. It helps you optimize content, include Schema Markup, add smart internal links, target the right keywords, run SEO checks, and more. Over 4 million WordPress websites trust Rank Math to grow their organic traffic faster. If you know you could be doing more for your website’s SEO but don’t have the time, Rank Math SEO is exactly what you need. Try the FREE Demo of Rank Math Features | Why Choose Rank Math? | Compare | Screenshots | Benefits [NEW] AI Visibility: Monitor Your Brand Across AI Search ➜ AI Visibility score to monitor your brand’s visibility ➜ Track how ChatGPT and other AI platforms mention your brand ➜ Monitor your brand’s sentiment across AI responses ➜ See how you compare against your competitors in AI Search ➜ Access complete transcripts of AI responses ➜ Check your brand’s AI Visibility across different countries Learn more about AI Visibility ➜ [NEW] AI Link Genius: Smart Internal Linking on Autopilot ➜ View every internal and external link on your site from one dashboard ➜ Update hundreds of links at once instead of editing pages one by one ➜ Get smart link suggestions as you write, right inside the editor ➜ Monitor broken links, redirects and overall link health ➜ Automatically link keyword variations to the right pages ➜ Get related post suggestions that keep readers on your site longer ➜ Find pages nobody links to and reconnect them to the rest of your site ➜ Export your link data for reports or audits Learn more about AI Link Genius ➜ Exclusive Content AI. Write Highly Optimized SEO Content Get Started with Content AI for FREE and enjoy: ➜ 40+ AI Tools Built for SEO ➜ Use AI SEO to write high-quality content ➜ Fix SEO Issues in One Click ➜ Generate AI Image Alt Text Instantly ➜ Get Instant SEO Help with RankBot ➜ Full Control via Role Manager Learn more about Content AI ➜ Rank Math Includes Every SEO Tool You Need to Rank Higher See the features only Rank Math offers, and find out why it’s the right SEO plugin for you. ✔ Intuitive Setup Wizard that configures Rank Math automatically for your site ✔ 16+ Schema types to add rich results in few clicks ✔ Optimize post for unlimited focus keywords (5 by default, unlimited with this filter) ✔ Content AI to help you write SEO-friendly content ✔ Google Search Console integration ✔ Google Analytics Integration ✔ LSI keyword suggestions to attract more traffic ✔ XML Sitemap with custom post type support ✔ Detailed content analysis with actionable SEO recommendations ✔ 30+ SEO tests with 1-click ✔ Module based system, enable only what you need ✔ SEO optimized breadcrumbs with theme support ✔ Smart Redirection manager to create redirects at scale ✔ Built-in 404 monitor to catch and fix broken links ✔ Internal link suggestions ✔ Role Manager to control what each team member can access ✔ Automatic ALT Text and Title tags for every image ✔ WordPress Multisite support Why is Rank Math such a game-changer? ✔ Fastest SEO plugin with barely any impact on site speed ✔ WooCommerce SEO with automatic Schema and meta tags ✔ Elementor SEO with native integration ✔ Divi SEO with native integration ✔ Local SEO tools to help nearby customers find your business ✔ Google AMP SEO with SEO data added automatically to AMP pages ✔ bbPress & BuddyPress SEO support with Q&A Schema ✔ 1-Click Import from Yoast ✔ 1-Click Import from AIO SEO ✔ 1-Click Import from SEOPress ✔ 1-Click Import from All In One Schema ✔ 1-Click Import from Redirection plugin ✔ Translation plugins support including WPML, Weglot, TranslatePress and Polylang ✔ Theme support including Astra, Kadence, Flothemes, OceanWP, Themify, Schema Theme and more ✔ Page builder support including Oxygen, WPBakery, Avada, Beaver Builder, and Page Builder Framework and more Who Can Benefit From Rank Math? Rank Math Plugin is perfect for: ✔ Bloggers ✔ eCommerce Store Owners ✔ Niche Sites ✔ Businesses ✔ Agencies ✔ Local Businesses ✔ Startups ✔ The Real Estate ✔ Artists & Photographers ✔ The Solution Offerer ✔ Directories ✔ Vloggers (Video Bloggers) ✔ Or any WordPress Website Take a sneak peek into Rank Math’s features Detailed Setup Tutorial BIG Publications are Raving About Rank Math Rank Math Reviews Rank Math Review – Why I Ditched Yoast For Rank Math Rank Math on Product Hunt Rank Math Plugin Review from an SEO Consultant and many more Check Rank Math PRO Which Makes SEO Easier & Faster Rank Math FREE VS PRO COMPARISON Unique Features of Rank Math PRO AI Link Genius ✔ AI-Powered Internal Link Suggestions ✔ Auto-Link Keyword Variations ✔ Automatic Link Audits & Monitor Link Health ✔ Bulk Link Update Tool Google Analytics & Search Console Integration ✔ Integrated Google Analytics, AdSense & Search Console Data ✔ Analyze the SEO Performance of Each Post & Page ✔ See Top Winning/Losing Posts & Keywords ✔ Keep Track of Position History ✔ Rank Tracker for Important Keywords Schema aka Structured Data aka Rich Snippets ✔ 20+ Pre-defined Schema types ✔ Import Schema from Other Websites ✔ Advanced Schema Builder to create any complex Schema Markup ✔ Schema Templates for Automation ✔ Conditional Schema Markup ✔ Multiple Location Schema On Any Page (using the Shortcode) SEO Automation ✔ Automated Image SEO ✔ Advanced Filtering for Images [https://i.rankmath.com/zAUHHP] ✔ Watermark Your images ✔ Advanced Post filtering ✔ Bulk Actions [index, noindex, redirect, etc.] ✔ Quick Edit SEO Details ✔ Bulk Import SEO Meta Details Using CSV file ✔ Auto Detect Videos and Generate Schema Markup for Them ✔ Auto Fetch Thumbnail, Duration of YouTube & Vimeo Videos ✔ Automatically Flush Facebook Thumbnails ✔ Open External Links in New Tabs ✔ Nofollow External Links ✔ Noindex Paginated, Archive, Search Result Pages ✔ Instant Indexing for Bing & Yandex [Also Google using our Instant Indexing Plugin] WooCommerce SEO ✔ Automatic Schema for WooCommerce Products ✔ Advanced Open Graph Tags for WooCommerce Products ✔ Automatic NoIndex Hidden Products ✔ Remove WooCommerce Product and Category Base ✔ Add Custom Brands to Products ✔ Add Global Identifier Schema Like GTIN/MPN – Even to Variations MANY MORE FEATURES ✔ PREMIUM 24x7x365 Dedicated Support Managers ✔ MOST COMPETITIVE PRICING EVER! CHECK ALL THE PREMIUM FEATURES AND PRICING HERE Connect with the Team and SEO Experts JOIN FACEBOOK GROUP COMMUNITY: The purpose of this Facebook group is to have a collective place where the community can help each other, and we can get some feedback to improve Rank Math as well. Joining the group is also a great way to connect with like-minded people and share your SEO experience. Branding Guideline Rank Math® SEO is a registered trademark. Please use the following format when mentioning the Rank Math plugin anywhere. * Rank Math [correct] * RankMath [incorrect] * Rankmath [incorrect] * rankmath [incorrect] * rankMath [incorrect] External Services Rank Math SEO connects to several external services to provide its features. Below is a full disclosure of each service, what data is sent, and when. Rank Math Keyword Suggestions API (api.rankmath.com) Used to provide live keyword suggestions as you type in the Focus Keyword field in the post editor. Keyword suggestions: Each time you type in the Focus Keyword field, the keyword and your site’s locale are sent to api.rankmath.com to retrieve related keyword suggestions. This call is made directly from your browser. No account or login is required for this feature. Service provider: Rank Math (ONE.COM INDIA PRIVATE LIMITED) Privacy policy: https://rankmath.com/privacy-policy/ Terms of service: https://rankmath.com/terms-of-service/ Rank Math API (rankmath.com) Used for site registration, license verification, and the SEO Analysis feature. Site registration / account connection: Sends your site URL, Rank Math username, and API key when you connect your Rank Math account. This is required to activate API-based features. Site disconnection: Sends your site URL, username, and API key when you disconnect your account to remove the registration from Rank Math’s servers. SEO Analysis: Sends your site URL, target keyword, and language when you run an SEO site analysis. Results are returned as a scored report. Dashboard widget: Fetches the latest Rank Math blog posts to display in the WordPress admin dashboard widget. No personal data is sent. Results are cached for 12 hours. Service provider: Rank Math (ONE.COM INDIA PRIVATE LIMITED) Privacy policy: https://rankmath.com/privacy-policy/ Terms of service: https://rankmath.com/terms-of-service/ Rank Math OAuth Connector (oauth.rankmath.com) Used to connect your site to Google services (Search Console, Google Analytics) via OAuth. OAuth token exchange: When you authorize Google services, your browser is redirected through oauth.rankmath.com to exchange the authorization code for access and refresh tokens. The authorization code is sent to this service; it returns OAuth tokens for use on your site. Token refresh: Sends your Google refresh token to obtain a new access token when the current one expires. Service provider: Rank Math (ONE.COM INDIA PRIVATE LIMITED) Privacy policy: https://rankmath.com/privacy-policy/ Terms of service: https://rankmath.com/terms-of-service/ Rank Math Content AI (cai.rankmath.com) Used to power AI-based content features. Only active when the Content AI module is enabled and you have a connected Rank Math account. Keyword research: Sends the target keyword, language, and country to fetch keyword suggestions and metrics. Bulk SEO meta generation: Sends post title and content to generate SEO titles and descriptions in bulk. Bulk image alt text generation: Sends image data (base64-encoded) to generate descriptive alt text for images. Default prompts sync: Sends your site URL, username, API key, and plugin version daily to keep AI prompt templates up to date. Credits/wallet check: Sends your username, API key, and site URL to check remaining Content AI usage credits. Service provider: Rank Math (ONE.COM INDIA PRIVATE LIMITED) Privacy policy: https://rankmath.com/privacy-policy/ Terms of service: https://rankmath.com/terms-of-service/ Google APIs (googleapis.com, google.com) Used by the Analytics module to retrieve website performance data. Only active when you have connected Google Analytics via the Rank Math Analytics module. Google Analytics data: Sends your OAuth access token to fetch data like traffic, keyword, impressions, etc for your site from the Google Analytics API. Token verification: Sends your OAuth access token to verify its validity and authorized scopes. Google Tag Manager script (googletagmanager.com): When a Google Analytics / Google Tag property ID is configured, the Google Tag (gtag.js) script is loaded from googletagmanager.com on your site’s front end. AMP analytics (cdn.ampproject.org): When AMP is active and a Google Analytics property is configured, the AMP analytics component script is loaded from cdn.ampproject.org. Service provider: Google LLC Privacy policy: https://policies.google.com/privacy Terms of service: https://policies.google.com/terms IndexNow API (api.indexnow.org) Used by the Instant Indexing module to notify search engines of new or updated content. Only active when the Instant Indexing module is enabled. Sends a list of URLs, your site URL, and your IndexNow API key to api.indexnow.org each time content is published or updated (or manually triggered). Service provider: Microsoft (Bing) Privacy policy: https://privacy.microsoft.com/en-us/privacystatement Terms of service: https://www.indexnow.org/faq Facebook Graph API (graph.facebook.com) Used to refresh Facebook’s cached preview when a post is updated. Only active when a Facebook App ID and App Secret are configured in Rank Math’s Social settings. Sends the post’s public URL and your Facebook App credentials to trigger a re-scrape of the page by Facebook. Service provider: Meta Platforms, Inc. Privacy policy: https://www.facebook.com/privacy/policy/ Terms of service: https://developers.facebook.com/terms/