Crawler Record vs Head Meta Data

Crawler Record logs the last time specific user agents (for search and AI chat/LLMs) accessed your content including: Google Bing ChatGPT (OpenAI) Claude (Anthropic) Perplexity DuckDuckGo Meta Apple Common Crawl You can view this information: Admin page: a grouped list of all crawlers, with last seen time, last page, and robots status for the crawler. Admin Bar (front-end): quick-glance status for the current singular or URL context (no dropdowns in wp-admin). Per Page/Post: From the edit screen on any page or post, you can see that last time each crawler visited that page. Recent Pages: From the admin page you can select a crawlwer and get a list of the recent pages this UA has visited. Robots-aware: The plugin checks your robots.txt and evaluates Allow/Disallow rules for a given path. If Settings → Reading → “Discourage search engines” is enabled, all agents are shown as blocked with a prominent warning. Performance-friendly by design: Write-throttling (default 10 minutes) and an auxiliary “last post ID per agent” record avoids heavy admin queries on large sites. Privacy-friendly: Saves only bot visit timestamps and last URLs crawled — no personal data. Learn how to use this plugin. Highlights Supports common user-agent variants for Google, Bing, OpenAI (ChatGPT), Anthropic (Claude), Perplexity, Meta, Apple, DuckDuckGo, and Common Crawl. Robots status computed from local robots.txt (physical or virtual) without outbound requests. Clear UI with grouped sections, microsecond timestamps, and a small diagnostics toggle showing the matched robots group and rule. Known limitations Can only track crawlers from the time the plugin is installed- cannot look into the past. robots.txt wildcards (*) and end-of-line marker ($) are not interpreted; matching is prefix-based only. Future versions may add full spec support. To reduce database load and avoid slowing the website, bot tracking is throttled, so not every crawler visit will necessarily be recorded. Privacy This plugin stores: – Timestamps of crawler visits (float, with microseconds) – Last URL seen per crawler (per-URL records) – Last post ID per crawler (for admin performance) It does not collect or store personal data about site visitors. No data is transmitted to third parties. License GPLv2 or later. See LICENSE file.

✨ Add meta tags to your site ✨ Blazing fast performance ✨ Uses only 60KB of code! Head Meta Data (HMD) improves the definition and semantic quality of your website by adding tags to the section of your web pages. Super lightweight, fast, and user-friendly. 🤖 Default Meta Tags Customize the perfect set of meta tags for your site. For example: 🤖 Dynamic Meta Tags Use shortcodes to dynamically output SEO meta tags: [hmd_post_title] | My Awesome Website This outputs dynamic titles and descriptions, useful for SEO. You also can add unique meta tags and markup on specific posts and pages. Check out the Installation tab for more information. ✨ Help bots understand your content better 🤖 Features Simple and easy to use Clean, standards-based code Customize all tags Add your own custom tags Add custom meta tags on any post or page Supports X (Twitter) Cards and Facebook Open Graph tags Supports , , , , and Automatically adds tags to the section of all pages Check out a Live Preview of your meta tags and custom tags Auto-populates tags using your site’s information Use shortcodes to include dynamic information Easily disable any unwanted tags Lightweight, fast, and secure ✨ Check out Head Meta Pro to define tags for each page view 🤖 Shortcodes Shortcodes enable you to include dynamic bits of information in your meta tags. Head Meta Data provides the following shortcodes: [hmd_post_excerpt] Outputs post excerpt [hmd_post_date] Outputs post date [hmd_post_author] Outputs post author [hmd_post_title] Outputs post title [hmd_post_cats] Outputs post categories [hmd_post_tags] Outputs post tags [hmd_site_tagline] Outputs site tagline [hmd_site_title] Outputs site title [hmd_year] Outputs current year [hmd_tab] Outputs tab space to tag markup So you can display your own set of custom meta tags exactly as desired. Learn more » 🤖 Pro Features The Pro version can do everything the free version can do, PLUS: Define meta tags for Custom Post Types Define meta tags for each type of page-view (e.g., home, posts, pages, archive, search) Define tags for Facebook (Open Graph) Define tags for X (Twitter) Define Facebook and X tags for each type of page view 40+ advanced shortcut variables like %POST_AUTHOR% and %ARCHIVE_TYPE% Dedicated “Meta Tags” meta box for posts and pages Add custom meta tags to any post or page-view Add custom script and style to any post or page-view Meta Tags box integrates with SEO plugins Additional meta tags beyond the free version Dedicated setting for image_src link tag Premium support for pro purchases ✨ Level up your meta tags with Head Meta Pro » 🤖 Privacy This plugin does not collect or store any user data. It does not set any cookies, and it does not connect to any third-party locations. Thus, this plugin does not affect user privacy in any way. Head Meta Data is developed and maintained by Jeff Starr, 15-year WordPress developer and book author. 🤖 Support Development I develop and maintain this free plugin with love for the WordPress community. To show support, you can make a donation or purchase one of my books: The Tao of WordPress Digging into WordPress .htaccess made easy WordPress Themes In Depth Wizard’s SQL Recipes for WordPress And/or purchase one of my premium WordPress plugins: BBQ Pro – Blazing fast WordPress firewall Blackhole Pro – Automatically block bad bots Banhammer Pro – Monitor traffic and ban the bad guys GA Google Analytics Pro – Connect WordPress to Google Analytics Head Meta Pro – Ultimate Meta Tags for WordPress REST Pro Tools – Awesome tools for managing the WP REST API Simple Ajax Chat Pro – Unlimited chat rooms USP Pro – Unlimited front-end forms Links, tweets and likes also appreciated. Thanks! 🙂