Crawler Record vs AI Kansoku Lab Tracker

Crawler Record logs the last time specific user agents (for search and AI chat/LLMs) accessed your content including: Google Bing ChatGPT (OpenAI) Claude (Anthropic) Perplexity DuckDuckGo Meta Apple Common Crawl You can view this information: Admin page: a grouped list of all crawlers, with last seen time, last page, and robots status for the crawler. Admin Bar (front-end): quick-glance status for the current singular or URL context (no dropdowns in wp-admin). Per Page/Post: From the edit screen on any page or post, you can see that last time each crawler visited that page. Recent Pages: From the admin page you can select a crawlwer and get a list of the recent pages this UA has visited. Robots-aware: The plugin checks your robots.txt and evaluates Allow/Disallow rules for a given path. If Settings → Reading → “Discourage search engines” is enabled, all agents are shown as blocked with a prominent warning. Performance-friendly by design: Write-throttling (default 10 minutes) and an auxiliary “last post ID per agent” record avoids heavy admin queries on large sites. Privacy-friendly: Saves only bot visit timestamps and last URLs crawled — no personal data. Learn how to use this plugin. Highlights Supports common user-agent variants for Google, Bing, OpenAI (ChatGPT), Anthropic (Claude), Perplexity, Meta, Apple, DuckDuckGo, and Common Crawl. Robots status computed from local robots.txt (physical or virtual) without outbound requests. Clear UI with grouped sections, microsecond timestamps, and a small diagnostics toggle showing the matched robots group and rule. Known limitations Can only track crawlers from the time the plugin is installed- cannot look into the past. robots.txt wildcards (*) and end-of-line marker ($) are not interpreted; matching is prefix-based only. Future versions may add full spec support. To reduce database load and avoid slowing the website, bot tracking is throttled, so not every crawler visit will necessarily be recorded. Privacy This plugin stores: – Timestamps of crawler visits (float, with microseconds) – Last URL seen per crawler (per-URL records) – Last post ID per crawler (for admin performance) It does not collect or store personal data about site visitors. No data is transmitted to third parties. License GPLv2 or later. See LICENSE file.

AI Kansoku Lab Tracker detects visits from AI-related crawlers such as GPTBot, ChatGPT-User, ClaudeBot, and PerplexityBot using PHP server-side detection. AI crawlers often do not execute JavaScript, so JavaScript-based analytics tools such as GA4 may not record these visits as normal pageviews. This plugin helps site owners observe AI crawler activity at the server level. Detected crawler visits are stored in your WordPress database. Optional external tracking is available only when you explicitly enable it in the settings page. Features 34 AI-related crawler patterns from 20+ companies – GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Anthropic-AI, PerplexityBot, Perplexity-User, Google-Extended, Google-NotebookLM, Google-Agent, ByteSpider, Meta-ExternalAgent, Meta-ExternalFetcher, AppleBot, AppleBot-Extended, DeepSeek, Grok, xAI-SearchBot, CopilotBot, Bingbot, AmazonBot, YandexBot, YouBot, Diffbot, ShapBot, TavilyBot, DuckAssistBot, Cohere-AI, Timpibot, FirecrawlAgent, and related access patterns Training vs. user-triggered classification – Crawler visits are classified as either “Autonomous AI (training/patrol)” or “Human-triggered AI reference,” showing whether your content is being collected for AI training or actually being looked up because someone asked an AI about it Local crawler logging – Detected visits are stored in your WordPress database 30-day trend analysis – Dashboard with 30-day / 7-day period switching AI Exposure Score – Visibility score based on crawler activity Crawler breakdown – See which AI-related services visit your site most Top pages – Identify which pages AI crawlers focus on Explicit opt-in for external tracking – External transmission only activates when you enable it via checkbox Minimal performance impact – Optional external requests use non-blocking POST requests Front-end only – Does not fire on admin, cron, AJAX, or REST API requests WordPress standards – Uses wp_remote_post, register_setting, nonce verification, and proper escaping throughout 日本語の説明 AI Kansoku Lab Tracker は、GPTBot・ChatGPT-User・ClaudeBot・PerplexityBot などの AI 関連クローラーがあなたのサイトを訪問した記録を、WordPress 上で検知・可視化するプラグインです。 AI クローラーは JavaScript を実行しないことが多いため、GA4 などの JavaScript ベースのアクセス解析では通常のページビューとして記録されない場合があります。本プラグインは PHP によるサーバーサイド検知を採用し、AI クローラーの訪問をサーバーレベルで観測できるようにします。 検知したクローラーのログは WordPress のローカルデータベースに保存されます。外部トラッキングを有効にした場合のみ、AI観測ラボへデータを送信します。 主な機能: 20社以上・34種類のAI関連クローラーパターンを検知 「学習・巡回AI」と「人間起点のAI参照」を区別して表示 — サイトがAIの学習データとして収集されているだけなのか、それとも誰かが実際にAIに質問した結果として読まれているのかが一目でわかります 検知ログを WordPress のローカルDBに保存 30日間のトレンド分析（7日間表示への切り替え対応） AI露出スコアによる可視化 クローラー種別の内訳表示 よく見られているページの可視化 外部トラッキングは明示的なオプトイン方式 管理画面・AJAX・REST API・WP-Cron では実行しない設計 開発・運営: AI観測ラボ How It Works When a front-end page is requested, the plugin checks the User-Agent string against known AI-related crawler patterns. If a match is found, the detected visit is stored in your WordPress database. If you have explicitly enabled external tracking and configured a Site ID, the plugin also sends an asynchronous POST request to the AI Kansoku Lab API. This optional external request uses a non-blocking request so that the site visitor experiences no delay. External Services This plugin may connect to the following external service: AI Kansoku Lab https://www.ai-kansoku.com External tracking is opt-in only. Data is only transmitted after you explicitly enable external tracking in the plugin settings page. When an AI-related crawler is detected and external tracking is enabled, the following data is sent to the AI Kansoku Lab API: User-Agent string of the detected crawler URL path of the visited page HTTP referer header, if present HTTP status code of the page, such as 200 or 404 Site ID configured in the plugin settings No personally identifiable information is collected. This plugin does not collect cookies, account credentials, form contents, or visitor IP addresses. Service URL: https://www.ai-kansoku.com Privacy Policy: https://www.ai-kansoku.com/privacy Terms of Service: https://www.blog.ai-kansoku.com/terms Requirements WordPress 6.0 or higher PHP 7.4 or higher A free AI Kansoku Lab account is optional and only required if you want to use the external dashboard integration. Privacy Policy This plugin stores detected crawler visits in your WordPress database. This plugin can also connect to an external service to log AI-related crawler visits in the AI Kansoku Lab dashboard. External tracking is opt-in only and only activates after explicit user consent via the settings page. Data transmitted to AI Kansoku Lab when external tracking is enabled: User-Agent string of the detected crawler URL path of the visited page HTTP referer header, if present HTTP status code of the page, such as 200 or 404 Site ID configured in plugin settings No personally identifiable information is collected. This plugin does not collect cookies, account credentials, form contents, or visitor IP addresses. Full privacy policy: https://www.ai-kansoku.com/privacy