AI Kansoku Lab Tracker vs Bisteinoff SEO Robots.txt

Have you encountered an obstacle while creating and editing robots.txt file on your website? Bisteinoff SEO Robots.txt is an easy-to-use plugin that helps you generate and configure a correct robots.txt file, which is essential for search engine optimization (SEO). This file defines crawling rules for search engine bots such as Google, Bing, Yahoo!, Yandex, and others. The plugin works perfectly both if the file robots.txt has never been created or if it already exists. Once installed the plugin makes an optimized robots.txt file that includes special rules common for WordPress websites. After that you can proceed further customization specific for your own website if needed. If the plugin detects one or several Sitemap XML files it will include them into robots.txt file. No FTP access, manual coding or file editing is required that makes managing settings easy and convenient! Key Features Automatic generation of optimized robots.txt with WordPress-specific rules Special rules for Google and Yandex search engines Custom rules support for any search engine bot Automatic sitemap detection and inclusion WooCommerce compatibility with specific rules Multisite support Easy-to-use admin interface Modern PHP architecture with namespaces for conflict-free operation

AI Kansoku Lab Tracker detects visits from AI-related crawlers such as GPTBot, ChatGPT-User, ClaudeBot, and PerplexityBot using PHP server-side detection. AI crawlers often do not execute JavaScript, so JavaScript-based analytics tools such as GA4 may not record these visits as normal pageviews. This plugin helps site owners observe AI crawler activity at the server level. Detected crawler visits are stored in your WordPress database. Optional external tracking is available only when you explicitly enable it in the settings page. Features 34 AI-related crawler patterns from 20+ companies – GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Anthropic-AI, PerplexityBot, Perplexity-User, Google-Extended, Google-NotebookLM, Google-Agent, ByteSpider, Meta-ExternalAgent, Meta-ExternalFetcher, AppleBot, AppleBot-Extended, DeepSeek, Grok, xAI-SearchBot, CopilotBot, Bingbot, AmazonBot, YandexBot, YouBot, Diffbot, ShapBot, TavilyBot, DuckAssistBot, Cohere-AI, Timpibot, FirecrawlAgent, and related access patterns Training vs. user-triggered classification – Crawler visits are classified as either “Autonomous AI (training/patrol)” or “Human-triggered AI reference,” showing whether your content is being collected for AI training or actually being looked up because someone asked an AI about it Local crawler logging – Detected visits are stored in your WordPress database 30-day trend analysis – Dashboard with 30-day / 7-day period switching AI Exposure Score – Visibility score based on crawler activity Crawler breakdown – See which AI-related services visit your site most Top pages – Identify which pages AI crawlers focus on Explicit opt-in for external tracking – External transmission only activates when you enable it via checkbox Minimal performance impact – Optional external requests use non-blocking POST requests Front-end only – Does not fire on admin, cron, AJAX, or REST API requests WordPress standards – Uses wp_remote_post, register_setting, nonce verification, and proper escaping throughout 日本語の説明 AI Kansoku Lab Tracker は、GPTBot・ChatGPT-User・ClaudeBot・PerplexityBot などの AI 関連クローラーがあなたのサイトを訪問した記録を、WordPress 上で検知・可視化するプラグインです。 AI クローラーは JavaScript を実行しないことが多いため、GA4 などの JavaScript ベースのアクセス解析では通常のページビューとして記録されない場合があります。本プラグインは PHP によるサーバーサイド検知を採用し、AI クローラーの訪問をサーバーレベルで観測できるようにします。 検知したクローラーのログは WordPress のローカルデータベースに保存されます。外部トラッキングを有効にした場合のみ、AI観測ラボへデータを送信します。 主な機能: 20社以上・34種類のAI関連クローラーパターンを検知 「学習・巡回AI」と「人間起点のAI参照」を区別して表示 — サイトがAIの学習データとして収集されているだけなのか、それとも誰かが実際にAIに質問した結果として読まれているのかが一目でわかります 検知ログを WordPress のローカルDBに保存 30日間のトレンド分析（7日間表示への切り替え対応） AI露出スコアによる可視化 クローラー種別の内訳表示 よく見られているページの可視化 外部トラッキングは明示的なオプトイン方式 管理画面・AJAX・REST API・WP-Cron では実行しない設計 開発・運営: AI観測ラボ How It Works When a front-end page is requested, the plugin checks the User-Agent string against known AI-related crawler patterns. If a match is found, the detected visit is stored in your WordPress database. If you have explicitly enabled external tracking and configured a Site ID, the plugin also sends an asynchronous POST request to the AI Kansoku Lab API. This optional external request uses a non-blocking request so that the site visitor experiences no delay. External Services This plugin may connect to the following external service: AI Kansoku Lab https://www.ai-kansoku.com External tracking is opt-in only. Data is only transmitted after you explicitly enable external tracking in the plugin settings page. When an AI-related crawler is detected and external tracking is enabled, the following data is sent to the AI Kansoku Lab API: User-Agent string of the detected crawler URL path of the visited page HTTP referer header, if present HTTP status code of the page, such as 200 or 404 Site ID configured in the plugin settings No personally identifiable information is collected. This plugin does not collect cookies, account credentials, form contents, or visitor IP addresses. Service URL: https://www.ai-kansoku.com Privacy Policy: https://www.ai-kansoku.com/privacy Terms of Service: https://www.blog.ai-kansoku.com/terms Requirements WordPress 6.0 or higher PHP 7.4 or higher A free AI Kansoku Lab account is optional and only required if you want to use the external dashboard integration. Privacy Policy This plugin stores detected crawler visits in your WordPress database. This plugin can also connect to an external service to log AI-related crawler visits in the AI Kansoku Lab dashboard. External tracking is opt-in only and only activates after explicit user consent via the settings page. Data transmitted to AI Kansoku Lab when external tracking is enabled: User-Agent string of the detected crawler URL path of the visited page HTTP referer header, if present HTTP status code of the page, such as 200 or 404 Site ID configured in plugin settings No personally identifiable information is collected. This plugin does not collect cookies, account credentials, form contents, or visitor IP addresses. Full privacy policy: https://www.ai-kansoku.com/privacy