AI Crawler and SEO Manager by akbansa for WordPress

The AI Crawler and SEO Manager by akbansa plugin generates an llms.txt file for crawler discovery and logs visits by AI-based crawlers such as GPTBot, ClaudeBot, and others. It also provides an analytics page in the WordPress admin dashboard to view crawler activity by path, URL, and timestamp.

Key Features:

Generates llms.txt file for crawler discovery.

Removes llms.txt on plugin uninstall

Schedules automatic generation of llms.txt daily at midnight (frequency configurable: daily, twice daily, hourly)

Uses atomic file writes for llms.txt and sitemap.xml for safety

Integrates with Yoast SEO, Rank Math, or All in One SEO for metadata (falls back to DB if not present)

Admin panel to select/exclude categories and content types for llms.txt

Logs every AI crawler request from the frontend (user agent, crawl info, page URL, IP, referrer)

Filters crawlers using config.json and stores all details in a dedicated DB table

Analytics dashboard in admin to view recent crawler activity

Robust AJAX security and error handling

Detects and logs visits from AI crawlers based on their User-Agent.

Stores crawler logs in a database for analysis.

Supports popular AI-based crawlers (e.g., OpenAI, Anthropic, GPTBot, etc.).