XeroWP
All-IN Crawler & AI Agent Manager – Block AI Bots, Bad Bots & Rate-Limit Crawlers
All-IN Crawler & AI Agent Manager – Block AI Bots, Bad Bots & Rate-Limit Crawlers

All-IN Crawler & AI Agent Manager – Block AI Bots, Bad Bots & Rate-Limit Crawlers

0/5 (0 ratings) 20 active installs Updated Nov 13, 2026
AI Crawlersbad botsbot blockerllms.txtrobots.txt
Download v1.2.2
Dashboard with statistics per category and bot, server load and top denied IPs.

Dashboard with statistics per category and bot, server load and top denied IPs.

A bot blocker and AI crawler manager for WordPress. Block AI crawlers like GPTBot, ClaudeBot and CCBot, stop bad bots and scrapers, and rate-limit aggressive crawlers, while Google, Bing and AI assistants like ChatGPT search can still find you. New installs start in Watch-only mode, so nothing is blocked until you choose.

Why use it

  • Protect your content from AI training without disappearing from AI answers.
  • Cut server load from aggressive crawlers, SEO tools and scrapers.
  • Stay in search: real search engines stay allowed and verified.

Block AI crawlers, keep AI search

About 120 known bots in 12 categories. Training crawlers (GPTBot, ClaudeBot, CCBot…) are handled separately from AI search and assistant agents (OAI-SearchBot, ChatGPT-User, Claude-User, Perplexity-User…). Allow, rate-limit or block per category, with per-bot overrides, custom User-Agent rules (text or regex) and one-click presets: Balanced, Maximum AI visibility, Protect content from AI training, Lockdown.

Stop bad bots, fake Googlebots and scrapers

Forward-confirmed reverse DNS plus optional official crawler IP lists (Google, Bing, Apple, OpenAI, Perplexity; off until you turn them on) catch impostors. Scanners, HTTP libraries and empty User-Agents get their own categories. IP / CIDR allow and block lists (IPv4 and IPv6) and excluded paths for webhooks.

Rate limit bots and protect server load

Rate limits per IP or per bot answer with 429 + Retry-After. Load shedding answers with 503 + Retry-After when server load per CPU core crosses your threshold. An optional per-IP limit covers “human” visitors (logged-in users are exempt). SEO-safe status codes only, never cached.

robots.txt and llms.txt manager

Manage robots.txt rules (including Google-Extended, Applebot-Extended and an optional Crawl-delay) and generate an llms.txt that skips noindex content.

Block before PHP

Export rules to .htaccess (Apache/LiteSpeed) or nginx. Inside WordPress, requests are decided while plugins load, before the theme runs. It fails open: if anything goes wrong inside the plugin, visitors are let through.

Reports and logs

7/14-day statistics, top denied IPs with one-click blocking, a denied-request log, a User-Agent tester and weekly or monthly email reports. Full Hebrew translation, RTL ready.

More from A-in

This plugin is complete on its own: no licence key, no locked features. Premium plugins, add-ons and support from the same developer are available at a-in.co.il.

External services

All-IN Crawler & AI Agent Manager works without any external service. These features contact third parties:

  • Official crawler IP lists (Bot Lists tab “Download official crawler IP lists”, off by default, opt-in). Nothing is downloaded until you tick the box and save; the “Download official IP lists” button is also disabled until then. Once enabled, the site downloads the public IP range files published by the crawler operators once a day, to verify that a request claiming to be their bot really comes from them. No data about your site or visitors is sent: it is a plain HTTPS GET with a generic all-in-crawler-ai-agent-manager/<version> User-Agent, so the publisher only sees your server’s IP address. Untick the box to stop the downloads at any time. Sources:
    • Google: https://developers.google.com/static/search/apis/ipranges/ (Terms, Privacy)
    • Bing: https://www.bing.com/toolbox/bingbot.json (Terms, Privacy)
    • Apple: https://search.developer.apple.com/applebot.json (Terms, Privacy)
    • OpenAI: https://openai.com/gptbot.json, searchbot.json, chatgpt-user.json (Terms, Privacy)
    • Perplexity: https://www.perplexity.com/perplexitybot.json, perplexity-user.json (Terms, Privacy)
  • Bot-list feed (Lists Bot list updates, off by default). The plugin ships its extra bot definitions in data/bots.json and needs no download. If you enter a feed URL of your own (for example your agency’s feed), the site requests that URL once a day, sending your site URL in the User-Agent and, if you set one, your access key as a Bearer token. There is no default remote feed.
  • DNS verification of search engine bots uses reverse and forward DNS lookups through your server’s resolver. No third-party API is called.