Crawlers
27 plugins tagged with crawlers
Block AI Crawlers
Adds robots.txt rules blocking AI crawlers and AI bots.
Robots.txt rewrite
Provide the easy managment of your robots.txt from admin side. It propose you the advanced then standard robots.txt content too.
AI Content Signals
Control how AI uses your content — search, AI answers or training — via robots.txt Content Signals, HTTP headers, meta tags and TDMRep.
CodingBunny LLMs.txt Generator
Advanced llms.txt generator with AI metadata, Schema.org types, canonical URLs, and differentiated licenses for optimal LLM indexing.
Robots Meta Whiz
Easy way to discourage search engines from indexing only specific pages / posts with custom meta tags.
MetaRobots by SEO-Sign
The easiest way to manage meta robots tag.
Serve Markdown
Serve Markdown versions of your posts and pages to AI agents and crawlers. Content negotiation, .md URLs, auto-discovery, and crawler logging.
ParseLess
Serve your posts as Markdown to AI crawlers and CLI tools — typically 95%+ less bandwidth, ~60–80% fewer DB queries, no theme bloat.
CeeWP Robots.txt Manager
Easily manage your WordPress site's virtual robots.txt file with custom rules. No physical file needed!
Log-Hero.com Plugin
See which search engines, AI crawlers and other bots visit your site, and what they find. The official plugin for log-hero.com.
Seo Crawlytics Data Export & Delete
Exports all entries of Seo Crawlytics and gives you the option to delete the entries that are older than 1 month.
Temso AI
Stream front-end request logs from your WordPress origin to Temso so AI-crawler and bot traffic shows up in your dashboard.
LJPL Armored robots.txt
Add some directives to your robots.txt file to keep your site safer
MSG LLMs TXT Manager
Serve your content as clean Markdown to AI agents, publish llms.txt and llms-full.txt, and see which agents actually fetch what.
PufferSights – AI Crawler Insights
Monitor 100+ known AI crawler and AI agent user agents with local analytics, AI referrals, llms.txt, and opt-in robots.txt policy tools.
Supertab Connect
Connect your WordPress site to the Supertab Connect platform for RSL license serving and enabling crawler authentication protocol.
PinMeOut
Control which AI crawlers may read your site, see which ones actually do, and make your pages worth citing. Works alongside your SEO plugin.
AI Rights Protection by SimpleFeed
Protect your rights by reviewing and blocking AI bots from using your content without compensation.
Frenemy
See every bot that reaches WordPress — search crawlers, AI agents, scrapers, impostors — classified and identity-verified, in one dashboard.
StudioMeyer Agent Readiness
Check what AI agents really get from your site, then publish the machine-readable files it is missing. Local only, no account, no cloud.
Rawi MD — Markdown for AI Crawlers
Give ChatGPT, Claude and other AI crawlers a clean Markdown copy of your posts, pages and products. Nothing changes for your visitors.
AI Mention Visibility
See which AI crawlers (GPTBot, ClaudeBot, Perplexity and more) reach your site, and the visitors AI sends you. Verified, and free.
Block old browser versions and suspicious browsers
With the help of WhatIsMyBrowser the plugin detects old and bad browsers and denies them access. A special robots.txt denies crawling by bad bots.
Trakkr Crawler Tracking
Record requests from known AI crawler user agents and view them in Trakkr. No front-end JavaScript is added.