Crawlers
18 plugins tagged with crawlers
Block AI Crawlers
Tell AI (Artificial Intelligence) companies not to scrape your site for their AI products.
Robots.txt rewrite
Provide the easy managment of your robots.txt from admin side. It propose you the advanced then standard robots.txt content too.
AI Content Signals
Control how AI uses your content — search, AI answers or training — via robots.txt Content Signals, HTTP headers, meta tags and TDMRep.
CodingBunny LLMs.txt Generator
Advanced llms.txt generator with AI metadata, Schema.org types, canonical URLs, and differentiated licenses for optimal LLM indexing.
Robots Meta Whiz
Easy way to discourage search engines from indexing only specific pages / posts with custom meta tags.
MetaRobots by SEO-Sign
The easiest way to manage meta robots tag.
Serve Markdown
Serve Markdown versions of your posts and pages to AI agents and crawlers. Content negotiation, .md URLs, auto-discovery, and crawler logging.
ParseLess
Serve your posts as Markdown to AI crawlers and CLI tools — typically 95%+ less bandwidth, ~60–80% fewer DB queries, no theme bloat.
CeeWP Robots.txt Manager
Easily manage your WordPress site's virtual robots.txt file with custom rules. No physical file needed!
Seo Crawlytics Data Export & Delete
Exports all entries of Seo Crawlytics and gives you the option to delete the entries that are older than 1 month.
Log-Hero.com Plugin
Analyze how search engines and other bots crawl and understand your web page. The official PHP Wordpress plugin for log-hero.com.
LJPL Armored robots.txt
Add some directives to your robots.txt file to keep your site safer
Botkibble
Serves every published post and page as Markdown for AI agents and crawlers. No configuration, no API keys. Activate and it works.
Supertab Connect
Connect your WordPress site to the Supertab Connect platform for RSL license serving and enabling crawler authentication protocol.
Temso AI
Stream front-end request logs from your WordPress origin to Temso so AI-crawler and bot traffic shows up in your dashboard.
Coywolf Robots.txt Manager
Manage robots.txt from the WordPress admin: build rules, block bots by category, and serve a physical or virtual robots.txt.
PufferSights – AI Crawler Insights
Monitor 100+ known AI crawler and AI agent user agents with local analytics, AI referrals, llms.txt, and opt-in robots.txt policy tools.
AI Rights Protection by SimpleFeed
Protect your rights by reviewing and blocking AI bots from using your content without compensation.