Crawlers
14 plugins tagged with crawlers
Block AI Crawlers
Tell AI (Artificial Intelligence) companies not to scrape your site for their AI products.
Robots.txt rewrite
Provide the easy managment of your robots.txt from admin side. It propose you the advanced then standard robots.txt content too.
AI Content Signals
Control how AI crawlers use your content with Cloudflare Content Signals in robots.txt. Includes EU Directive 2019/790 rights reservation.
CodingBunny LLMs.txt Generator
Advanced llms.txt generator with AI metadata, Schema.org types, canonical URLs, and differentiated licenses for optimal LLM indexing.
Robots Meta Whiz
Easy way to discourage search engines from indexing only specific pages / posts with custom meta tags.
MetaRobots by SEO-Sign
The easiest way to manage meta robots tag.
CeeWP Robots.txt Manager
Easily manage your WordPress site's virtual robots.txt file with custom rules. No physical file needed!
Serve Markdown
Serve Markdown versions of your posts and pages to AI agents and crawlers. Content negotiation, .md URLs, auto-discovery, and crawler logging.
Seo Crawlytics Data Export & Delete
Exports all entries of Seo Crawlytics and gives you the option to delete the entries that are older than 1 month.
Log-Hero.com Plugin
Analyze how search engines and other bots crawl and understand your web page. The official PHP Wordpress plugin for log-hero.com.
LJPL Armored robots.txt
Add some directives to your robots.txt file to keep your site safer
Botkibble
Serves every published post and page as Markdown for AI agents and crawlers. No configuration, no API keys. Activate and it works.
Supertab Connect
Connect your WordPress site to the Supertab Connect platform for RSL license serving and enabling crawler authentication protocol.
AI Rights Protection by SimpleFeed
Protect your rights by reviewing and blocking AI bots from using your content without compensation.