Crawlers
40 plugins tagged with crawlers
Block AI Crawlers
Adds robots.txt rules blocking AI crawlers and AI bots.
Robots.txt rewrite
Provide the easy managment of your robots.txt from admin side. It propose you the advanced then standard robots.txt content too.
AI Content Signals
Control how AI uses your content — search, AI answers or training — via robots.txt Content Signals, HTTP headers, meta tags and TDMRep.
CodingBunny LLMs.txt Generator
Advanced llms.txt generator with AI metadata, Schema.org types, canonical URLs, and differentiated licenses for optimal LLM indexing.
Robots Meta Whiz
Easy way to discourage search engines from indexing only specific pages / posts with custom meta tags.
MetaRobots by SEO-Sign
The easiest way to manage meta robots tag.
Serve Markdown
Serve Markdown versions of your posts and pages to AI agents and crawlers. Content negotiation, .md URLs, auto-discovery, and crawler logging.
ParseLess
Serve AI the good version of your content, see which bots really read it, and cap what they cost you. Markdown for crawlers and agents — 95%+ less ban …
Log-Hero.com Plugin
See which search engines, AI crawlers and other bots visit your site, and what they find. The official plugin for log-hero.com.
CeeWP Robots.txt Manager
Easily manage your WordPress site's virtual robots.txt file with custom rules. No physical file needed!
AMDT AI Traffic Analytics
See which AI bots crawl your site, whether they are who they claim to be, and who arrives from ChatGPT, Claude or Perplexity. Publishes llms.txt.
Temso AI
Stream front-end request logs from your WordPress origin to Temso so AI-crawler and bot traffic shows up in your dashboard.
Seo Crawlytics Data Export & Delete
Exports all entries of Seo Crawlytics and gives you the option to delete the entries that are older than 1 month.
OtterlyAI Agent Analytics
Reports every AI crawler and bot request on your site to OtterlyAI Agent Analytics. Runs server-side, in the background.
MSG LLMs TXT Manager
Serve your content as clean Markdown to AI agents, publish llms.txt and llms-full.txt, and see which agents actually fetch what.
LJPL Armored robots.txt
Add some directives to your robots.txt file to keep your site safer
Rawi MD — Markdown for AI Crawlers
Give ChatGPT, Claude and other AI crawlers a clean Markdown copy of your posts, pages and products. Nothing changes for your visitors.
PufferSights – AI Crawler Insights
Monitor 135+ known AI crawlers and agents with local analytics, AI referrals, llms.txt, and opt-in robots.txt policy tools.
Arbling Beacon
See which AI assistants send visitors to your site, and which AI crawlers read it, in your Arbling dashboard.
CLEO by RegenAI
See which AI systems read your pages. Requires a free or paid Cleo account.
Presswell Content Extractor
Serve AI-agent-friendly Markdown and an llms.txt index from your WordPress content.
getReport
See which AI crawlers visit (verified), which AI assistants send you visitors, and how quotable every post is. Plus one-click fixes with Undo. Free.
Block old browser versions and suspicious browsers
With the help of WhatIsMyBrowser the plugin detects old and bad browsers and denies them access. A special robots.txt denies crawling by bad bots.
AI Mention Visibility
See which AI crawlers (GPTBot, ClaudeBot, Perplexity and more) reach your site, and the visitors AI sends you. Verified, and free.