Crawler
32 plugins tagged with crawler
Email Address Encoder
A lightweight plugin that protects email addresses from email-harvesting robots, by encoding them into decimal and hexadecimal entities.
Robots.txt Editor
Robots.txt for WordPress
Spider Analyser – WordPress搜索引擎蜘蛛分析插件
Spider Analyser是一款用于跟踪WordPress网站各种搜索引擎蜘蛛爬行日志的插件，并进行详细的蜘蛛爬行数据统计、蜘蛛行为分析、蜘蛛爬取分析及伪蜘蛛拦截等。
No-Bot Registration
Prevent bots from creating accounts by blacklisting domains and usernames and present people with a human friendly security question.
Robots.txt rewrite
Provide the easy managment of your robots.txt from admin side. It propose you the advanced then standard robots.txt content too.
Marquee image crawler
Marquee image crawler is a continuous scrolling image plugin. This plugin crawls the images left or right.
Mass Ping Tool for SEO – WordPress ping list to get indexed faster on Google, Yandex, …
Mass ping tool for SEO plugin is one of the fastest ways to get indexed by Google. After pinging, Google bots, crawlers will visit your website and in …
Block Archive.org via WordPress robots.txt
Blocks the archive.org bots through the WordPress virtual robots.txt file.
Bisteinoff SEO Robots.txt
An easy-to-use plugin that generates and configures a proper robots.txt file, essential for effective search engine optimization (SEO).
bodi0`s Bots visits counter
Counts the visits from web spiders, crawlers and bots in your blog, with ability to get the blog rankings.
Block Chat GPT via robots.txt
Blocks the ChatGPT-User bot that is used by plugins in ChatGPT to crawl websites through the WordPress virtual robots.txt file
Block eCommerce Assets via robots.txt
Blocks some WooCommerce assets and Search pages that should not be indexed through robots.txt
AI Deny
Easily block over 30 of the most common AI user-agents known for crawling websites, keeping unauthorized bots at bay.
Warmer
Automatically warm up your WordPress site's cache by crawling pages to ensure optimal performance for your visitors.
Findkit – Site Search
Powerful cloud based search engine for your website. Search as you type, real-time index updates, and embeddable UI. Free for small websites.
Block Common Crawl via robots.txt
Blocks the Common Crawl bot (CCBot) that gathers data that is used to train AI algorithms like ChatGPT through the WordPress virtual robots.txt file
Block Feed and Comments via robots.txt
Blocks some Feed and Comment resources in order to avoid spam bots
Azayem Bots Tracker – Bot Visits Logger
A custom plugin to log search engine bot visits and analyze their crawl budget directly inside WordPress (no external services required).
JaspreetChahal’s wordpress bot detector lite
This plugin detects a few bots (e.g. Google, Google ads, Alta vista etc) and sends an email based on interval set by you. you can choose what format e …
SEObot Monitor for Googlebot, Bingbot and search engine spiders
With SEObot Monitor for Googlebot you will be able to dump the server logs to Google Analytics, easily and automatically. This will allow you to dete …
Page Modified
Display the results of your Page Modified crawls right within your WordPress Admin.
Robots.txt Extender
Dynamic robots.txt for Multisite! Change parameters, or don't, for each site of your network without losing the defaults from WordPress Includes.
Valvai – AI Traffic Control Center
Block, log and monetise AI crawlers directly from your WordPress dashboard.
AI Bot Monitor
Tracks and logs AI, Search and Social bots including ChatGPT, Perplexity, Googlebot, Bing and Meta crawlers.