In today's digital landscape, bots play a crucial role in various aspects of online activities, from search engine indexing to monitoring website health. However, not all bots are beneficial; some can be harmful, causing security risks and performance issues.
If you are using Cloudflare for website security and optimization, you may need to whitelist specific bots to ensure they are not blocked by default security settings. This guide will walk you through the steps to whitelist the Links Indexer Bot on Cloudflare WAF (Web Application Firewall).
Cloudflare offers robust security features that protect your website from various threats, but sometimes these settings can block legitimate bots that are beneficial for your site. One such bot is the Links Indexer Bot, which helps index your website URLs by crawling your website's links. To ensure it can perform its job without being blocked, you need to whitelist it in your Cloudflare WAF settings.
Log into your Cloudflare account and find the Security tab in the left menu. Click on the WAF.
This will open the WAF (Web Application Firewall) security options. Navigate to the Firewall rules tab.
Click on Custom Rules. This section allows us to create and manage Cloudflare Firewall rules. Then click on Create rulebutton.
Set the rule name:
Define the matching condition:
LinksIndexerBot/1.0
.Review the expression preview:
(http.user_agent contains "LinksIndexerBot/1.0")
.Set the action to take:
Once you fill out the form, click Deploy in the bottom-right corner.
Congratulations! You've successfully configured your Cloudflare firewall to whitelist the LinksIndexerBot, ensuring it can access your site without encountering security blocks. Now, sit back and let Links Indexer handle the rest of the indexing and crawling. 🚀
By Pushkraj Jori