How to Whitelist Links Indexer Bot on Cloudflare?

In today's digital landscape, bots play a crucial role in various aspects of online activities, from search engine indexing to monitoring website health. However, not all bots are beneficial; some can be harmful, causing security risks and performance issues.

If you are using Cloudflare for website security and optimization, you may need to whitelist specific bots to ensure they are not blocked by default security settings. This guide will walk you through the steps to whitelist the Links Indexer Bot on Cloudflare WAF (Web Application Firewall).

Introduction

Cloudflare offers robust security features that protect your website from various threats, but sometimes these settings can block legitimate bots that are beneficial for your site. One such bot is the Links Indexer Bot, which helps index your website URLs by crawling your website's links. To ensure it can perform its job without being blocked, you need to whitelist it in your Cloudflare WAF settings.

Step 1

Log into your Cloudflare account and find the Security tab in the left menu. Click on the WAF.

Cloudflare > Security > WAF

Step 2

This will open the WAF (Web Application Firewall) security options. Navigate to the Firewall rules tab.

Step 3

Click on Custom Rules. This section allows us to create and manage Cloudflare Firewall rules. Then click on Create rulebutton.

Cloudflare Create Rule Button

Step 4

Set the rule name:

• In the Rule name field, enter "Links Indexer Bot" for a descriptive name.

Define the matching condition:

• In the If incoming requests match… section, set up the following condition:
• Field: Select User Agent.
• Operator: Select contains.
• Value: Enter LinksIndexerBot/1.0.

Review the expression preview:

• Ensure the Expression Preview shows: (http.user_agent contains "LinksIndexerBot/1.0").

Set the action to take:

• In the Then take action… section, choose Skip.
• Under WAF components to skip, select All Super Bot Fight Mode Rules

Cloudflare Rules Field Preview

Step 5

Once you fill out the form, click Deploy in the bottom-right corner.

Congratulations! You've successfully configured your Cloudflare firewall to whitelist the LinksIndexerBot, ensuring it can access your site without encountering security blocks. Now, sit back and let Links Indexer handle the rest of the indexing and crawling. 🚀