How to Whitelist Links Indexer Bot on Cloudflare?

Bots do a lot of the web’s behind-the-scenes work. Search engines use them to index pages and monitoring tools use them to check whether your site is up, but plenty of other bots exist just to probe, spam, or overload websites.

If your site sits behind Cloudflare, its default security settings will happily block the good bots along with the bad ones. When that happens to a bot you actually want, like the Links Indexer Bot, you have to allow it explicitly through the Cloudflare WAF (Web Application Firewall). This guide walks you through exactly how to do that.

Introduction

Cloudflare’s security features are good at keeping unwanted traffic out, but they don’t always know which bots are welcome. The Links Indexer Bot crawls your site’s links so your URLs get indexed. If Cloudflare challenges or blocks it, the bot simply can’t do its job. Adding one rule that lets it through fixes that.

Step 1

Log into your Cloudflare account and find the Security tab in the left menu. Click on WAF.

Step 2

This opens the WAF (Web Application Firewall) options. Go to the Firewall rules tab.

Step 3

Click on Custom Rules. This is where Cloudflare keeps its firewall rules. Then click the Create rule button.

Step 4

Name the rule:

• In the Rule name field, enter “Links Indexer Bot” so you know what this rule does later.

Define the matching condition:

• In the If incoming requests match… section, set up the following condition:
• Field: Select User Agent.
• Operator: Select contains.
• Value: Enter LinksIndexerBot/1.0.

Review the expression preview:

• Check that the Expression Preview reads: (http.user_agent contains "LinksIndexerBot/1.0").

Set the action to take:

• In the Then take action… section, choose Skip.
• Under WAF components to skip, select All Super Bot Fight Mode Rules

Step 5

Once you fill out the form, click Deploy in the bottom-right corner.

That’s it. Your Cloudflare firewall now lets the LinksIndexerBot through instead of challenging it, so indexing runs without interruptions. The rest of the crawling and indexing is Links Indexer’s job, not yours.