LinksIndexerBot

What does the LinksIndexerBot do?

LinksIndexerBot is our web crawler which is a very important tool in our sitemap campaigns - since our service is a website URL indexing and crawling, we need to automatically parse third-party sites to verify their URLs and status. It is an indispensable part of our technology that aggregates site data into a brief URL profile for every site - that usually includes querying your site for metadata, favicon and making a screenshot of its homepage and some other actions. Such site profiles support the way our service works. To be fully transparent: LinksIndexerBot never harvests any e-mail addresses or content that is not related to sitemap campaigns.

We want our crawler to be as 'polite' as possible (sending a minimal number of queries to your site) but if it causes you any issues, please let us know via the contact form and provide any info that might be helpful.

How do I prevent the LinksIndexerBot from visiting my site?

Our crawler will obey any standard-conforming rule you provide in your robots.txt file. To disallow LinksIndexerBot visiting and parsing your site, you can put the following lines into your robots.txt file:

User-agent: LinksIndexerBot Disallow: /

This will result in our crawler visiting your site only once and not returning any time soon (accessing just this one file to execute your robots policy). LinksIndexerBot generally obeys such robots.txt directives as: Allow, Disallow, Crawl-delay, Host. You may look for specifications of these directives on http://www.robotstxt.org.

LinksIndexerBot keeps crawling your site despite the robots.txt instructions

If LinksIndexerBot visits your website too frequently and ignores the robots.txt commands, please contact us.

What information does LinksIndexerBot collect?

LinksIndexerBot is a web crawler used exclusively for our sitemap campaigns and URL indexing service. It collects only the information necessary to verify URLs and their status, including page metadata, favicon information, and a screenshot of the homepage. It does not harvest e-mail addresses or content unrelated to sitemap campaigns. For more details on how we process this information, please review our Privacy Policy.

IP Addresses and Crawl Behavior

LinksIndexerBot may operate from multiple IP addresses or ranges, which may change over time due to infrastructure changes. While we strive to maintain polite crawl rates (minimum 10 seconds between requests to the same host), you can specify a preferred crawl delay in your robots.txt file using the Crawl-delay directive. We honor this directive where technically feasible.