Focus Page Crawling Error On Websites With CloudFlare

When encountering a 403 Forbidden error during Focus Page crawling for audit, especially when using Cloudflare, the most common factor is factors is the Bot Protection and WAF Rules.

Cloudflare’s Web Application Firewall (WAF) is designed to protect websites from malicious traffic, including bots. If your crawler’s behavior triggers WAF rules (such as excessive requests or suspicious patterns), Cloudflare may block access with a 403 error.

To whitelist your IP address in Cloudflare’s Web Application Firewall (WAF) and avoid getting a 403 error during URL crawling, follow these steps:

1. Log into your Cloudflare account.
2. Select the site where you want to whitelist (allow) your IP.
3. In the left sidebar, click Security > WAF.
4. Navigate to the Tools tab.
5. Under IP Access Rules, add the IP address 5.161.83.61:
   • Click on the field for IP or IP range.
   • Enter your IP address 5.161.83.61.
   • Select Allow, All Websites in Account.
6. Click to add the rule and allow Squirrly Crawler to access the website and pass the bot protection.