Millions of pages.
Findings you can fix.

Bot identity

Match the private key and User-Agent together. IP allowlists alone are not reliable because crawl traffic may use distributed or rotating egress.

User-Agent: Mozilla/5.0 (compatible; LoudSEOBot/1.0; +https://pageperfect.ai/bot)
LoudSEOBot-Key: <PROVIDED PRIVATELY BY LOUD>
loud-int: true

Recommended hardening: restrict the exception to GET and HEAD, exclude authentication/checkout/admin paths, and apply an agreed per-origin ceiling. Crawl Perfect itself currently performs page requests with GET.

Cloudflare WAF

1. Open the Cloudflare dashboard, select the domain, then choose Security → WAF → Custom rules.
2. Create a custom rule named Allow verified LoudSEOBot.
3. Use the expression below, replacing the placeholder with the key Loud provided privately.
4. Choose Skip. Select managed rules, rate limiting rules, Browser Integrity Check, Security Level, and Managed Challenge as appropriate for your policy.
5. Deploy the rule above broader challenge/block rules, then tell Loud it is ready for validation.

(http.request.headers["loudseobot-key"][0] eq "<PROVIDED PRIVATELY BY LOUD>") and
(http.user_agent contains "LoudSEOBot/") and
(http.request.method in {"GET" "HEAD"})

Shopify

Shopify limits direct server configuration, so check both the theme's robots template and any third-party security or bot-blocking app.

1. In Shopify Admin, go to Online Store → Themes, open the active theme menu, and choose Edit code.
2. Open templates/robots.txt.liquid. If it does not exist, create a custom robots template only after reviewing Shopify's generated defaults.
3. Ensure LoudSEOBot is not blocked. Retain exclusions for admin, accounts, carts, and checkout.
4. Review installed security, anti-bot, and fraud apps. Add LoudSEOBot using its UA and private key where supported.
5. Shopify Plus clients can ask their Merchant Success or support contact for infrastructure-level allowlisting.

User-agent: LoudSEOBot
Allow: /
Disallow: /admin/
Disallow: /account/
Disallow: /cart
Disallow: /checkout/

AWS WAF, Akamai, F5, and other platforms

Create a high-priority positive-match rule requiring all of these conditions:

• LoudSEOBot-Key exactly equals the key provided privately.
• User-Agent contains LoudSEOBot/.
• Method is GET or HEAD.

Set the action to allow or bypass bot/challenge controls. Keep application authorization controls in force and exclude sensitive paths. In AWS WAF, use an AND statement with byte-match statements for the header, UA, and method. In Akamai Bot Manager or F5, use the equivalent positive header match and bypass only the bot challenge layer.

Nginx and Apache

These examples create a verified flag. Connect that flag to your existing WAF/rate-limit configuration; exact directives vary by stack.

Nginx

map $http_user_agent $is_loudseobot {
    default 0;
    ~*LoudSEOBot/ 1;
}
map "$is_loudseobot:$http_loudseobot_key" $loud_bot_verified {
    default 0;
    "1:<PROVIDED PRIVATELY BY LOUD>" 1;
}
# Use $loud_bot_verified in your WAF/rate-limit policy to bypass
# bot challenges for GET and HEAD only. Keep auth/admin paths excluded.

Apache

SetEnvIf User-Agent ".*LoudSEOBot/.*" is_loudseobot
SetEnvIfExpr "%{HTTP:LoudSEOBot-Key} == '<PROVIDED PRIVATELY BY LOUD>' && reqenv('is_loudseobot') == '1'" loud_bot_verified
# Reference loud_bot_verified in your WAF or ModSecurity allow/skip rule.
# Limit the exception to GET/HEAD and exclude auth, checkout and admin.

Validate before scaling

1. Deploy the rule in staging first when available, then production.
2. Loud runs a small smoke crawl across representative templates.
3. Confirm 200 responses and zero CAPTCHA, 403, 429, or managed-challenge responses in WAF logs.
4. Agree on the initial ceiling and crawl window.
5. Scale while Crawl Perfect's adaptive controller monitors origin and pipeline health.

Success: sustained throughput, complete page coverage, zero challenge pages, and no measurable impact on real users.