I’m getting a crawl error on Cloudflare cdn-cgi files.
Crawl error on Cloudflare cdn-cgi files.
If you are getting a crawl error in Google Webmaster Tools for a cdn-cgi file, please disallow Google from crawling that with a rule in robots.txt for:
Disallow: /cdn-cgi/
An additional option using WordPress:
Solution provided by Ben
You can make modifications in your /wp-includes/functions.php file. to create an additional Disallow rule. A sample of what it would look like is below:
function do_robots() {
header( ‘Content-Type: text/plain; charset=utf-8’ );
do_action( ‘do_robotstxt’ );
$output = “User-agent: *\n”;
$public = get_option( ‘blog_public’ );
if ( ‘0’ == $public ) {
$output .= “Disallow: /\n”;
} else {
$site_url = parse_url( site_url() );
$path = ( !empty( $site_url[‘path’] ) ) ? $site_url[‘path’] : ”;
$output .= “Disallow: $path/wp-admin/\n”;
$output .= “Disallow: $path/wp-includes/\n”;
$output .= “Disallow: $path/cdn-cgi/\n”;
}
Note: This won’t impact the actual crawl for your site at all. The error for /cdn-cgi/ is also not going to impact your rankings at all. Google is attempting to crawl a Cloudflare resource. One that doesn’t need to be crawl, and they possibly can’t read the DOM or JavaScript in that resource.
Please refer to the following article to know more.
Knowledge Base: I’m getting Google Crawler Errors. What should I do?