{"id":1459,"date":"2020-06-07T21:20:58","date_gmt":"2020-06-07T13:20:58","guid":{"rendered":"https:\/\/www.wesbytes.com\/guide\/?post_type=kb&p=1459"},"modified":"2026-01-12T14:57:02","modified_gmt":"2026-01-12T06:57:02","slug":"im-getting-a-crawl-error-on-cloudflare-cdn-cgi-files","status":"publish","type":"kb","link":"https:\/\/www.servergigabit.com\/guide\/kb\/im-getting-a-crawl-error-on-cloudflare-cdn-cgi-files","title":{"rendered":"I\u2019m getting a crawl error on Cloudflare cdn-cgi files."},"content":{"rendered":"

\"Cloudflare<\/h3>\n

I\u2019m getting a crawl error on Cloudflare cdn-cgi files.<\/strong><\/h3>\n

Crawl error on Cloudflare<\/a> cdn-cgi files.<\/p>\n

If you are getting a crawl error in Google Webmaster Tools for a cdn-cgi file, please disallow Google from crawling that with a rule in robots.txt for:<\/p>\n

Disallow: \/cdn-cgi\/<\/code><\/p>\n

An additional option using WordPress:<\/strong><\/p>\n

Solution provided by Ben
\n<\/strong><\/p>\n

You can make modifications in your \/wp-includes\/functions.php file. to create an additional Disallow rule. A sample of what it would look like is below:<\/p>\n

function do_robots() {
\nheader( \u2018Content-Type: text\/plain; charset=utf-8\u2019 );<\/p>\n

do_action( \u2018do_robotstxt\u2019 );<\/p>\n

$output = \u201cUser-agent: *\\n\u201d;
\n$public = get_option( \u2018blog_public\u2019 );
\nif ( \u20180\u2019 == $public ) {
\n$output .= \u201cDisallow: \/\\n\u201d;
\n} else {
\n$site_url = parse_url( site_url() );
\n$path = ( !empty( $site_url[\u2018path\u2019] ) ) ? $site_url[\u2018path\u2019] : \u201d;
\n$output .= \u201cDisallow: $path\/wp-admin\/\\n\u201d;
\n$output .= \u201cDisallow: $path\/wp-includes\/\\n\u201d;
\n$output .= \u201cDisallow: $path\/cdn-cgi\/\\n\u201d;
\n}<\/p>\n

 <\/p>\n

Note: This won\u2019t impact the actual crawl for your site at all. The error for \/cdn-cgi\/ is also not going to impact your rankings at all. Google is attempting to crawl a Cloudflare resource. One that doesn\u2019t need to be crawl, and they possibly can\u2019t read the DOM or JavaScript in that resource.<\/p>\n

 <\/p>\n

Please refer to the following article to know more.
\nKnowledge Base: I\u2019m getting Google Crawler Errors. What should I do?<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

I\u2019m getting a crawl error on Cloudflare cdn-cgi files. Crawl error on Cloudflare cdn-cgi files. If you are getting a crawl error in Google Webmaster Tools for a cdn-cgi file, please disallow Google from crawling that with a rule in robots.txt for: Disallow: \/cdn-cgi\/ An additional option using WordPress: Solution provided by Ben You can make modifications in your \/wp-includes\/functions.php…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"kbtopic":[43],"kbtag":[1477],"class_list":["post-1459","kb","type-kb","status-publish","hentry","kbtopic-cloudflare","kbtag-cloudflare-cdn-cgi-files"],"_links":{"self":[{"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kb\/1459","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kb"}],"about":[{"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/types\/kb"}],"author":[{"embeddable":true,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/comments?post=1459"}],"version-history":[{"count":4,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kb\/1459\/revisions"}],"predecessor-version":[{"id":6102,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kb\/1459\/revisions\/6102"}],"wp:attachment":[{"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/media?parent=1459"}],"wp:term":[{"taxonomy":"kbtopic","embeddable":true,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kbtopic?post=1459"},{"taxonomy":"kbtag","embeddable":true,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kbtag?post=1459"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}