{"id":1528,"date":"2020-06-07T21:55:13","date_gmt":"2020-06-07T13:55:13","guid":{"rendered":"https:\/\/www.wesbytes.com\/guide\/?post_type=kb&#038;p=1528"},"modified":"2026-01-12T14:42:00","modified_gmt":"2026-01-12T06:42:00","slug":"im-getting-google-crawler-errors-what-should-i-do","status":"publish","type":"kb","link":"https:\/\/www.servergigabit.com\/guide\/kb\/im-getting-google-crawler-errors-what-should-i-do","title":{"rendered":"I\u2019m getting Google Crawler Errors. What should I do?"},"content":{"rendered":"<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h5 class=\"serp-title\">I\u2019m getting\u00a0<mark>Google Crawler Errors<\/mark>. What should I do?<\/h5>\n<h2 class=\"mkb-anchor__title\">General guidelines<\/h2>\n<\/div>\n<p>Review the\u00a0following guidelines if you\u2019re experiencing crawler errors in the Google Webmaster Tools Health dashboard.<\/p>\n<ul>\n<li>Monitor the performance and availability of your website directly to rule out any errors produced by your own web server.\u00a0There are a number of free and paid services which you can use:\n<ul>\n<li>Pingdom<\/li>\n<li>StatusCake<\/li>\n<li>Monitor.Us<\/li>\n<li>updown<\/li>\n<\/ul>\n<\/li>\n<li>Do not block Google crawler IPs in the <a href=\"https:\/\/www.cloudflare.com\/\" rel=\"nofollow noopener\" target=\"_blank\">Cloudflare<\/a> Threat Control Panel (you can verify whether an address belongs to Google using the method they mention in\u00a0this article). Blocking IPs in their\u00a0range\u00a0will prevent Google from crawling your website.<\/li>\n<li>Do not block the United States as a country in the Cloudflare Threat Control Panel. Many search engines are based in the United States and this setting could prevent them from crawling your website.<\/li>\n<li>Do not block Google\u2019s User-Agents in your\u00a0.htaccess, server configuration,\u00a0robots.txt, or web application. Google may use a variety of User-Agents to crawl your website. Please review this\u00a0list of common User-Agents Google uses when crawling\u00a0your website. When making modifications, you can test your robots.txt file on the, \u201cBlocked URLs page<strong>\u201c<\/strong>\u00a0under the, \u201cHealth\u201d section<strong>\u00a0<\/strong>in Google Webmaster Tools.<\/li>\n<li>Do not allow\u00a0crawling of files in the \/cdn-cgi\/ directory. This path is used internally by Cloudflare and Google will encounter errors when crawling it.<\/li>\n<\/ul>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h2 class=\"mkb-anchor__title\"><\/h2>\n<\/div>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h2 class=\"mkb-anchor__title\">Troubleshooting<\/h2>\n<\/div>\n<p>Troubleshooting steps for the most commonly reported errors are below. For anything not mentioned here, please follow the steps in the additional troubleshooting section and then open a support ticket.<\/p>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\"><\/div>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h2 class=\"mkb-anchor__title\">HTTP 4xx Errors<\/h2>\n<p>&nbsp;<\/p>\n<\/div>\n<p><em>Google detected a significant increase in the number of URLs we were blocked from crawling due to authorization permission errors.<\/em><\/p>\n<p>HTTP errors in the\u00a0400 range\u00a0are the most common type of error users will see in Google Webmaster Tools Health Dashboard. Errors in this range indicate a request failure. Cloudflare acts as a messenger and delivers these errors from your web server to Google. These\u00a0errors may be caused by something as simple as a missing page\u00a0on your web server or a malformed link in your HTML. The solution will depend upon the problem encountered. Once you have identified and corrected the problem, Google will make additional attempts to crawl your website and errors will clear automatically.<\/p>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h2 class=\"mkb-anchor__title\"><\/h2>\n<\/div>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h2 class=\"mkb-anchor__title\">HTTP 5xx Errors<\/h2>\n<\/div>\n<p>HTTP errors in the\u00a0500 range, indicate that Cloudflare or your web server experienced an internal error.\u00a0Unfortunately, the Google Webmaster Tools Health Dashboard provides a limited diagnostic view into these kinds of errors.\u00a0If your web server is producing errors in the 500 range, it can be challenging to isolate these errors from errors that Cloudflare may be producing. The best solution is to monitor the health of your web server to help\u00a0correlate dates of any outages with dates of errors.<\/p>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h2 class=\"mkb-anchor__title\"><\/h2>\n<\/div>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h2 class=\"mkb-anchor__title\">Increased \u201cTime spent downloading a page\u201d<\/h2>\n<\/div>\n<p>File a Cloudflare\u00a0Support Ticket\u00a0if you are experiencing increased page download times after enabling Cloudflare. Please include the diagnostic output that the process below will generate in your inquiry.<\/p>\n<p>1. \u00a0 \u00a0Login to your Google Webmaster Tools account and navigate to the Health section of the affected domain.<\/p>\n<p>2. \u00a0 \u00a0Click,\u00a0<strong>Fetch as Google<\/strong>\u00a0in the left hand navigation.<\/p>\n<p>3. \u00a0 \u00a0Type,\u00a0<em>cdn-cgi\/trace<\/em>\u00a0in lowercase in the form field and then click\u00a0<strong>FETCH<\/strong>.<\/p>\n<p>4. \u00a0 \u00a0Click,\u00a0<strong>Success<\/strong>\u00a0under the\u00a0<strong>Fetch Status<\/strong> column.<\/p>\n<p>5. \u00a0 \u00a0Copy all of the text in the grey area and submit this to Cloudflare\u00a0Customer Support.<\/p>\n<p>&nbsp;<\/p>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h2 class=\"mkb-anchor__title\">DNS Errors<\/h2>\n<\/div>\n<p><em>Over the last 24 hours, Googlebot encountered a number\u00a0errors while attempting to retrieve DNS information for your site. The overall error rate for DNS queries for your site is\u00a0percent.<\/em><\/p>\n<p>Most DNS failures will be handled gracefully by browsers and your visitors will not be impacted. Steps for further troubleshooting will vary depending on how your Cloudflare service is configured. Cloudflare can be enabled either through usage of a CNAME or through zone delegation. If you are using Cloudflare through a hosting provider, you are likely using a CNAME setup.<\/p>\n<p>To verify which configuration you\u2019re using, open a terminal and execute the following command. Replace\u00a0<em>www.example.com<\/em>\u00a0with your domain registered with Cloudflare.<\/p>\n<pre>dig +short SOA www.example.com<\/pre>\n<p>If you get output from the command similar to the text below, you are using a CNAME setup. Please proceed to the CNAME section.<\/p>\n<pre>example.com.cdn.cloudflare.net.<\/pre>\n<p>If you get output from the command similar to the text below, you are using Cloudflare through zone delegation. Please proceed to the Delegation section.<\/p>\n<pre>josh.ns.cloudflare.com. dns.cloudflare.com. 2013050901 10000 2400 604800 3600<\/pre>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h3 class=\"mkb-anchor__title\"><\/h3>\n<\/div>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h3 class=\"mkb-anchor__title\">CNAME<\/h3>\n<\/div>\n<p>In a CNAME setup, your hosting provider is primarily responsible for your DNS service. In order to investigate DNS errors, you will need to contact their technical support team. Provide them with the date that Google encountered DNS errors to help with their investigation. Additionally, you should view the\u00a0Cloudflare System Status\u00a0page and observe any network outages on the date the errors were encountered by Google.<\/p>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h3 class=\"mkb-anchor__title\">Delegation<\/h3>\n<\/div>\n<p>At all plan levels, Cloudflare will provide at least two authoritative name server records for your domain. Browsers and operating system APIs handle failure of all but one of these nameservers gracefully without exposing an error to your visitors. If Google reports DNS errors to you often, please file a Cloudflare\u00a0Support Ticket\u00a0and provide the date and time that Google observed the errors.<\/p>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h2 class=\"mkb-anchor__title\">Additional troubleshooting<\/h2>\n<\/div>\n<p>Follow the following steps below to export crawler errors as a .csv file from your Google Webmaster Tools Dashboard. Include this file\u00a0when reporting errors to Cloudflare Technical Support.<\/p>\n<p>1. \u00a0 \u00a0Login to your Google Webmaster Tools account and navigate to the Health section of the affected domain.<\/p>\n<p>2. \u00a0 \u00a0Click,\u00a0<strong>Crawl Errors<\/strong>\u00a0in the left hand navigation.<\/p>\n<p>3. \u00a0 \u00a0Click,\u00a0<strong>Download<\/strong> to export the list of errors as a .csv file.<\/p>\n<p>4. \u00a0 \u00a0Attach the downloaded .csv file to the Cloudflare Support Ticket.<\/p>\n<p>&nbsp;<\/p>\n<p><span data-sheets-root=\"1\">Please refer to the following article to know more.<br \/>\nKnowledge Base: <a href=\"https:\/\/www.servergigabit.com\/guide\/kb\/how-do-i-block-bots-and-crawlers\">How do I block bots and crawlers?<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I\u2019m getting\u00a0Google Crawler Errors. What should I do? General guidelines Review the\u00a0following guidelines if you\u2019re experiencing crawler errors in the Google Webmaster Tools Health dashboard. Monitor the performance and availability of your website directly to rule out any errors produced by your own web server.\u00a0There are a number of free and paid services which you can use: Pingdom StatusCake Monitor.Us&hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"kbtopic":[43],"kbtag":[1441],"class_list":["post-1528","kb","type-kb","status-publish","hentry","kbtopic-cloudflare","kbtag-google-crawler-errors"],"_links":{"self":[{"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kb\/1528","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kb"}],"about":[{"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/types\/kb"}],"author":[{"embeddable":true,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/comments?post=1528"}],"version-history":[{"count":4,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kb\/1528\/revisions"}],"predecessor-version":[{"id":6077,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kb\/1528\/revisions\/6077"}],"wp:attachment":[{"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/media?parent=1528"}],"wp:term":[{"taxonomy":"kbtopic","embeddable":true,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kbtopic?post=1528"},{"taxonomy":"kbtag","embeddable":true,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kbtag?post=1528"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}