{"id":1529,"date":"2020-06-07T21:59:49","date_gmt":"2020-06-07T13:59:49","guid":{"rendered":"https:\/\/www.wesbytes.com\/guide\/?post_type=kb&#038;p=1529"},"modified":"2026-01-12T14:40:57","modified_gmt":"2026-01-12T06:40:57","slug":"using-google-cloud-platform-to-analyze-cloudflare-logs","status":"publish","type":"kb","link":"https:\/\/www.servergigabit.com\/guide\/kb\/using-google-cloud-platform-to-analyze-cloudflare-logs","title":{"rendered":"Using Google Cloud Platform to Analyze Cloudflare Logs"},"content":{"rendered":"<h4><strong>Using Google Cloud Platform to Analyze Cloudflare Logs<\/strong><\/h4>\n<p>This tutorial covers how to configure certain <a href=\"https:\/\/cloud.google.com\/?_gl=1*19dsao4*_up*MQ..&amp;gclid=Cj0KCQiAvOjKBhC9ARIsAFvz5lia0I_CTdw-NywphQqcavTLR-QlEpAtZXQbKPcGNOlnQjsrM9XVdW0aAulsEALw_wcB&amp;gclsrc=aw.ds\" rel=\"nofollow noopener\" target=\"_blank\">Google Cloud Platform<\/a> (GCP) components so that you can analyze your Cloudflare Logs data.<\/p>\n<p>Before proceeding, you need to enable\u00a0Cloudflare Logpush in Google Cloud Storage\u00a0to ensure your log data is available for analyzing.<\/p>\n<p>The components we\u2019ll use in this tutorial include :<\/p>\n<ul>\n<li><strong>Google Cloud Function<\/strong>\u00a0to import logs from Google Cloud Storage to Google BigQuery<\/li>\n<li><strong>Google BigQuery<\/strong>\u00a0to make log data available to the reporting engine, and<\/li>\n<li><strong>Google Data Studio<\/strong>\u00a0to run interactive reports<\/li>\n<\/ul>\n<p>The following diagram depicts how data flows from Cloudflare Logs through the different components of the Google Cloud Platform discussed in this tutorial.<\/p>\n<p>Google Cloud is offering a credit towards a new Google Cloud account to help you get start. To learn more, visit Google Cloud Platform Partner Credit.<\/p>\n<hr \/>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h3 class=\"mkb-anchor__title\">Task 1 \u2013 Use Google Cloud Function to import log data into Google BigQuery<\/h3>\n<\/div>\n<p>After you configured Cloudflare Logpush to send your logs to a Google Cloud Storage bucket, your log data updates every five minutes by default.<\/p>\n<p>Google BigQuery makes data available for both querying using Structured Query Language (SQL) and for configuring as a data source for the Google Data Studio reporting engine. BigQuery is a highly scalable cloud database where SQL queries run quite fast.<\/p>\n<p>Importing data from Google Cloud Storage into Google BigQuery requires creating a function using Google Cloud Function and running it in the Google Cloud Shell. This function triggers every time new Cloudflare log data is upload to your Google Cloud Storage bucket.<\/p>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h4 class=\"mkb-anchor__title\">Clone and deploy a Google Cloud Function<\/h4>\n<p>&nbsp;<\/p>\n<\/div>\n<p>To a create a cloud function to import data from Google Cloud Storage into Google BigQuery, you will need the following GitHub repository from Cloudflare:\u00a0https:\/\/github.com\/cloudflare\/GCS-To-Big-Query.<\/p>\n<p>To clone and deploy the cloud function:<\/p>\n<p>1. \u00a0 \u00a0Run the Google Cloud Platform shell by opening the\u00a0<strong>Google Cloud Platform\u00a0<\/strong>console and clicking the\u00a0<strong>Google Shell<\/strong>\u00a0icon (<em>Activate Cloud Shell<\/em>).<\/p>\n<p>2. \u00a0 \u00a0Run the following command to download the\u00a0<em>master<\/em>\u00a0zipped archive, uncompress the files to new a directory, and change the command line prompt to the new directory:<\/p>\n<p><code>\u00a0 curl -LO \"https:\/\/github.com\/cloudflare\/GCS-To-Big-Query\/archive\/master.zip\" &amp;&amp; unzip master.zip &amp;&amp; cd GCS-To-Big-Query-master\u00a0\u00a0<\/code><\/p>\n<p>3. \u00a0 \u00a0Next, edit the\u00a0<code>deploy.sh<\/code>\u00a0file and make sure that:<\/p>\n<p>a.\u00a0\u00a0<strong>BUCKET_NAME<\/strong>\u00a0is set to the bucket you create when you configure Cloudflare Logpush with Google Cloud Platform.<\/p>\n<p>b.\u00a0\u00a0<strong>DATASET<\/strong>\u00a0and\u00a0<strong>TABLE<\/strong>\u00a0are unique names.<\/p>\n<p>The contents of\u00a0<code>deploy.sh<\/code>\u00a0should look similar to this:<\/p>\n<pre><code data-lang=\"bashj\">    .    .    .    BUCKET_NAME=\"my_cloudflarelogs_gcp_storage_bucket\"    DATASET=\"my_cloudflare_logs\"    TABLE=\"cloudflare_logs\"    .    .    .    <\/code><\/pre>\n<p>4. \u00a0 \u00a0Then in the\u00a0<strong>Google Shell<\/strong>, run the following command to deploy your instance of the cloud function:<\/p>\n<pre><code data-lang=\"bash\">    sh .\/deploy.sh    <\/code><\/pre>\n<p>Once you\u2019ve deployed your new cloud function, verify that it appears in the\u00a0<strong>Cloud Functions<\/strong>\u00a0interface by navigating to\u00a0<strong>Google Cloud Platform<\/strong>\u00a0&gt;\u00a0<strong>Compute<\/strong>\u00a0&gt;\u00a0<strong>Cloud Functions<\/strong>.<\/p>\n<p>Also, verify that the data now appears in your table in\u00a0<strong>BigQuery<\/strong>\u00a0by navigating to the appropriate project in\u00a0<strong>Google Cloud Platform<\/strong>\u00a0&gt;\u00a0<strong>Big Data<\/strong>\u00a0&gt;\u00a0<strong>BigQuery<\/strong>.<\/p>\n<p>If everything is configure correctly, you can now query any request or visualize data with Google Data Studio or any other analytics tool that supports BigQuery as an input source.<\/p>\n<hr \/>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h3 class=\"mkb-anchor__title\">Task 2 \u2013 Analyze log data with Google Data Studio<\/h3>\n<\/div>\n<p>To analyze and visualize logs, you can use\u00a0<strong>Google Data Studio<\/strong>\u00a0or any other 3rd party services that supports Google BigQuery as an input source.<\/p>\n<p>With Google Data Studio, you can generate graphs and charts from a Google BigQuery table. You can also refresh the data in your reports and get real-time analytics.<\/p>\n<p>&nbsp;<\/p>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h4 class=\"mkb-anchor__title\">About the Cloudflare Logs Insights Template<\/h4>\n<\/div>\n<p>Cloudflare has published a\u00a0Logs Insights Template\u00a0in the\u00a0<strong>Google Data Studio Report Gallery<\/strong>.<\/p>\n<p>&nbsp;<\/p>\n<p>The Cloudflare Insights Template features several dashboards, or report pages, to help you analyze your Cloudflare Logs data. You can also use filters within the dashboards to narrow down the analysis by date and time, device type, country, user agent, client IP, hostname, and more. These insights further help with debugging and tracing.<\/p>\n<p>The following dashboards are include in the Insights template:<\/p>\n<ul>\n<li><strong>Snapshot<\/strong>: Gives you an overview of the most important metrics from your Cloudflare logs, including total number of requests, top visitors by geography, IP, user agent, traffic type, total number of threats, and bandwidth usage.<\/li>\n<li><strong>Security<\/strong>: Provides insights on threat identification and mitigations by our\u00a0<strong>Web Application Firewall<\/strong>, including\u00a0<strong>Firewall Rules<\/strong>,\u00a0<strong>Rate Limiting<\/strong>, and\u00a0<strong>IP Firewall<\/strong>. Metrics include total threats stopped, threat traffic source, blocked IPs and user agents, top threat requests, Firewall events (SQL injections, XSS, etc.), and rate limiting. Use this data to fine tune the firewall to target obvious threats and avoid false positives.<\/li>\n<li><strong>Performance<\/strong>: Helps you identify and address issues like slow pages and caching misconfigurations. Metrics include total vs. cached bandwidth, cache ratio, top uncached requests, static vs. dynamic content, slowest URIs, and more.<\/li>\n<li><strong>Reliability<\/strong>: Provides insights on the availability of your websites and applications. Metrics include origin response error ratio, origin response status over time, percentage of 3xx\/4xx\/5xx errors over time, and more.<\/li>\n<\/ul>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h4 class=\"mkb-anchor__title\"><\/h4>\n<\/div>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h4 class=\"mkb-anchor__title\">Create a report based on the Insights Template<\/h4>\n<\/div>\n<p>To create a report for your log data based on the Cloudflare template:<\/p>\n<p>1. \u00a0 \u00a0In Data Studio, open the Cloudflare\u00a0template\u00a0and click\u00a0<strong>Use Template<\/strong>. A\u00a0<em>Create new report<\/em>\u00a0dialog opens.<\/p>\n<p>2. \u00a0 \u00a0Under the\u00a0<strong>New Data Source<\/strong>\u00a0dropdown, select\u00a0<strong>Create New Data Source<\/strong>. A page opens where you can enter additional configuration details.<\/p>\n<p>3. \u00a0 \u00a0Under\u00a0<strong>Google Connectors<\/strong>, locate the\u00a0<strong>BigQuery<\/strong>\u00a0card and click\u00a0<strong>Select<\/strong>.<\/p>\n<p>4. \u00a0 \u00a0Next under\u00a0<strong>MY PROJECTS<\/strong>, select your\u00a0<strong>Project<\/strong>,\u00a0<strong>Dataset<\/strong>, and\u00a0<strong>Table<\/strong>.<\/p>\n<p>5. \u00a0 \u00a0Click\u00a0<strong>Connect<\/strong>\u00a0in the upper right.<\/p>\n<p>6. \u00a0 \u00a0In the list of Cloudflare Logs fields, locate\u00a0<em>EdgeStartTimestamp<\/em>, click the three vertical dots and select\u00a0<strong>Duplicate<\/strong>. This creates\u00a0<em>Copy of EdgeStartTimestamp<\/em>right below\u00a0<em>EdgeStartTimestamp<\/em>.<\/p>\n<p>7. \u00a0 \u00a0Update the\u00a0<strong>Type<\/strong>\u00a0for\u00a0<em>Copy of EdgeStartTimestamp<\/em>\u00a0to set it to\u00a0<em>Date &amp; Time<\/em>\u00a0&gt;<em>Date Hour (YYYYMMDDHH)<\/em>.<\/p>\n<p>8. \u00a0 \u00a0Next, update the\u00a0<strong>Type<\/strong>\u00a0for each of the following fields as indicated below:<\/p>\n<table>\n<thead>\n<tr>\n<th>Cloudflare Log Field<\/th>\n<th>Type<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>ZoneID<\/td>\n<td>Text<\/td>\n<\/tr>\n<tr>\n<td>EdgeColoID<\/td>\n<td>Text<\/td>\n<\/tr>\n<tr>\n<td>ClientSrcPort<\/td>\n<td>Text<\/td>\n<\/tr>\n<tr>\n<td>EdgeResponseStatus<\/td>\n<td>Number<\/td>\n<\/tr>\n<tr>\n<td>EdgeRateLimitID<\/td>\n<td>Text<\/td>\n<\/tr>\n<tr>\n<td>Copy of EdgeStartTimestamp<\/td>\n<td>Date &amp; Time &gt; Date Hour (YYYYMMDDHH)<\/td>\n<\/tr>\n<tr>\n<td>OriginResponseStatus<\/td>\n<td>Number<\/td>\n<\/tr>\n<tr>\n<td>ClientASN<\/td>\n<td>Text<\/td>\n<\/tr>\n<tr>\n<td>ClientCountry<\/td>\n<td>Geo &gt; Country<\/td>\n<\/tr>\n<tr>\n<td>CacheResponseStatus<\/td>\n<td>Text<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>9. \u00a0 \u00a0Next, add a new field to identify and calculate threat. In the top right corner, click\u00a0<strong>+ ADD A FIELD<\/strong>, then in the add field UI:<\/p>\n<p>a. For\u00a0<strong>Field Name<\/strong>, type\u00a0<em>Threats<\/em>.<\/p>\n<p>b. In the\u00a0<strong>Formula<\/strong>\u00a0text box, paste the following code:<\/p>\n<pre>CASE \r\nWHEN EdgePathingSrc = \"user\" AND EdgePathingOp = \"ban\" AND EdgePathingStatus = \"ip\" THEN \"ip block\" \r\nWHEN EdgePathingSrc = \"user\" AND EdgePathingOp = \"ban\" AND EdgePathingStatus = \"ctry\" THEN \"country block\" \r\nWHEN EdgePathingSrc = \"user\" AND EdgePathingOp = \"ban\" AND EdgePathingStatus = \"zl\" THEN \"routed by zone lockdown\"   \r\nWHEN EdgePathingSrc = \"user\" AND EdgePathingOp = \"ban\" AND EdgePathingStatus = \"ua\" THEN \"blocked user agent\" \r\nWHEN EdgePathingSrc = \"user\" AND EdgePathingOp = \"ban\" AND EdgePathingStatus = \"rateLimit\" THEN \"rate-limiting rule\"  \r\nWHEN EdgePathingSrc = \"bic\" AND EdgePathingOp = \"ban\" AND EdgePathingStatus = \"unknown\" THEN \"browser integrity check\" \r\nWHEN EdgePathingSrc = \"hot\" AND EdgePathingOp = \"ban\" AND EdgePathingStatus = \"unknown\" THEN \"blocked hotlink\" \r\nWHEN EdgePathingSrc = \"macro\" AND EdgePathingOp = \"chl\" AND EdgePathingStatus = \"captchaFail\" THEN \"CAPTCHA challenge failed\" WHEN EdgePathingSrc = \"macro\" AND EdgePathingOp = \"chl\" AND EdgePathingStatus = \"jschlFail\" THEN \"java script challenge failed\" \r\nWHEN EdgePathingSrc = \"filterBasedFirewall\" AND EdgePathingOp = \"ban\" AND EdgePathingStatus = \"unknown\" THEN \"blocked by filter based firewall\" \r\nWHEN EdgePathingSrc = \"filterBasedFirewall\" AND EdgePathingOp = \"chl\" THEN \"challenged by filter based firewall\" \r\nElse \"Other\" \r\nEND<\/pre>\n<p>c. Click\u00a0<strong>Save<\/strong>\u00a0in the lower right corner.<\/p>\n<p>10. \u00a0 \u00a0Finally, add another new field for grouping status error codes. In the top right corner, click\u00a0<strong>+ ADD A FIELD<\/strong>, then in the add field UI:<\/p>\n<p>a. For\u00a0<strong>Field Name<\/strong>, type\u00a0<em>EdgeResponseStatus_Class<\/em>.<\/p>\n<p>b. In the\u00a0<strong>Formula<\/strong>\u00a0text box, paste the following code:<\/p>\n<pre>CASE \r\nWHEN EdgeResponseStatus &gt;199 AND  EdgeResponseStatus &lt; 300  THEN \"2xx\" \r\nWHEN EdgeResponseStatus &gt;299 AND  EdgeResponseStatus &lt; 400  THEN \"3xx\" \r\nWHEN EdgeResponseStatus &gt;399 AND  EdgeResponseStatus &lt; 500  THEN \"4xx\" \r\nWHEN EdgeResponseStatus &gt;499 AND  EdgeResponseStatus &lt; 600  THEN \"5xx\" \r\nWHEN EdgeResponseStatus = 0 THEN \"0 - Served from CF Edge\" \r\nElse \"Other\" \r\nEND<\/pre>\n<p>c. Click\u00a0<strong>Save<\/strong>\u00a0in the lower right corner.<\/p>\n<p>11. \u00a0 \u00a0To finish, click\u00a0<strong>Add to Report<\/strong>\u00a0in the upper right.<\/p>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h4 class=\"mkb-anchor__title\"><\/h4>\n<\/div>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h4 class=\"mkb-anchor__title\">Refreshing fields and filters manually<\/h4>\n<p>&nbsp;<\/p>\n<\/div>\n<p>After you\u2019ve added your report, you will notice that not all report components render successfully. To fix this, you need to resolve any errors related to invalid dimensions, metrics, or filters that appear in the affect report components.<\/p>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h5 class=\"mkb-anchor__title\">Fix invalid metric or dimension errors<\/h5>\n<\/div>\n<p>The following table summarizes which specific components require to be fix:<\/p>\n<p>&nbsp;<\/p>\n<table>\n<thead>\n<tr>\n<th>Report page<\/th>\n<th>Components<\/th>\n<th>Field to add<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td rowspan=\"3\">2 Security Cloudflare<\/td>\n<td><em>Threats\u00a0<\/em>(scorecard)<\/td>\n<td>\u00a0Threats (Metric)<\/td>\n<\/tr>\n<tr>\n<td><em>Threats \u2013 Record Count<\/em>\u00a0(table)<\/td>\n<td>\u00a0Threats (Dimension)<\/td>\n<\/tr>\n<tr>\n<td>\u00a0<em>Threats Over Time\u00a0<\/em>(area chart)<\/td>\n<td>\u00a0Threats (Breadown Dimension)<\/td>\n<\/tr>\n<tr>\n<td>3 Reliability Cloudflare<\/td>\n<td><em>Status Codes Last 24 hours\u00a0<\/em>(bar chart)<\/td>\n<td><em>Copy of EdgeStartTimeStamp\u00a0<\/em>(Dimension)<\/td>\n<\/tr>\n<tr>\n<td>5 Last 100s Requests Cloudflare<\/td>\n<td><em>Last 100 Requests<\/em>\u00a0(table)<\/td>\n<td>\u00a0<em>Copy of EdgeStartTimeStamp<\/em><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>For each of the report components listed above:<\/p>\n<ol>\n<li>Select the report component affected.<\/li>\n<li>On the menu to the right, under the\u00a0<strong>Data<\/strong>\u00a0tab, locate and click\u00a0<strong>Invalid Dimension<\/strong>\u00a0or\u00a0<strong>Invalid Metric<\/strong>\u00a0(as applicable). The\u00a0<strong>Field Picker<\/strong>\u00a0panel opens.<\/li>\n<li>Search or type for the field to add, then click to select it.<\/li>\n<li>To finish, click away from the panel to return to the main report.<\/li>\n<\/ol>\n<p>The component should now render correctly.<\/p>\n<div class=\"mkb-anchor mkb-clearfix mkb-back-to-top-inline\">\n<h5 class=\"mkb-anchor__title\">Update data filters<\/h5>\n<\/div>\n<p>This fix applies to report page:\u00a0<strong>3 Reliability Cloudflare<\/strong>, for the following scorecard components in the report:<\/p>\n<ul>\n<li><strong>5xx Errors<\/strong><\/li>\n<li><strong>4xx Errors<\/strong><\/li>\n<li><strong>3xx Errors<\/strong><\/li>\n<\/ul>\n<p>To update the filter associated with each scorecard:<\/p>\n<ol>\n<li>Select the report component affected.<\/li>\n<li>On the menu to the right, under the\u00a0<strong>Data<\/strong>\u00a0tab, locate the\u00a0<strong>Filters<\/strong>\u00a0&gt;\u00a0<strong>Scorecard Filter<\/strong>\u00a0section and click the\u00a0<strong>pencil<\/strong>\u00a0next to the filter name to edit it. The\u00a0<strong>Edit Filter<\/strong>panel opens.<\/li>\n<li>In the filtering criteria section, click the dropdown and scroll or search for the field\u00a0<em>EdgeResponseStatus_Class<\/em>\u00a0and select it.<\/li>\n<li>To finish, click\u00a0<strong>Save<\/strong>\u00a0in the lower right corner.<\/li>\n<\/ol>\n<p>The component should now render correctly.<\/p>\n<p>&nbsp;<\/p>\n<p>Please refer to the following article to know more.<br \/>\nKnowledge Base: <a href=\"https:\/\/www.servergigabit.com\/guide\/kb\/cloudflare-logs-formerly-els\">Cloudflare Logs (formerly ELS)<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Using Google Cloud Platform to Analyze Cloudflare Logs This tutorial covers how to configure certain Google Cloud Platform (GCP) components so that you can analyze your Cloudflare Logs data. Before proceeding, you need to enable\u00a0Cloudflare Logpush in Google Cloud Storage\u00a0to ensure your log data is available for analyzing. The components we\u2019ll use in this tutorial include : Google Cloud Function\u00a0to&hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"kbtopic":[43],"kbtag":[1439,1440],"class_list":["post-1529","kb","type-kb","status-publish","hentry","kbtopic-cloudflare","kbtag-cloudflare-logs","kbtag-google-cloud-platform"],"_links":{"self":[{"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kb\/1529","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kb"}],"about":[{"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/types\/kb"}],"author":[{"embeddable":true,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/comments?post=1529"}],"version-history":[{"count":4,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kb\/1529\/revisions"}],"predecessor-version":[{"id":6076,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kb\/1529\/revisions\/6076"}],"wp:attachment":[{"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/media?parent=1529"}],"wp:term":[{"taxonomy":"kbtopic","embeddable":true,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kbtopic?post=1529"},{"taxonomy":"kbtag","embeddable":true,"href":"https:\/\/www.servergigabit.com\/guide\/wp-json\/wp\/v2\/kbtag?post=1529"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}