Log Architecture & Structural Overview

The AWS CloudFront Standard CDN Access Log Parser is an essential telemetry stream within the AWS Cloud Infrastructure ecosystem. Parse AWS CloudFront CDN distribution access logs. Extract edge server locations, cache hit/miss statuses, and response latencies.

This schema defines a structure of 17 extracted attributes, including 2 numeric metrics and 15 string dimensions. In production observability architectures, these tokens provide high-cardinality indexing keys for telemetry pipelines before shipping to storage backends such as ClickHouse, Elasticsearch, Amazon S3, or Datadog.

Raw Telemetry Ingestion Profile

A typical raw event line for aws-cloudfront-standard averages 166 bytes across 17 tokens. Modern collectors such as Fluent Bit and Vector require zero-backtracking regular expressions to avoid CPU spikes during traffic surges.

Extracted Field Schema & Data Types

The transpiled Grok pattern extracts the following schema fields from each raw event line. Data collectors cast these values according to the typed mappings below.

Field Name	Inferred Type	Description & Collector Semantics
year	string	Year of event.
month	string	Month of event.
day	string	Day of event.
time	string	UTC timestamp HH:MM:SS.
edge_location	string	CloudFront edge point-of-presence (PoP) code.
bytes_sent	integer	Total bytes sent by CloudFront to the viewer.
client_ip	string	IP address of the client viewer.
method	string	HTTP method verb.
host	string	CloudFront distribution domain name.
uri	string	Requested URI stem path.
status	integer	HTTP status code sent to viewer.
referrer	string	HTTP Referer header value or hyphen.
user_agent	string	Client User-Agent header value or hyphen.
query	string	Query string portion of request or hyphen.
cookie	string	Cookies passed in request.
edge_result_type	string	Cache classification (Hit, Miss, RefreshHit, Error).
edge_request_id	string	Opaque unique CloudFront request trace ID.

Common Regex Traps & Production Edge Cases

Engineers frequently encounter ingestion failures or pipeline drops due to subtle variations in real-world event logs. Watch out for these verified pitfalls:

1CloudFront logs use tab delimiters (\t). Regex engines must preserve tab literals or handle \t escapes.

2The first two lines of CloudFront log files in S3 are comment headers (#Version and #Fields); filter them before regex matching.

3Fields with no value are recorded as hyphens (-). Use (?:%{TYPE}|-) to prevent parse failures.

Production Collector Setup & Configurations

Pre-configured parser definitions ready to be dropped into your infrastructure repository.

Fluent Bit (parsers.conf)

Format: regex

# ==============================================================================
# Fluent Bit Parser Configuration (parsers.conf)
# ==============================================================================
[PARSER]
    Name        logpipeline_parser
    Format      regex
    Regex       ^(?<year>\b[0-9]{4}\b)-(?<month>(?:0?[1-9]|1[0-2]))-(?<day>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))\t(?<time>(?:(?:(?:2[0123]|[01]?[0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))))\t(?<edge_location>\S+)\t(?:(?<bytes_sent>(?:[+-]?(?:[0-9]+)))|-)\t(?<client_ip>(?:(?:(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))))|(?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(?:\.?|\b))))\t(?<method>\b\w+\b)\t(?<host>\S+)\t(?<uri>\S+)\t(?<status>(?:[+-]?(?:[0-9]+)))\t(?:(?<referrer>\S+)|-)\t(?:(?<user_agent>.*?)|-)\t(?:(?<query>\S+)|-)\t(?:(?<cookie>.*?)|-)\t(?<edge_result_type>\b\w+\b)\t(?<edge_request_id>\S+)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S%z
    Types       bytes_sent:integer status:integer

# ==============================================================================
# Fluent Bit Pipeline Filter (fluent-bit.conf)
# ==============================================================================
[FILTER]
    Name         parser
    Match        *
    Key_Name     log
    Parser       logpipeline_parser
    Reserve_Data On

Vector.dev (Remap VRL)

parse_regex!

# ==============================================================================
# Vector.dev Remap Language (VRL) Transform
# Use inside a 'remap' transform in vector.yaml
# ==============================================================================
.parsed, err = parse_regex(.message, r'^(?<year>\b[0-9]{4}\b)-(?<month>(?:0?[1-9]|1[0-2]))-(?<day>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))\t(?<time>(?:(?:(?:2[0123]|[01]?[0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))))\t(?<edge_location>\S+)\t(?:(?<bytes_sent>(?:[+-]?(?:[0-9]+)))|-)\t(?<client_ip>(?:(?:(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))))|(?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(?:\.?|\b))))\t(?<method>\b\w+\b)\t(?<host>\S+)\t(?<uri>\S+)\t(?<status>(?:[+-]?(?:[0-9]+)))\t(?:(?<referrer>\S+)|-)\t(?:(?<user_agent>.*?)|-)\t(?:(?<query>\S+)|-)\t(?:(?<cookie>.*?)|-)\t(?<edge_result_type>\b\w+\b)\t(?<edge_request_id>\S+)$')

if err == null {
    . = merge(., .parsed)
    del(.parsed)

    # Type coercions
    .bytes_sent = to_int!(.bytes_sent)
    .status = to_int!(.status)

} else {
    log("LogPipeline parsing warning: " + err, level: "warn")
}

# ==============================================================================
# vector.yaml Pipeline Component
# ==============================================================================
transforms:
  parse_logs:
    type: remap
    inputs: ["source_logs"]
    source: |
      .parsed, err = parse_regex(.message, r'^(?<year>\b[0-9]{4}\b)-(?<month>(?:0?[1-9]|1[0-2]))-(?<day>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))\t(?<time>(?:(?:(?:2[0123]|[01]?[0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))))\t(?<edge_location>\S+)\t(?:(?<bytes_sent>(?:[+-]?(?:[0-9]+)))|-)\t(?<client_ip>(?:(?:(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))))|(?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(?:\.?|\b))))\t(?<method>\b\w+\b)\t(?<host>\S+)\t(?<uri>\S+)\t(?<status>(?:[+-]?(?:[0-9]+)))\t(?:(?<referrer>\S+)|-)\t(?:(?<user_agent>.*?)|-)\t(?:(?<query>\S+)|-)\t(?:(?<cookie>.*?)|-)\t(?<edge_result_type>\b\w+\b)\t(?<edge_request_id>\S+)$')
      if err == null {
        . = merge(., .parsed)
        del(.parsed)
      }

Datadog Log Pipeline Grok Parser

match_rules

# ==============================================================================
# Datadog Log Processing Pipeline Grok Parser
# Navigate to: Logs -> Configuration -> Pipelines -> Add Processor -> Grok Parser
# ==============================================================================

# Match Rule:
rule %{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}\t%{TIME:time}\t%{NOTSPACE:edge_location}\t(?:%{INT:bytes_sent}|-)\t%{IPORHOST:client_ip}\t%{WORD:method}\t%{NOTSPACE:host}\t%{NOTSPACE:uri}\t%{INT:status}\t(?:%{NOTSPACE:referrer}|-)\t(?:%{DATA:user_agent}|-)\t(?:%{NOTSPACE:query}|-)\t(?:%{DATA:cookie}|-)\t%{WORD:edge_result_type}\t%{NOTSPACE:edge_request_id}

# Complete Datadog Pipeline Processor JSON:
{
  "type": "grok-parser",
  "name": "LogPipeline Grok Parser",
  "is_enabled": true,
  "source": "message",
  "samples": [],
  "grok": {
    "match_rules": "rule %{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}\\t%{TIME:time}\\t%{NOTSPACE:edge_location}\\t(?:%{INT:bytes_sent}|-)\\t%{IPORHOST:client_ip}\\t%{WORD:method}\\t%{NOTSPACE:host}\\t%{NOTSPACE:uri}\\t%{INT:status}\\t(?:%{NOTSPACE:referrer}|-)\\t(?:%{DATA:user_agent}|-)\\t(?:%{NOTSPACE:query}|-)\\t(?:%{DATA:cookie}|-)\\t%{WORD:edge_result_type}\\t%{NOTSPACE:edge_request_id}",
    "support_rules": ""
  }
}

# Target Fields Created:
# year (string), month (string), day (string), time (string), edge_location (string), bytes_sent (integer), client_ip (string), method (string), host (string), uri (string), status (integer), referrer (string), user_agent (string), query (string), cookie (string), edge_result_type (string), edge_request_id (string)

OpenTelemetry Collector (transform processor)

regex_parser

# ==============================================================================
# OpenTelemetry Collector Configuration (otel-collector-config.yaml)
# Option 1: Filelog Receiver with regex_parser Operator
# ==============================================================================
receivers:
  filelog:
    include: [ /var/log/**/*.log ]
    start_at: beginning
    operators:
      - type: regex_parser
        id: logpipeline_regex_parser
        regex: '^(?<year>\b[0-9]{4}\b)-(?<month>(?:0?[1-9]|1[0-2]))-(?<day>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))\t(?<time>(?:(?:(?:2[0123]|[01]?[0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))))\t(?<edge_location>\S+)\t(?:(?<bytes_sent>(?:[+-]?(?:[0-9]+)))|-)\t(?<client_ip>(?:(?:(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))))|(?:\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(?:\.?|\b))))\t(?<method>\b\w+\b)\t(?<host>\S+)\t(?<uri>\S+)\t(?<status>(?:[+-]?(?:[0-9]+)))\t(?:(?<referrer>\S+)|-)\t(?:(?<user_agent>.*?)|-)\t(?:(?<query>\S+)|-)\t(?:(?<cookie>.*?)|-)\t(?<edge_result_type>\b\w+\b)\t(?<edge_request_id>\S+)$'
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S%z'

# ==============================================================================
# Option 2: Transform Processor (OTel Transformation Language - OTTL)
# ==============================================================================
processors:
  transform:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - merge_maps(attributes, extract_patterns(body, "^(?<year>\\b[0-9]{4}\\b)-(?<month>(?:0?[1-9]|1[0-2]))-(?<day>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))\\t(?<time>(?:(?:(?:2[0123]|[01]?[0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))))\\t(?<edge_location>\\S+)\\t(?:(?<bytes_sent>(?:[+-]?(?:[0-9]+)))|-)\\t(?<client_ip>(?:(?:(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))))|(?:\\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(?:\\.?|\\b))))\\t(?<method>\\b\\w+\\b)\\t(?<host>\\S+)\\t(?<uri>\\S+)\\t(?<status>(?:[+-]?(?:[0-9]+)))\\t(?:(?<referrer>\\S+)|-)\\t(?:(?<user_agent>.*?)|-)\\t(?:(?<query>\\S+)|-)\\t(?:(?<cookie>.*?)|-)\\t(?<edge_result_type>\\b\\w+\\b)\\t(?<edge_request_id>\\S+)$"), "insert")

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [transform]
      exporters: [otlp]

Logstash Filter Configuration

filter.grok

# ==============================================================================
# Logstash Pipeline Configuration (/etc/logstash/conf.d/logpipeline.conf)
# ==============================================================================
filter {
  grok {
    match => { "message" => "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}\t%{TIME:time}\t%{NOTSPACE:edge_location}\t(?:%{INT:bytes_sent:integer}|-)\t%{IPORHOST:client_ip}\t%{WORD:method}\t%{NOTSPACE:host}\t%{NOTSPACE:uri}\t%{INT:status:integer}\t(?:%{NOTSPACE:referrer}|-)\t(?:%{DATA:user_agent}|-)\t(?:%{NOTSPACE:query}|-)\t(?:%{DATA:cookie}|-)\t%{WORD:edge_result_type}\t%{NOTSPACE:edge_request_id}" }
    tag_on_failure => [ "_grokparsefailure" ]
  }

  date {
    match => [ "time", "ISO8601", "dd/MMM/yyyy:HH:mm:ss Z" ]
    target => "@timestamp"
    remove_field => [ "time" ]
  }
}