AWS Cloud Infrastructure100% Verified RegexZero-Allocation Web Worker

AWS Application Load Balancer (ALB) Access Log Parser

Debug and parse AWS ALB access logs. Generate verified Fluent Bit, Vector VRL, and Datadog Grok pipelines. Test pattern matching, inspect named capture groups, and export production-ready parser definitions across Fluent Bit, Vector VRL, Datadog Pipelines, Logstash, and OpenTelemetry.

Live Interactive Debugger & Generator

Matches execute locally in-browser via Web Worker

Interactive Test & Config Generator Sandbox

Extraction Pattern (Grok / PCRE Expression)

Pattern Valid (20 fields)

Detected Fields:typetimestampelbclient_ipclient_port:integertarget_iptarget_port:integerrequest_processing_time:floattarget_processing_time:floatresponse_processing_time:floatelb_status_code:integertarget_status_code:integerreceived_bytes:integersent_bytes:integermethodrequest_urlhttp_versionuser_agentssl_cipherssl_protocol

Raw Log Stream Sandbox(0/3 matched)

No log lines provided. Paste lines or select a preset above.

No matching lines available to generate JSON output.

AdvertisementActive Viewability 30s

Log Architecture & Structural Overview

The AWS Application Load Balancer (ALB) Access Log Parser is an essential telemetry stream within the AWS Cloud Infrastructure ecosystem. Debug and parse AWS ALB access logs. Generate verified Fluent Bit, Vector VRL, and Datadog Grok pipelines.

This schema defines a structure of 20 extracted attributes, including 9 numeric metrics and 11 string dimensions. In production observability architectures, these tokens provide high-cardinality indexing keys for telemetry pipelines before shipping to storage backends such as ClickHouse, Elasticsearch, Amazon S3, or Datadog.

Raw Telemetry Ingestion Profile

A typical raw event line for aws-alb-access-logs averages 256 bytes across 20 tokens. Modern collectors such as Fluent Bit and Vector require zero-backtracking regular expressions to avoid CPU spikes during traffic surges.

Extracted Field Schema & Data Types

The transpiled Grok pattern extracts the following schema fields from each raw event line. Data collectors cast these values according to the typed mappings below.

Field Name	Inferred Type	Description & Collector Semantics
type	string	Request protocol scheme (http, https, h2, ws, wss).
timestamp	string	ISO-8601 completion timestamp in UTC.
elb	string	Resource ID of the Application Load Balancer.
client_ip	string	IP address of the requesting client.
client_port	integer	Port of the requesting client.
target_ip	string	IP address of the target container or EC2 instance.
target_port	integer	Port of the target backend service.
request_processing_time	float	Seconds elapsed from ALB receiving request to sending to target.
target_processing_time	float	Seconds elapsed from ALB sending request to receiving header from target.
response_processing_time	float	Seconds elapsed from ALB receiving header to sending response to client.
elb_status_code	integer	HTTP response code generated by ALB.
target_status_code	integer	HTTP response code returned by target backend.
received_bytes	integer	Bytes received from client (including headers).
sent_bytes	integer	Bytes sent to client (including headers).
method	string	HTTP method verb (GET, POST, PUT, DELETE).
request_url	string	Full requested URI string.
http_version	string	HTTP protocol version.
user_agent	string	Client User-Agent header string.
ssl_cipher	string	SSL/TLS negotiation cipher.
ssl_protocol	string	SSL/TLS negotiated protocol version.

Common Regex Traps & Production Edge Cases

Engineers frequently encounter ingestion failures or pipeline drops due to subtle variations in real-world event logs. Watch out for these verified pitfalls:

1Target IP can be '-' if the ALB intercepted and responded directly (e.g. 403 or redirect). Use optional regex wrappers when handling direct ALB intercepts.

2gRPC requests through ALB have special status codes; ensure elb_status_code parsing supports custom downstream error mappings.

3HTTP/2 multiplexing may alter timing precision; ensure float parsers do not truncate microsecond precision.

Production Collector Setup & Configurations

Pre-configured parser definitions ready to be dropped into your infrastructure repository.

Fluent Bit (parsers.conf)

Format: regex

# ==============================================================================
# Fluent Bit Parser Configuration (parsers.conf)
# ==============================================================================
[PARSER]
    Name        logpipeline_parser
    Format      regex
    Regex       ^(?<type>\S+) (?<timestamp>(?:\b[0-9]{4}\b)-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01]?[0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?) (?<elb>\S+) (?<client_ip>(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)))):(?<client_port>(?:[+-]?(?:[0-9]+))) (?<target_ip>(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)))):(?<target_port>(?:[+-]?(?:[0-9]+))) (?<request_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) (?<target_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) (?<response_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) (?<elb_status_code>(?:[+-]?(?:[0-9]+))) (?<target_status_code>(?:[+-]?(?:[0-9]+))) (?<received_bytes>(?:[+-]?(?:[0-9]+))) (?<sent_bytes>(?:[+-]?(?:[0-9]+))) "(?<method>\b\w+\b) (?<request_url>\S+) HTTP/(?<http_version>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+)))))" "(?<user_agent>.*?)" (?<ssl_cipher>\S+) (?<ssl_protocol>\S+)$
    Time_Key    timestamp
    Time_Format %Y-%m-%dT%H:%M:%S%z
    Types       client_port:integer target_port:integer request_processing_time:float target_processing_time:float response_processing_time:float elb_status_code:integer target_status_code:integer received_bytes:integer sent_bytes:integer

# ==============================================================================
# Fluent Bit Pipeline Filter (fluent-bit.conf)
# ==============================================================================
[FILTER]
    Name         parser
    Match        *
    Key_Name     log
    Parser       logpipeline_parser
    Reserve_Data On

Vector.dev (Remap VRL)

parse_regex!

# ==============================================================================
# Vector.dev Remap Language (VRL) Transform
# Use inside a 'remap' transform in vector.yaml
# ==============================================================================
.parsed, err = parse_regex(.message, r'^(?<type>\S+) (?<timestamp>(?:\b[0-9]{4}\b)-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01]?[0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?) (?<elb>\S+) (?<client_ip>(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)))):(?<client_port>(?:[+-]?(?:[0-9]+))) (?<target_ip>(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)))):(?<target_port>(?:[+-]?(?:[0-9]+))) (?<request_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) (?<target_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) (?<response_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) (?<elb_status_code>(?:[+-]?(?:[0-9]+))) (?<target_status_code>(?:[+-]?(?:[0-9]+))) (?<received_bytes>(?:[+-]?(?:[0-9]+))) (?<sent_bytes>(?:[+-]?(?:[0-9]+))) "(?<method>\b\w+\b) (?<request_url>\S+) HTTP/(?<http_version>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+)))))" "(?<user_agent>.*?)" (?<ssl_cipher>\S+) (?<ssl_protocol>\S+)$')

if err == null {
    . = merge(., .parsed)
    del(.parsed)

    # Type coercions
    .client_port = to_int!(.client_port)
    .target_port = to_int!(.target_port)
    .request_processing_time = to_float!(.request_processing_time)
    .target_processing_time = to_float!(.target_processing_time)
    .response_processing_time = to_float!(.response_processing_time)
    .elb_status_code = to_int!(.elb_status_code)
    .target_status_code = to_int!(.target_status_code)
    .received_bytes = to_int!(.received_bytes)
    .sent_bytes = to_int!(.sent_bytes)

} else {
    log("LogPipeline parsing warning: " + err, level: "warn")
}

# ==============================================================================
# vector.yaml Pipeline Component
# ==============================================================================
transforms:
  parse_logs:
    type: remap
    inputs: ["source_logs"]
    source: |
      .parsed, err = parse_regex(.message, r'^(?<type>\S+) (?<timestamp>(?:\b[0-9]{4}\b)-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01]?[0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?) (?<elb>\S+) (?<client_ip>(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)))):(?<client_port>(?:[+-]?(?:[0-9]+))) (?<target_ip>(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)))):(?<target_port>(?:[+-]?(?:[0-9]+))) (?<request_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) (?<target_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) (?<response_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) (?<elb_status_code>(?:[+-]?(?:[0-9]+))) (?<target_status_code>(?:[+-]?(?:[0-9]+))) (?<received_bytes>(?:[+-]?(?:[0-9]+))) (?<sent_bytes>(?:[+-]?(?:[0-9]+))) "(?<method>\b\w+\b) (?<request_url>\S+) HTTP/(?<http_version>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+)))))" "(?<user_agent>.*?)" (?<ssl_cipher>\S+) (?<ssl_protocol>\S+)$')
      if err == null {
        . = merge(., .parsed)
        del(.parsed)
      }

Datadog Log Pipeline Grok Parser

match_rules

# ==============================================================================
# Datadog Log Processing Pipeline Grok Parser
# Navigate to: Logs -> Configuration -> Pipelines -> Add Processor -> Grok Parser
# ==============================================================================

# Match Rule:
rule %{NOTSPACE:type} %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb} %{IP:client_ip}:%{INT:client_port} %{IP:target_ip}:%{INT:target_port} %{NUMBER:request_processing_time} %{NUMBER:target_processing_time} %{NUMBER:response_processing_time} %{INT:elb_status_code} %{INT:target_status_code} %{INT:received_bytes} %{INT:sent_bytes} "%{WORD:method} %{NOTSPACE:request_url} HTTP/%{NUMBER:http_version}" "%{DATA:user_agent}" %{NOTSPACE:ssl_cipher} %{NOTSPACE:ssl_protocol}

# Complete Datadog Pipeline Processor JSON:
{
  "type": "grok-parser",
  "name": "LogPipeline Grok Parser",
  "is_enabled": true,
  "source": "message",
  "samples": [],
  "grok": {
    "match_rules": "rule %{NOTSPACE:type} %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb} %{IP:client_ip}:%{INT:client_port} %{IP:target_ip}:%{INT:target_port} %{NUMBER:request_processing_time} %{NUMBER:target_processing_time} %{NUMBER:response_processing_time} %{INT:elb_status_code} %{INT:target_status_code} %{INT:received_bytes} %{INT:sent_bytes} \"%{WORD:method} %{NOTSPACE:request_url} HTTP/%{NUMBER:http_version}\" \"%{DATA:user_agent}\" %{NOTSPACE:ssl_cipher} %{NOTSPACE:ssl_protocol}",
    "support_rules": ""
  }
}

# Target Fields Created:
# type (string), timestamp (string), elb (string), client_ip (string), client_port (integer), target_ip (string), target_port (integer), request_processing_time (float), target_processing_time (float), response_processing_time (float), elb_status_code (integer), target_status_code (integer), received_bytes (integer), sent_bytes (integer), method (string), request_url (string), http_version (string), user_agent (string), ssl_cipher (string), ssl_protocol (string)

OpenTelemetry Collector (transform processor)

regex_parser

# ==============================================================================
# OpenTelemetry Collector Configuration (otel-collector-config.yaml)
# Option 1: Filelog Receiver with regex_parser Operator
# ==============================================================================
receivers:
  filelog:
    include: [ /var/log/**/*.log ]
    start_at: beginning
    operators:
      - type: regex_parser
        id: logpipeline_regex_parser
        regex: '^(?<type>\S+) (?<timestamp>(?:\b[0-9]{4}\b)-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01]?[0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?) (?<elb>\S+) (?<client_ip>(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)))):(?<client_port>(?:[+-]?(?:[0-9]+))) (?<target_ip>(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)))):(?<target_port>(?:[+-]?(?:[0-9]+))) (?<request_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) (?<target_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) (?<response_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) (?<elb_status_code>(?:[+-]?(?:[0-9]+))) (?<target_status_code>(?:[+-]?(?:[0-9]+))) (?<received_bytes>(?:[+-]?(?:[0-9]+))) (?<sent_bytes>(?:[+-]?(?:[0-9]+))) "(?<method>\b\w+\b) (?<request_url>\S+) HTTP/(?<http_version>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+)))))" "(?<user_agent>.*?)" (?<ssl_cipher>\S+) (?<ssl_protocol>\S+)$'
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%dT%H:%M:%S%z'

# ==============================================================================
# Option 2: Transform Processor (OTel Transformation Language - OTTL)
# ==============================================================================
processors:
  transform:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - merge_maps(attributes, extract_patterns(body, "^(?<type>\\S+) (?<timestamp>(?:\\b[0-9]{4}\\b)-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01]?[0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?) (?<elb>\\S+) (?<client_ip>(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)))):(?<client_port>(?:[+-]?(?:[0-9]+))) (?<target_ip>(?:(?:(?:(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,7}:|(?:[0-9A-Fa-f]{1,4}:){1,6}:[0-9A-Fa-f]{1,4}|(?:[0-9A-Fa-f]{1,4}:){1,5}(?::[0-9A-Fa-f]{1,4}){1,2}|(?:[0-9A-Fa-f]{1,4}:){1,4}(?::[0-9A-Fa-f]{1,4}){1,3}|(?:[0-9A-Fa-f]{1,4}:){1,3}(?::[0-9A-Fa-f]{1,4}){1,4}|(?:[0-9A-Fa-f]{1,4}:){1,2}(?::[0-9A-Fa-f]{1,4}){1,5}|[0-9A-Fa-f]{1,4}:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|:(?:(?::[0-9A-Fa-f]{1,4}){1,7}|:)|fe80:(?::[0-9A-Fa-f]{0,4}){0,4}%[0-9a-zA-Z]+|::(?:ffff(?::0{1,4})?:)?(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])|(?:[0-9A-Fa-f]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])\\.){3}(?:25[0-5]|(?:2[0-4]|1?[0-9])?[0-9])))|(?:(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)))):(?<target_port>(?:[+-]?(?:[0-9]+))) (?<request_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\\.[0-9]+)?|\\.[0-9]+))))) (?<target_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\\.[0-9]+)?|\\.[0-9]+))))) (?<response_processing_time>(?:(?:(?:[+-]?(?:[0-9]+(?:\\.[0-9]+)?|\\.[0-9]+))))) (?<elb_status_code>(?:[+-]?(?:[0-9]+))) (?<target_status_code>(?:[+-]?(?:[0-9]+))) (?<received_bytes>(?:[+-]?(?:[0-9]+))) (?<sent_bytes>(?:[+-]?(?:[0-9]+))) \"(?<method>\\b\\w+\\b) (?<request_url>\\S+) HTTP/(?<http_version>(?:(?:(?:[+-]?(?:[0-9]+(?:\\.[0-9]+)?|\\.[0-9]+)))))\" \"(?<user_agent>.*?)\" (?<ssl_cipher>\\S+) (?<ssl_protocol>\\S+)$"), "insert")

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [transform]
      exporters: [otlp]

Logstash Filter Configuration

filter.grok

# ==============================================================================
# Logstash Pipeline Configuration (/etc/logstash/conf.d/logpipeline.conf)
# ==============================================================================
filter {
  grok {
    match => { "message" => "%{NOTSPACE:type} %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb} %{IP:client_ip}:%{INT:client_port:integer} %{IP:target_ip}:%{INT:target_port:integer} %{NUMBER:request_processing_time:float} %{NUMBER:target_processing_time:float} %{NUMBER:response_processing_time:float} %{INT:elb_status_code:integer} %{INT:target_status_code:integer} %{INT:received_bytes:integer} %{INT:sent_bytes:integer} \"%{WORD:method} %{NOTSPACE:request_url} HTTP/%{NUMBER:http_version}\" \"%{DATA:user_agent}\" %{NOTSPACE:ssl_cipher} %{NOTSPACE:ssl_protocol}" }
    tag_on_failure => [ "_grokparsefailure" ]
  }

  date {
    match => [ "timestamp", "ISO8601", "dd/MMM/yyyy:HH:mm:ss Z" ]
    target => "@timestamp"
    remove_field => [ "timestamp" ]
  }
}