Log Architecture & Structural Overview

The AWS Lambda Execution & REPORT Log Parser is an essential telemetry stream within the AWS Cloud Infrastructure ecosystem. Parse AWS Lambda execution logs and REPORT metrics: duration, billed duration, memory size, and max memory used.

This schema defines a structure of 5 extracted attributes, including 4 numeric metrics and 1 string dimensions. In production observability architectures, these tokens provide high-cardinality indexing keys for telemetry pipelines before shipping to storage backends such as ClickHouse, Elasticsearch, Amazon S3, or Datadog.

Raw Telemetry Ingestion Profile

A typical raw event line for aws-lambda-logs averages 139 bytes across 5 tokens. Modern collectors such as Fluent Bit and Vector require zero-backtracking regular expressions to avoid CPU spikes during traffic surges.

Extracted Field Schema & Data Types

The transpiled Grok pattern extracts the following schema fields from each raw event line. Data collectors cast these values according to the typed mappings below.

Field Name	Inferred Type	Description & Collector Semantics
request_id	string	Unique invocation UUID identifier.
duration_ms	float	Actual execution runtime in milliseconds.
billed_duration_ms	integer	Execution duration rounded up to nearest 1ms billing bracket.
memory_size_mb	integer	Configured function RAM allocation in MB.
max_memory_used_mb	integer	Peak memory consumption reached by function.

Common Regex Traps & Production Edge Cases

Engineers frequently encounter ingestion failures or pipeline drops due to subtle variations in real-world event logs. Watch out for these verified pitfalls:

1Lambda REPORT lines are tab-separated; standard space-separated Grok expressions will fail.

2Init Duration (cold start) only appears on cold invocations; make Init Duration optional in full pipeline rules.

Production Collector Setup & Configurations

Pre-configured parser definitions ready to be dropped into your infrastructure repository.

Fluent Bit (parsers.conf)

Format: regex

# ==============================================================================
# Fluent Bit Parser Configuration (parsers.conf)
# ==============================================================================
[PARSER]
    Name        logpipeline_parser
    Format      regex
    Regex       ^REPORT RequestId: (?<request_id>[A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12})\tDuration: (?<duration_ms>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) ms\tBilled Duration: (?<billed_duration_ms>(?:[+-]?(?:[0-9]+))) ms\tMemory Size: (?<memory_size_mb>(?:[+-]?(?:[0-9]+))) MB\tMax Memory Used: (?<max_memory_used_mb>(?:[+-]?(?:[0-9]+))) MB$
    Time_Key    timestamp
    Time_Format %Y-%m-%dT%H:%M:%S%z
    Types       duration_ms:float billed_duration_ms:integer memory_size_mb:integer max_memory_used_mb:integer

# ==============================================================================
# Fluent Bit Pipeline Filter (fluent-bit.conf)
# ==============================================================================
[FILTER]
    Name         parser
    Match        *
    Key_Name     log
    Parser       logpipeline_parser
    Reserve_Data On

Vector.dev (Remap VRL)

parse_regex!

# ==============================================================================
# Vector.dev Remap Language (VRL) Transform
# Use inside a 'remap' transform in vector.yaml
# ==============================================================================
.parsed, err = parse_regex(.message, r'^REPORT RequestId: (?<request_id>[A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12})\tDuration: (?<duration_ms>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) ms\tBilled Duration: (?<billed_duration_ms>(?:[+-]?(?:[0-9]+))) ms\tMemory Size: (?<memory_size_mb>(?:[+-]?(?:[0-9]+))) MB\tMax Memory Used: (?<max_memory_used_mb>(?:[+-]?(?:[0-9]+))) MB$')

if err == null {
    . = merge(., .parsed)
    del(.parsed)

    # Type coercions
    .duration_ms = to_float!(.duration_ms)
    .billed_duration_ms = to_int!(.billed_duration_ms)
    .memory_size_mb = to_int!(.memory_size_mb)
    .max_memory_used_mb = to_int!(.max_memory_used_mb)

} else {
    log("LogPipeline parsing warning: " + err, level: "warn")
}

# ==============================================================================
# vector.yaml Pipeline Component
# ==============================================================================
transforms:
  parse_logs:
    type: remap
    inputs: ["source_logs"]
    source: |
      .parsed, err = parse_regex(.message, r'^REPORT RequestId: (?<request_id>[A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12})\tDuration: (?<duration_ms>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) ms\tBilled Duration: (?<billed_duration_ms>(?:[+-]?(?:[0-9]+))) ms\tMemory Size: (?<memory_size_mb>(?:[+-]?(?:[0-9]+))) MB\tMax Memory Used: (?<max_memory_used_mb>(?:[+-]?(?:[0-9]+))) MB$')
      if err == null {
        . = merge(., .parsed)
        del(.parsed)
      }

Datadog Log Pipeline Grok Parser

match_rules

# ==============================================================================
# Datadog Log Processing Pipeline Grok Parser
# Navigate to: Logs -> Configuration -> Pipelines -> Add Processor -> Grok Parser
# ==============================================================================

# Match Rule:
rule REPORT RequestId: %{UUID:request_id}\tDuration: %{NUMBER:duration_ms} ms\tBilled Duration: %{INT:billed_duration_ms} ms\tMemory Size: %{INT:memory_size_mb} MB\tMax Memory Used: %{INT:max_memory_used_mb} MB

# Complete Datadog Pipeline Processor JSON:
{
  "type": "grok-parser",
  "name": "LogPipeline Grok Parser",
  "is_enabled": true,
  "source": "message",
  "samples": [],
  "grok": {
    "match_rules": "rule REPORT RequestId: %{UUID:request_id}\\tDuration: %{NUMBER:duration_ms} ms\\tBilled Duration: %{INT:billed_duration_ms} ms\\tMemory Size: %{INT:memory_size_mb} MB\\tMax Memory Used: %{INT:max_memory_used_mb} MB",
    "support_rules": ""
  }
}

# Target Fields Created:
# request_id (string), duration_ms (float), billed_duration_ms (integer), memory_size_mb (integer), max_memory_used_mb (integer)

OpenTelemetry Collector (transform processor)

regex_parser

# ==============================================================================
# OpenTelemetry Collector Configuration (otel-collector-config.yaml)
# Option 1: Filelog Receiver with regex_parser Operator
# ==============================================================================
receivers:
  filelog:
    include: [ /var/log/**/*.log ]
    start_at: beginning
    operators:
      - type: regex_parser
        id: logpipeline_regex_parser
        regex: '^REPORT RequestId: (?<request_id>[A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12})\tDuration: (?<duration_ms>(?:(?:(?:[+-]?(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+))))) ms\tBilled Duration: (?<billed_duration_ms>(?:[+-]?(?:[0-9]+))) ms\tMemory Size: (?<memory_size_mb>(?:[+-]?(?:[0-9]+))) MB\tMax Memory Used: (?<max_memory_used_mb>(?:[+-]?(?:[0-9]+))) MB$'
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%dT%H:%M:%S%z'

# ==============================================================================
# Option 2: Transform Processor (OTel Transformation Language - OTTL)
# ==============================================================================
processors:
  transform:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - merge_maps(attributes, extract_patterns(body, "^REPORT RequestId: (?<request_id>[A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12})\\tDuration: (?<duration_ms>(?:(?:(?:[+-]?(?:[0-9]+(?:\\.[0-9]+)?|\\.[0-9]+))))) ms\\tBilled Duration: (?<billed_duration_ms>(?:[+-]?(?:[0-9]+))) ms\\tMemory Size: (?<memory_size_mb>(?:[+-]?(?:[0-9]+))) MB\\tMax Memory Used: (?<max_memory_used_mb>(?:[+-]?(?:[0-9]+))) MB$"), "insert")

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [transform]
      exporters: [otlp]

Logstash Filter Configuration

filter.grok

# ==============================================================================
# Logstash Pipeline Configuration (/etc/logstash/conf.d/logpipeline.conf)
# ==============================================================================
filter {
  grok {
    match => { "message" => "REPORT RequestId: %{UUID:request_id}\tDuration: %{NUMBER:duration_ms:float} ms\tBilled Duration: %{INT:billed_duration_ms:integer} ms\tMemory Size: %{INT:memory_size_mb:integer} MB\tMax Memory Used: %{INT:max_memory_used_mb:integer} MB" }
    tag_on_failure => [ "_grokparsefailure" ]
  }

  date {
    match => [ "timestamp", "ISO8601", "dd/MMM/yyyy:HH:mm:ss Z" ]
    target => "@timestamp"
    remove_field => [ "timestamp" ]
  }
}