☰ All modules Tier 3 · Monitoring (CloudWatch, X-Ray, CloudTrail, EventBridge) GuideCheat sheet ‹ other-servicescontainers ›
Contents
  1. CloudWatch metrics and custom metrics
  2. Metrics from logs: EMF and metric filters
  3. CloudWatch Logs ingestion, retention and the agent
  4. Logs Insights, subscriptions and S3 export
  5. CloudWatch alarms and Synthetics canaries
  6. EventBridge rules, event patterns and targets
  7. EventBridge schema registry and cross-account events
  8. Distributed tracing with X-Ray
  9. Trace sampling, APIs and troubleshooting
  10. CloudTrail auditing and choosing the tool
  11. Numbers to know
  12. Decision rules
  13. Memory hooks

Monitoring (CloudWatch, X-Ray, CloudTrail, EventBridge) — Cheat Sheet (DVA-C02)

Verified against AWS documentation on 2026-10-12 · ⚠️ = changed vs older material: learn the value shown here (older practice material may say otherwise; full list in notes/monitoring-guide.md#changed-vs-older-material) · explanations and examples: notes/monitoring-guide.md.

CloudWatch metrics and custom metrics

Topic Current facts Memory hook
Metric identity Namespace + metric name + dimensions; change any of them and it's a different metric Name, place, labels
Custom namespace Required on every PutMetricData call; 1–255 characters; must not start with AWS/ (reserved) AWS/ belongs to AWS
Dimensions per metric Up to 30; every unique combination of values is a separate, separately billed metric Every label combo costs
Custom metric aggregation Not aggregated across dimensions: read the exact dimension set you published (or use SEARCH) Publish what you'll query
StorageResolution 60 = standard 1-minute (default); 1 = high resolution, readable at 1, 5, 10, 30 s or multiples of 60 1 = one second
Metric retention Under 60 s: 3 hours · 1 minute: 15 days · 5 minutes: 63 days · 1 hour: 455 days (15 months) 3 h, 15 d, 63 d, 15 mo
Metric timestamps Accepted from 2 weeks in the past to 2 hours in the future; older points are rejected Two weeks back, two hours ahead
PutMetricData request size ⚠️ Up to 1,000 metrics and 1 MB per call (gzip allowed); 150 values per Values array (older: 20 metrics, 40 KB)
Percentiles on custom metrics Need raw values (Value or Values + Counts); a statistic set (StatisticValues) loses them Sets lose the shape
New metric visibility A new metric can take up to 15 minutes to appear in ListMetrics
Inactive metrics No data for 2 weeks → hidden in console and list-metrics, still readable with get-metric-data; expire after 15 months
GetMetricData Up to 500 queries and 100,800 data points per call; metric math; one Metrics Insights query (last 2 weeks) The bulk reader
IAM for PutMetricData cloudwatch:PutMetricData on *; restrict with the cloudwatch:namespace condition key Scope by namespace
CloudWatch free tier 10 metrics shared by custom and detailed monitoring, 10 standard alarm metrics, 3 dashboards, 1 million API requests
Cross-account observability Monitoring account sink + source account links share metrics, logs and traces within one Region One pane, many accounts

Metrics from logs: EMF and metric filters

Topic Current facts Memory hook
Embedded metric format (EMF) JSON log event with _aws → Timestamp (ms) + CloudWatchMetrics [Namespace, Dimensions sets, Metrics]; values are top-level fields Metrics ride inside logs
EMF limits 100 metric definitions per event; 30 keys per dimension set; up to 100 values per array; 14 days back to 2 hours ahead
EMF transport Lambda: print to stdout; servers and containers: CloudWatch agent; direct: PutLogEvents (x-amzn-logs-format: json/emf optional) Lambda just prints
EMF failures EMFParsingErrors and EMFValidationErrors metrics in the AWS/Logs namespace
EMF dimension sets Each set in Dimensions creates its own metric; keep request or order IDs as plain fields IDs in fields, not dimensions
Metric filter basics Filter pattern on a log group → custom metric; not retroactive; up to 100 filters per log group Only from now on
Metric filter dimensions Up to 3 (JSON or space-delimited patterns only); no default value when dimensions are set; 1,000 distinct name/value pairs → the filter may be disabled Three max
Metric filter default value Emitted (often 0) when logs arrive but nothing matches, so graphs have no gaps Zero beats a gap
Filter pattern syntax ERROR, ?ERROR ?WARN (OR), ERROR -Timeout (exclude), { $.latencyMs > 1000 }, [ip, user, ts, req, status=5*, bytes], %regex%, "" = all
Log class for EMF and filters Both work only on Standard-class log groups
Contributor Insights Rules that rank top-N contributors (IPs, keys, callers) from structured logs Who's the noisiest?

CloudWatch Logs ingestion, retention and the agent

Topic Current facts Memory hook
Log group vs log stream Group holds the settings (retention, KMS key, class, filters); stream = one source's sequence of events Folder vs file
Default log retention Never expire; put-retention-policy 1 day to 10 years (3,653 days); delete-retention-policy restores never Never, unless you say so
Maximum log event size ⚠️ 1 MB per event since 2025-04-02 (older: 256 KB)
PutLogEvents batch ≤ 1,048,576 bytes (messages + 26 bytes per event), ≤ 10,000 events, chronological, ≤ 24-hour span
PutLogEvents sequence token ⚠️ Ignored; parallel writes to one stream are fine; throttle 5,000 requests/s per account (older: token required, 5/s per stream) Tokens are gone
Log classes Standard (all features) vs Infrequent Access (cheaper ingestion; no metric or subscription filters, Live Tail, EMF); fixed at creation IA = archive-ish, query-only
Data protection policy Masks credentials, card numbers and PII at ingestion (new events only); viewing unmasked needs logs:Unmask
Live Tail Up to 10 log groups, 3-hour sessions, 15 concurrent sessions, billed per minute; aws logs start-live-tail tail -f for CloudWatch
Log writer permissions logs:CreateLogStream and logs:PutLogEvents (plus logs:CreateLogGroup) on arn:aws:logs:…:log-group:<name>:*
Unified CloudWatch agent Log files, memory/disk/process metrics, StatsD/collectd, EMF, traces (1.300025.0+); fleet config in SSM parameter AmazonCloudWatch-* One agent for everything
Agent IAM Instance role with CloudWatchAgentServerPolicy; plus a network path to the CloudWatch endpoints
CloudWatch Logs agent ⚠️ awslogs is deprecated and fails on IMDSv2-only instances → use the unified agent Old agent, retire it

Logs Insights, subscriptions and S3 export

Topic Current facts Memory hook
Logs Insights languages ⚠️ Logs Insights QL, OpenSearch PPL and OpenSearch SQL (older: one language) Three dialects
Logs Insights limits ⚠️ 100 concurrent QL queries (15 PPL/SQL); up to 100,000 QL results, 10,000 per page; 60-minute timeout; results kept 7 days 100 queries, 60 minutes, 7 days
Logs Insights API start-query (epoch seconds, ≤ 50 named log groups or a SOURCE command) → queryId → get-query-results; billed per GB scanned Start, then fetch
Subscription destinations Kinesis Data Streams, Amazon Data Firehose, Lambda, OpenSearch Service
Subscription filters per log group ⚠️ 5 per log group plus 1 account-level filter per Region (older: 2)
Subscription delivery base64 + gzip, usually under 3 minutes, at least once; retryable errors retried for 24 hours; Standard class only
Subscription permissions Kinesis or Firehose: role trusted by logs.amazonaws.com; Lambda: resource policy for logs.amazonaws.com with the log group as source ARN Streams need a role, Lambda needs a policy
Cross-account log streaming Recipient: put-destination (stream + role) and put-destination-policy (logs:PutSubscriptionFilter); sender: filter → destination ARN Destination in the middle
Kinesis subscription throttling Default distribution ByLogStream can overload a shard; use --distribution Random
Export to S3 create-export-task (from/to in ms); up to 12 hours of lag; 1 active task per account; SSE-S3 or SSE-KMS, not DSSE-KMS Batch, not stream

CloudWatch alarms and Synthetics canaries

Topic Current facts Memory hook
Alarm states OK, ALARM, INSUFFICIENT_DATA; a new alarm starts in INSUFFICIENT_DATA
M out of N evaluation DatapointsToAlarm (M) of EvaluationPeriods (N); Period × N ≤ 1 day (periods under 1 h) or ≤ 7 days
Alarm periods ⚠️ 10, 20 or 30 s (high-resolution alarm, needs StorageResolution 1) or any multiple of 60 s
TreatMissingData missing (default), notBreaching, breaching, ignore; AWS/DynamoDB alarms always ignore Breaching = heartbeat lost
Alarm actions ⚠️ SNS, Lambda (since 2023-12), EC2 stop/terminate/reboot/recover, Auto Scaling, SSM OpsItem or incident, investigations; ≤ 5 per state
Alarm action frequency Once per state change; only Auto Scaling actions repeat every minute while in ALARM
Lambda alarm action Function resource policy for lambda.alarms.cloudwatch.amazonaws.com with AWS:SourceArn = alarm ARN; invoked asynchronously
Composite alarms ⚠️ ALARM()/OK()/INSUFFICIENT_DATA() with AND/OR/NOT and AT_LEAST; SNS, Lambda, OpsItems, investigations; no EC2/Auto Scaling actions Combine to cut noise
Composite alarm limits Up to 100 child alarms; actions suppressor alarm with wait and extension periods; not cross-account
Testing an alarm set-alarm-state sets the state until the next evaluation and runs the real actions Test fires for real
Alarm mute rules Since 2026-02-10: cron()/at() windows (1 minute to 15 days) mute actions; muted actions re-run at window end if the state persists
Alarm events State changes go to EventBridge as CloudWatch Alarm State Change (source aws.cloudwatch)
Canary languages ⚠️ Node.js (Puppeteer or Playwright), Python (Selenium), Java; Node.js canaries drive Chrome or Firefox
Canary execution Lambda function in your account; run once, rate() down to 1 minute, or cron(); artifacts in S3; optional VPC
Canary blueprints ⚠️ Heartbeat, API canary, broken link checker, visual monitoring, canary recorder, GUI workflow builder, multi checks
Canary metrics CloudWatchSynthetics: SuccessPercent, Duration, Failed, 2xx/4xx/5xx; dimension CanaryName Alarm on SuccessPercent

EventBridge rules, event patterns and targets

Topic Current facts Memory hook
Event buses Default (AWS service events), custom (your applications), partner (SaaS)
PutEvents size ⚠️ ≤ 10 entries and ≤ 1 MB per request to default/Classic buses since 2026-01-29 (older: 256 KB)
PutEvents partial failure HTTP 200 with FailedEntryCount > 0 → resend the entries that have an ErrorCode; unknown bus name = silent drop 200 isn't success
Rule limits 5 targets per rule (fixed); 300 rules per bus (adjustable); event pattern ≤ 2,048 characters Five targets max
Event pattern logic Values in an array = OR; separate fields = AND; case-sensitive; prefix, suffix, anything-but, numeric, cidr, exists, wildcard, $or
Target permissions ⚠️ Lambda/SNS/SQS: resource policy (events.amazonaws.com) or execution role (since 2025-03); Kinesis, Step Functions, ECS: role; Logs: resource policy
Target retry policy Default 24 hours and 185 attempts; missing permissions or deleted targets aren't retried
Target DLQ Standard SQS queue (no FIFO) in the same Region; attributes ERROR_CODE, ERROR_MESSAGE, RULE_ARN, TARGET_ARN
Target input Matched event, InputPath, constant Input, or input transformer (InputPathsMap + InputTemplate)
Scheduling ⚠️ Scheduled rules are legacy (UTC, default bus); EventBridge Scheduler: at(), rate(), cron(), time zones, flexible windows, role per target Scheduler knows time zones
Archive and replay Archive per bus (pattern, retention days or forever); replay only to the source bus, adds replay-name, ≤ 10 concurrent
EventBridge Pipes Point-to-point: SQS, Kinesis, DynamoDB Streams, MSK, Kafka or MQ → filter → enrich → one target Pipe = one-to-one
EventBridge metrics MatchedEvents, Invocations, FailedInvocations, ThrottledRules, InvocationsSentToDLQ
Event ordering Classic buses: no ordering guarantee; new Custom Event Bus: FIFO within an EventGroupId per subscriber

EventBridge schema registry and cross-account events

Topic Current facts Memory hook
Schema registries aws.events (AWS schemas), discovered-schemas, custom; OpenAPI 3 or JSONSchema Draft4; versioned
Schema discovery aws schemas create-discoverer on a bus; infers the original (untransformed) events; first 5 million events per month free; not on buses with a customer managed key
Code bindings Java (Java8), Python (Python36), TypeScript (TypeScript3), Go (Go1) J, P, T, G
Event bus resource policy Grants events:PutEvents to account IDs, or to * with aws:PrincipalOrgID
Cross-account sender Rule targeting the receiver bus ARN with RoleArn (required for new cross-account bus targets since 2023-03-02)
Cross-account receiver Needs its own rules; events never hop on to a third account or Region One hop only
Direct cross-account targets ⚠️ SQS, SNS, Lambda, Kinesis Data Streams, API Gateway in another account of the same Region since 2025-01-21; execution role in the bus account + target resource policy that allows the bus account
New Custom Event Bus ⚠️ Since 2026-09-24: subscribers, retention 1–365 days, EventGroupId ordering, 5-minute dedup, AWS RAM sharing, aws eventsv2; old buses = "Classic"

Distributed tracing with X-Ray

Topic Current facts Memory hook
Trace building blocks Segment per service → subsegments for calls and code blocks → trace = all segments with one trace ID
Inferred segments Built from your subsegments for services that don't trace themselves (DynamoDB, S3, external APIs)
Trace header X-Amzn-Trace-Id: Root=1-<8 hex time>-<24 hex>;Parent=<id>;Sampled=1
X-Ray limits Segment document ≤ 64 KB; traces kept 30 days; 50 annotations indexed per trace
Annotations String, number or Boolean; indexed; usable in filter expressions as annotation[key] Annotate to search
Metadata Any type including objects; not indexed; grouped in namespaces Metadata to store
X-Ray error classes Error = 4xx, Throttle = 429, Fault = 5xx Faults are the server's
X-Ray SDK and daemon ⚠️ Maintenance mode since 2026-02-25 (security fixes only); use OpenTelemetry (ADOT or OTel SDKs) SDK frozen, OTel forward
Trace collectors Daemon: UDP 2000 + TCP 2000 sampling proxy; replacements: CloudWatch agent 1.300025.0+ (OTLP 4317/4318) or OTel Collector
OpenTelemetry mapping Server spans → segments; span attributes → metadata unless listed in aws.xray.annotations
Tracing IAM AWSXRayDaemonWriteAccess: PutTraceSegments, PutTelemetryRecords, GetSamplingRules, GetSamplingTargets, GetSamplingStatisticSummaries
Elastic Beanstalk tracing aws:elasticbeanstalk:xray → XRayEnabled: true, instance profile write policy, instrumented code Flag + role + code
ECS tracing Daemon or agent sidecar on UDP 2000; bridge mode: link + AWS_XRAY_DAEMON_ADDRESS; permissions on the task role Sidecar on Fargate
ALB and X-Ray Adds X-Amzn-Trace-Id but sends no segments and isn't a node
X-Ray encryption Always encrypted at rest; optional aws/xray or symmetric customer managed key

Trace sampling, APIs and troubleshooting

Topic Current facts Memory hook
Default sampling rule Reservoir 1 request per second + 5 % of the rest; can be modified, not deleted One a second, five percent
Sampling rule fields Priority 1–9999 (lowest first), reservoir (per second, fleet-wide), fixed rate 0–1, service, type, host, method, URL path
Parent-based sampling Decided at the trace root and passed in Sampled=; downstream rules usually never apply Root decides
Lambda sampling Fixed at 1 per second + 5 %, not changed by sampling rules
Adaptive sampling SamplingRateBoost (MaxRate, CooldownWindowMinutes) raises sampling during anomalies (since 2025-09-25)
Reading traces GetTraceSummaries (filter → IDs) then BatchGetTraces (≤ 5 IDs per call); GetServiceGraph, GetTraceGraph Summaries, then batch
Writing traces PutTraceSegments (segments) and PutTelemetryRecords (collector health)
Filter expressions service("checkout") { fault }, responsetime > 2, http.status = 500, annotation[tier] = "gold", user = "U1"
X-Ray groups Named filter expression with its own trace map and CloudWatch metrics
Transaction Search 100 % of spans as logs in aws/spans, 1 % indexed as trace summaries free by default; UpdateTraceSegmentDestination → CloudWatchLogs; then BatchGetTraces can't find traces (use StartTraceRetrieval)
X-Ray free tier 100,000 traces recorded and 1,000,000 traces retrieved or scanned per month
"No traces" checklist Instrumented clients, reachable collector, write permissions, request sampled, active tracing on, right Region

CloudTrail auditing and choosing the tool

Topic Current facts Memory hook
CloudTrail Event history On by default, free, last 90 days of management events per Region; one lookup attribute per search 90 days for free
LookupEvents 2 requests per second per account and Region; up to 50 results per page
Trails ⚠️ Console: multi-Region only and logging starts; CLI/API: single-Region default and start-logging needed; 5 trails per Region CLI starts small and silent
Trail delivery S3 gzip files, optional CloudWatch Logs and EventBridge; events about 5 minutes after the call (not guaranteed)
Trail log file encryption Console default SSE-KMS; CLI/API without --kms-key-id → SSE-S3
Log file integrity validation --enable-log-file-validation: SHA-256 hashes, RSA-signed hourly digest files; check with validate-logs Digest proves no tampering
Data events ⚠️ Opt-in and billed; advanced event selectors cover S3, Lambda, DynamoDB, SNS, SQS, Kinesis and many more Data = data plane
CloudTrail Insights ⚠️ API call rate (write management) and API error rate (read/write management); data-event Insights on trails since 2025-11-20 Insights = unusual rates
CloudTrail Lake ⚠️ Closed to new customers since 2026-05-31; use CloudWatch or a trail to S3 queried with Athena
Trail to CloudWatch Logs Role trusted by cloudtrail.amazonaws.com with logs:CreateLogStream and logs:PutLogEvents; events over 256 KB not sent
CloudTrail to EventBridge Default bus only, detail-type AWS API Call via CloudTrail, needs a trail with logging on; read-only calls need rule state ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS (CLI or CloudFormation; on both rules when forwarding)
Choosing the tool Health and alerts → CloudWatch; log search → Logs Insights; request path and latency → X-Ray; who did what → CloudTrail

Numbers to know

Item Value
Dimensions per custom metric 30
PutMetricData per call ⚠️ 1,000 metrics, 1 MB
Metric timestamp window 2 weeks back, 2 hours ahead
High-resolution (sub-minute) data retention 3 hours
1-minute / 5-minute / 1-hour data retention 15 days / 63 days / 455 days
New metric visible in ListMetrics Up to 15 minutes
GetMetricData per call 500 queries, 100,800 data points
Maximum log event size ⚠️ 1 MB
PutLogEvents batch 1,048,576 bytes, 10,000 events, 24-hour span
Log retention range 1 day to 3,653 days (10 years), or never expire
Metric filters per log group 100
Dimensions per metric filter 3
EMF metric definitions per log event 100
Subscription filters per log group ⚠️ 5 (plus 1 account-level filter per Region)
Logs Insights query timeout 60 minutes
Logs Insights results retention 7 days
Logs Insights concurrent queries ⚠️ 100 (QL), 15 (PPL/SQL)
Logs Insights maximum results ⚠️ 100,000 (QL), 10,000 per GetQueryResults page
S3 export lag and concurrency Up to 12 hours; 1 active export task per account
Live Tail session 10 log groups, 3 hours, 15 concurrent sessions
Alarm periods ⚠️ 10, 20, 30 seconds or multiples of 60
Alarm actions per state 5
Child alarms per composite alarm 100
Canary minimum frequency Once per minute
PutEvents per request ⚠️ 10 entries, 1 MB
Targets per EventBridge rule 5
Rules per event bus (default quota) 300
EventBridge default retry 24 hours, 185 attempts
New Custom Event Bus retention ⚠️ 1–365 days (24 hours default)
X-Ray segment document 64 KB
X-Ray trace retention 30 days
X-Ray indexed annotations per trace 50
X-Ray default sampling 1 request/s + 5 %
BatchGetTraces trace IDs per call 5
X-Ray daemon port UDP 2000
CloudTrail Event history 90 days
LookupEvents 2 requests/s, 50 results per page
Trails per Region 5
CloudTrail event size limit for CloudWatch Logs and EventBridge 256 KB
CloudTrail delivery delay About 5 minutes on average

Decision rules

Memory hooks