Monitoring (CloudWatch, X-Ray, CloudTrail, EventBridge) — DVA-C02 Mastery Guide

• Sources: notes/source/13.Monitoring.pdf (course chapter 13: monitoring, troubleshooting and audit with CloudWatch, X-Ray, CloudTrail and EventBridge). The course serves as the syllabus only; all wording here is original.
• Written 2026-10-12; values checked against AWS docs on 2026-10-12. The marker ⚠️ Changed: <current>. Older material may say <old>. flags a value or behaviour that differs from the course or from older practice tests: learn the current value. ⚠️ unverified marks a claim that could not be confirmed in current AWS documentation: don't rely on it. Every change is listed in Changed vs older material and, once confirmed, in notes/current-facts.md.
• Tier 3 (supporting), bank target 30 questions. Domain: Troubleshooting and Optimization (config key troubleshooting).
• How to use it: /explain <topic> teaches from these sections (concept → exam facts → developer angle → traps → 3 check questions); /quiz monitoring asks bank questions whose source_note links point at the ## N. headings below; /drill monitoring fires short recall prompts from notes/monitoring-cheat-sheet.md. Suggested loop: read a section, answer its two practice questions before opening the answers, /explain what is still unclear, then /quiz and /drill.

How Monitoring (CloudWatch, X-Ray, CloudTrail, EventBridge) appears on DVA-C02

Domain 4 (Troubleshooting and Optimization, 18 % of the scored exam) is where this module lives. Its task statements ask you to find root causes from metrics, logs and traces, to instrument code so it can be observed, and to tune a workload with what you measured. EventBridge also shows up in Domain 1 questions about event-driven integration, and CloudTrail in Domain 2 questions about auditing.

What the exam expects from a developer:

• Emit the right telemetry from code: custom metrics with PutMetricData or the embedded metric format (EMF), structured JSON logs, trace annotations, and alarms that notify on quotas, errors or finished deployments.
• Query and route it: Logs Insights queries, metric filters, subscription filters to Kinesis, Firehose or Lambda, exports to S3, EventBridge rules on AWS events and API calls.
• Read the symptom: an alarm stuck in INSUFFICIENT_DATA, a trace with no downstream nodes, a metric that never appears, logs that stop arriving, an event rule that matches but never invokes its target.
• Pick the tool: CloudWatch for "how much / how often / is it healthy", X-Ray for "where did this request spend its time or fail", CloudTrail for "who called which API, when, from where". Logs record individual events, metrics are numeric time series, traces follow one request; observability means you can explain a new failure from those three signals without shipping new code.

Typical question shapes: a scenario with one deciding constraint (least operational overhead, no code changes, near real time, least privilege, longest retention at lowest cost); a missing-data or missing-trace troubleshooting case; "Which TWO steps…" with five options; the exact API action, CLI command, IAM permission or setting that does the job.

Out of scope: OS-level agent administration and network design. EC2-specific metrics, detailed monitoring, status checks and instance recovery are owned by the EC2 module (see notes/ec2-mastery-guide.md#11-cloudwatch-for-ec2-metrics-agent-and-status-checks). Lambda logging controls and active tracing: notes/lambda-guide.md#12-observability-with-logs-metrics-and-traces. API Gateway execution and access logs: notes/api-gateway-guide.md#12-logging-metrics-and-tracing. KMS encryption of log groups: notes/security-encryption-guide.md#12-cloudwatch-logs-encryption-and-codebuild-secrets. Firehose and Kinesis Data Streams themselves: notes/sqs-sns-kinesis-guide.md. ECS log drivers (awslogs, FireLens): notes/containers-guide.md#2-task-definitions-and-container-settings. Quota-usage alarms (AWS/Usage, SERVICE_QUOTA()): notes/cli-sdk-policies-guide.md#8-service-quotas-and-api-rate-limits. Pipeline and deployment notifications: notes/cicd-guide.md#12-manual-approvals-and-pipeline-monitoring.

⚠️ Changed: CloudWatch Evidently was discontinued on 2025-10-17 (full shutdown); feature flags and launches now use AWS AppConfig feature flags, and A/B or multivariate experiments use AWS AppConfig experimentation (generally available since 2026-07-01, results analysed in CloudWatch or your own tools). Older material may say CloudWatch Evidently is the service for feature flags and A/B tests. AppConfig itself: notes/other-services-guide.md#10-appconfig-feature-flags-validators-and-deployment-strategies.

1. CloudWatch metrics and custom metrics

Concept

• A metric is a time-ordered series of data points identified by three things: a namespace (a container such as AWS/Lambda or your own OrdersApp), a metric name, and zero or more dimensions (name/value pairs such as Service=checkout). Change any of them and you have a different metric.
• AWS services publish their metrics for free. Anything only your code knows (business counts, queue lag measured inside the app, latency of a third-party call) is a custom metric that you publish yourself, and each one is billed.
• Data is stored at standard resolution (1-minute) or high resolution (1-second, custom metrics only). CloudWatch keeps fine-grained data briefly and rolls it up into coarser periods over 15 months.
• You read metrics through graphs, dashboards, GetMetricData, metric math and Metrics Insights (SQL-like) queries, and you act on them with alarms (§5).

Exam facts

• Namespace rules: required on every PutMetricData call (there is no default); 1–255 characters; don't start your own with AWS/ (reserved for AWS services). Metrics exist only in the Region where they were published.
• Dimensions: up to 30 per metric. Every unique combination of dimension values is a separate metric that is billed separately, so high-cardinality values (request IDs, user IDs, IP addresses) explode cost. CloudWatch does not aggregate across dimensions of custom metrics: you can only read the exact dimension combinations you published (the metric math SEARCH function is the exception).
• Resolution: StorageResolution = 60 (default, standard) or 1 (high resolution). High-resolution data can be read at periods of 1, 5, 10 or 30 seconds or any multiple of 60. AWS service metrics are standard resolution.
• Retention (automatic roll-up): periods under 60 s kept 3 hours; 1-minute data 15 days; 5-minute data 63 days; 1-hour data 455 days (15 months). After 15 days, 1-minute data is only readable at 5-minute granularity, and so on. Metrics can't be deleted; they expire 15 months after their last data point. Metrics with no new data for 2 weeks disappear from the console and list-metrics, but get-metric-data still returns their stored points.
• Timestamps: accepted from 2 weeks in the past to 2 hours in the future; a point older than that is rejected, so keep server clocks synchronised and use UTC. Points stamped 3–24 hours in the past can take up to 2 hours to become readable; points 24 hours old or older can take at least 48 hours. A brand-new metric can take up to 15 minutes to show up in ListMetrics.
• ⚠️ Changed: one PutMetricData call carries up to 1,000 metrics and 1 MB (gzip compression allowed), with up to 150 values in a Values/Counts array per metric. Older material may say 20 metrics and 40 KB per call.
• Three ways to send values in one call: a single Value; Values + Counts arrays (raw values with how many times each occurred: keeps percentiles); a statistic set (StatisticValues: SampleCount, Sum, Minimum, Maximum), the most compact form, which loses percentiles unless every sample was identical.
• Units (Count, Milliseconds, Bytes, Percent, Count/Second…) are aggregated separately; None is the default. Values must be within ±2^360; NaN and infinities are rejected.
• Statistics: SampleCount, Sum, Average, Minimum, Maximum, percentiles (p99), trimmed mean (tm90) and others. Percentiles need raw values and no negative numbers.
• Reading: GetMetricData is the bulk API: up to 500 queries (metrics, math expressions or one Metrics Insights query) and up to 100,800 data points per call, paginated with NextToken. Metrics Insights queries cover the last 2 weeks (3 hours when evaluated by an alarm). GetMetricStatistics reads one metric at a time.
• Errors: InvalidParameterValue (bad or out-of-range value), MissingParameter, InvalidParameterCombination (all HTTP 400) and InternalServiceError (500). Throttled calls are retried by the SDK with exponential backoff (see notes/cli-sdk-policies-guide.md#9-retries-and-exponential-backoff-with-jitter).
• Free tier: basic AWS metrics, 10 metrics shared between custom metrics and EC2 detailed-monitoring metrics, 1 million API requests (not GetMetricData), 3 dashboards, 10 standard-resolution alarm metrics. Custom metrics are prorated by the hour, charged only for hours in which data was sent.
• Cross-account observability: a monitoring account holds a sink, each source account creates a link (Observability Access Manager). It shares metrics, log groups, traces and Application Signals data within one Region; a monitoring account can link up to 100,000 source accounts and a source account up to 5 monitoring accounts. Logs and metrics sharing costs nothing extra.
• EC2-specific metrics (AWS/EC2, detailed monitoring, CWAgent memory and disk metrics, status checks): see notes/ec2-mastery-guide.md#11-cloudwatch-for-ec2-metrics-agent-and-status-checks.

Developer angle

Publish a high-resolution metric, a batch of raw values, and a statistic set (the credentials come from the instance or task role):

aws cloudwatch put-metric-data --namespace OrdersApp --metric-name CheckoutLatency \
  --dimensions Service=checkout,Env=prod --unit Milliseconds --value 182 --storage-resolution 1

aws cloudwatch put-metric-data --namespace OrdersApp --metric-name CheckoutLatency \
  --dimensions Service=checkout,Env=prod --unit Milliseconds \
  --statistic-values Sum=1460,Minimum=95,Maximum=410,SampleCount=8

aws cloudwatch list-metrics --namespace OrdersApp --metric-name CheckoutLatency

import boto3

cloudwatch = boto3.client("cloudwatch")
cloudwatch.put_metric_data(
    Namespace="OrdersApp",
    MetricData=[{
        "MetricName": "CheckoutLatency",
        "Dimensions": [{"Name": "Service", "Value": "checkout"}],
        "Unit": "Milliseconds",
        "Values": [95.0, 120.0, 410.0],   # raw values: percentiles stay available
        "Counts": [3.0, 4.0, 1.0],        # how often each value occurred
        "StorageResolution": 60,
    }],
)

Read an error rate with metric math in one call (queries.json):

[
  {"Id": "errors", "ReturnData": false,
   "MetricStat": {"Metric": {"Namespace": "AWS/Lambda", "MetricName": "Errors",
                             "Dimensions": [{"Name": "FunctionName", "Value": "checkout"}]},
                  "Period": 60, "Stat": "Sum"}},
  {"Id": "calls", "ReturnData": false,
   "MetricStat": {"Metric": {"Namespace": "AWS/Lambda", "MetricName": "Invocations",
                             "Dimensions": [{"Name": "FunctionName", "Value": "checkout"}]},
                  "Period": 60, "Stat": "Sum"}},
  {"Id": "errorRate", "Expression": "100 * errors / calls", "Label": "Error rate %"}
]

aws cloudwatch get-metric-data --metric-data-queries file://queries.json \
  --start-time 2026-10-12T08:00:00Z --end-time 2026-10-12T09:00:00Z

• IAM: publishing needs cloudwatch:PutMetricData (no resource-level ARNs; restrict with the cloudwatch:namespace condition key); reading needs cloudwatch:GetMetricData or cloudwatch:GetMetricStatistics and cloudwatch:ListMetrics.
• Publish from a background thread or batch per minute: a synchronous PutMetricData call per request adds latency and cost. EMF (§2) avoids the API call entirely.

Traps

• Trap: "Publish a metric with a RequestId dimension so we can drill into each request." → Reality: each value creates a new billed metric; put identifiers in logs or traces, not in dimensions.
• Trap: "Query the metric without dimensions to get the fleet total." → Reality: custom metrics are only readable with the exact dimension set you published; publish an extra fleet-level series (no dimensions) or use SEARCH.
• Trap: "A statistic set gives the same p99 as raw values." → Reality: percentiles need raw values (Values/Counts), except in the degenerate case where all samples are equal.
• Trap: "Back-fill last quarter's figures with PutMetricData." → Reality: points older than 2 weeks are rejected.
• Trap: "High-resolution data is kept for 15 days at 1-second granularity." → Reality: sub-minute data lives 3 hours, then it is only available as 1-minute (15 days), 5-minute (63 days) and 1-hour (455 days) aggregates.
• Trap: "The metric was published, so list-metrics must show it immediately." → Reality: a new metric can take up to 15 minutes to appear in ListMetrics; the data itself is usually readable much sooner.

Practice questions

1. Forty containers report payment latency for the same service. The team wants a single CloudWatch metric that shows p99 latency across all containers, while keeping the number of PutMetricData calls low. What should the developer do?

   • A. Call PutMetricData once per request and add a ContainerId dimension to every data point.
   • B. Aggregate in each container and publish a statistic set with Sum, Minimum, Maximum and SampleCount every minute.
   • C. Buffer raw latencies in each container and publish them with the Values and Counts arrays under one shared dimension set every minute.
   • D. Publish the per-container average every minute as a single Value.

2. A nightly job back-fills order totals into a custom metric with PutMetricData, using timestamps from the last 30 days. Some calls fail, and the newly created metric doesn't appear in aws cloudwatch list-metrics right after the job starts. Which TWO statements explain this? (Select TWO.)

   • A. Custom metrics must be published to a namespace that starts with AWS/.
   • B. CloudWatch rejects data points whose timestamps are more than two weeks in the past.
   • C. Each PutMetricData call accepts at most 20 metrics, so the larger batches fail.
   • D. A new metric can take up to 15 minutes to appear in ListMetrics results.
   • E. Metrics published with StorageResolution 60 are visible only after 15 days.

2. Metrics from logs with EMF and metric filters

Concept

• Sometimes the data you need is already in your logs. CloudWatch Logs can create metrics from log events in two ways, both asynchronous and free of extra API calls from your code:
  • Embedded metric format (EMF): your code writes a structured JSON log event that declares its metrics in an _aws metadata block; CloudWatch extracts them on ingestion. You keep the whole event (with request IDs and context) searchable in Logs Insights while getting cheap aggregate metrics.
  • Metric filters: you attach a filter pattern to a log group; each matching event increments or contributes a value to a metric. No code change, works on plain-text logs, but only for events ingested after the filter exists.
• Both produce custom metrics (billed as such) that you graph and alarm on like any other metric.

Exam facts

• EMF structure: a JSON object with _aws containing Timestamp (epoch milliseconds) and CloudWatchMetrics, an array of directives each with Namespace, Dimensions (an array of dimension sets, each an array of up to 30 keys) and Metrics (each with Name, optional Unit, optional StorageResolution 1 or 60). Dimension and metric values are top-level members of the same JSON object; nested fields can't be referenced.
• EMF limits: up to 100 metric definitions per log event; a numeric array target can hold up to 100 values; timestamps 14 days back to 2 hours ahead; the event is an ordinary log event, so it follows the log event size limit. ⚠️ Changed: a log event, EMF documents included, can be up to 1 MB. Older material may say 256 KB.
• Every dimension set creates a metric: [["Service"], ["Service", "Operation"]] publishes two metrics per metric name. High-cardinality keys (requestId) belong as plain top-level fields, not in Dimensions.
• How EMF gets in: Lambda: print EMF JSON to stdout (no agent; Lambda ships it). EC2/on-premises/containers: the unified CloudWatch agent with EMF enabled. Direct: PutLogEvents, optionally with the header x-amzn-logs-format: json/emf (not required). Client libraries (aws-embedded-metrics) and Powertools for AWS Lambda build the JSON for you. Failures to extract show up in the AWS/Logs namespace (EMF parsing and validation error metrics).
• EMF works only for log groups in the Standard log class.
• Metric filters: up to 100 per log group (not adjustable); not retroactive; supported only in the Standard log class; at-least-once (duplicates possible). Each filter has a pattern, a metric name and namespace, a metric value (1, or a field such as $.bytes), an optional unit, an optional default value (emitted when logs arrive but nothing matches, so the graph shows 0 instead of gaps) and up to 3 dimensions taken from fields of the event (dimensions work only with JSON and space-delimited patterns).
• A filter with dimensions can't have a default value. CloudWatch Logs may disable a filter that produces 1,000 different dimension name/value pairs within a short time, to prevent runaway cost.
• Filter pattern syntax: terms (ERROR, "Out of memory"), OR with ? (?ERROR ?Exception), exclusion with - (ERROR -Timeout), JSON selectors ({ $.level = "ERROR" }, { $.latencyMs > 1000 }, { $.statusCode >= 500 }, { ($.level = "ERROR") && ($.service = "checkout") }), space-delimited fields ([ip, user, ts, request, status=5*, bytes]), and regular expressions between % signs. An empty pattern "" matches everything.
• If log transformation is configured on the log group, metric filters see the transformed events.
• Contributor Insights is the related tool for top-N questions ("which 10 IP addresses cause most 4xx?"): rules over structured logs, billed per rule and matched event.

Developer angle

An EMF log line written by a Lambda function (two dimension sets → two metrics per name):

import json
import time

def handler(event, context):
    order = event["order"]
    print(json.dumps({
        "_aws": {
            "Timestamp": int(time.time() * 1000),
            "CloudWatchMetrics": [{
                "Namespace": "OrdersApp",
                "Dimensions": [["Service"], ["Service", "Channel"]],
                "Metrics": [{"Name": "OrderValue", "Unit": "None"},
                            {"Name": "OrdersPlaced", "Unit": "Count"}],
            }],
        },
        "Service": "checkout",
        "Channel": order["channel"],
        "OrderValue": order["total"],
        "OrdersPlaced": 1,
        "orderId": order["id"],          # searchable context, not a dimension
        "requestId": context.aws_request_id,
    }))
    return {"ok": True}

A metric filter on JSON logs, tested before it goes live:

aws logs test-metric-filter --filter-pattern '{ $.level = "ERROR" }' \
  --log-event-messages '{"level":"ERROR","msg":"card declined"}' '{"level":"INFO","msg":"ok"}'

aws logs put-metric-filter --log-group-name /orders/app --filter-name app-errors \
  --filter-pattern '{ $.level = "ERROR" }' \
  --metric-transformations metricName=AppErrors,metricNamespace=OrdersApp,metricValue=1,defaultValue=0

aws logs describe-metric-filters --log-group-name /orders/app

• IAM: creating filters needs logs:PutMetricFilter (and logs:TestMetricFilter to test); the metric is published by CloudWatch Logs, so the writer of the logs needs no cloudwatch:PutMetricData.
• Choosing: plain-text logs you can't change → metric filter; code you own and many dimensions or values → EMF; values you need immediately at sub-minute granularity from long-running code → PutMetricData with StorageResolution 1 (EMF also supports StorageResolution 1).

Traps

• Trap: "Create the metric filter now and it will count last week's errors." → Reality: filters only see events ingested after creation; use Logs Insights for history.
• Trap: "Use a metric filter with five dimensions." → Reality: a metric filter takes at most 3 dimensions; EMF dimension sets take up to 30 keys.
• Trap: "Put orderId in the EMF Dimensions so each order is traceable." → Reality: that creates one metric per order; keep it as a plain field and search it in Logs Insights.
• Trap: "EMF needs the agent everywhere." → Reality: in Lambda, writing the JSON to stdout is enough.
• Trap: "A default value of 0 works with dimensions." → Reality: you can't set a default value on a filter that has dimensions.
• Trap: "Metric filters work on any log group." → Reality: not on Infrequent Access log groups (Standard class only); the same goes for EMF extraction.

Practice questions

1. A Lambda function processes orders. The team needs a metric OrdersPlaced broken down by four attributes (Region, Channel, PaymentType, Tier), and every order's full context must stay searchable in the logs. The function must not make extra synchronous API calls. What should the developer do?

   • A. Call PutMetricData from the handler with the four attributes as dimensions.
   • B. Write each order as an embedded metric format JSON line to standard output with a dimension set containing the four keys.
   • C. Create a metric filter on the function's log group that extracts the four attributes as dimensions.
   • D. Record the four attributes as X-Ray annotations and graph them in CloudWatch.

2. A developer adds a metric filter for ERROR to the log group /orders/app and creates an alarm on the resulting metric. Which TWO statements are correct? (Select TWO.)

   • A. The filter also counts ERROR events ingested during the week before it was created.
   • B. The filter can't be created if /orders/app uses the Infrequent Access log class.
   • C. Adding requestId as a filter dimension can cause CloudWatch Logs to disable the filter.
   • D. A default value of 0 can be combined with up to three dimensions.
   • E. Metrics created by metric filters are stored at 1-second resolution by default.

3. CloudWatch Logs ingestion retention and the agent

Concept

• Log group: a named container (for example /orders/app or /aws/lambda/checkout) that owns the settings: retention, KMS key, log class, data protection, metric and subscription filters. Log stream: a sequence of events from one source inside a group (one instance, container or function instance). Log event: a timestamp plus a message.
• Logs arrive from AWS services (Lambda, API Gateway, ECS with the awslogs driver, Elastic Beanstalk, VPC Flow Logs, Route 53 query logs, CloudTrail), from the unified CloudWatch agent on servers, or from your code calling PutLogEvents through an SDK.
• By default logs are kept forever and encrypted at rest with a service-managed key; you choose a retention period, an optional customer managed KMS key, and a log class per group.

Exam facts

• Retention values: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288 or 3653 days (1 day to 10 years), or never expire (the default; restore it with DeleteRetentionPolicy). Expired events are typically deleted within 72 hours and stop counting toward storage once marked.
• PutLogEvents limits: batch up to 1,048,576 bytes (sum of UTF-8 messages plus 26 bytes per event), up to 10,000 events, events in chronological order, a batch may not span more than 24 hours; events more than 2 hours in the future or older than 14 days (or older than the retention period) are rejected individually and reported in rejectedLogEventsInfo (tooNewLogEventStartIndex, tooOldLogEventEndIndex, expiredLogEventEndIndex).
• ⚠️ Changed: a single log event can be up to 1 MB (since 2025-04-02, also for the OTLP endpoint). Older material may say 256 KB.
• ⚠️ Changed: the sequence token is ignored, several PutLogEvents calls may write to the same stream in parallel, and throttling is per account and Region (5,000 requests per second by default, adjustable). Older material may say every call needs the previous call's nextSequenceToken, fails with InvalidSequenceTokenException or DataAlreadyAcceptedException otherwise, and each stream accepts only 5 requests per second.
• UnrecognizedClientException from PutLogEvents usually means invalid access keys; ResourceNotFoundException means the group or stream doesn't exist (create them first, or let the agent or service do it).
• Log classes (chosen at creation, can't be changed later): Standard (all features) and Infrequent Access (lower ingestion price; storage and Logs Insights prices are the same): IA keeps ingestion, KMS encryption, cross-account sharing, masking, export to S3 and most Logs Insights commands, but has no metric filters, subscription filters, Live Tail, EMF extraction, anomaly detection, field indexes, or GetLogEvents/FilterLogEvents. A third DELIVERY class exists only for Lambda logs delivered to S3 or Firehose (kept 2 days).
• Data protection policies (account-level and/or one per log group) detect sensitive data (credentials, credit cards, PII, PHI) with managed or custom data identifiers at ingestion and mask it at every egress point (console, Logs Insights, metric filters, subscription filters). Viewing unmasked data needs logs:Unmask. Only events ingested after the policy are masked. Findings raise the LogEventsWithFindings metric.
• Live Tail: streams new events as they are ingested, with filter terms and highlighting; up to 10 log groups per session, sessions up to 3 hours, 15 concurrent sessions per account by default, billed per minute; Standard class only; aws logs start-live-tail from the CLI.
• Encryption: always encrypted; associating a customer managed KMS key (create-log-group --kms-key-id or associate-kms-key): see notes/security-encryption-guide.md#12-cloudwatch-logs-encryption-and-codebuild-secrets.
• Quotas worth recognising: 1,000,000 log groups per account (adjustable); CreateLogStream 50 TPS; DescribeLogStreams 25 TPS; FilterLogEvents and GetLogEvents are low-TPS read APIs, so don't poll them from production code.
• Getting logs off a server: nothing reaches CloudWatch Logs from EC2 or on-premises servers unless an agent or your code sends it. The unified CloudWatch agent collects log files (and Windows event logs, journald), system metrics (memory, disk, swap, processes via procstat), StatsD/collectd custom metrics, EMF, and since version 1.300025.0 X-Ray and OpenTelemetry traces. Configuration lives in a JSON file or, for a fleet, in an SSM Parameter Store parameter named AmazonCloudWatch-*. The instance role needs CloudWatchAgentServerPolicy. Agent configuration details, IMDSv2 and EC2 metrics: notes/ec2-mastery-guide.md#11-cloudwatch-for-ec2-metrics-agent-and-status-checks.
• ⚠️ Changed: the old CloudWatch Logs agent (awslogs) is deprecated and doesn't work on instances that require IMDSv2; use the unified CloudWatch agent. Older material may say the CloudWatch Logs agent is a valid way to ship logs.

Developer angle

aws logs create-log-group --log-group-name /orders/app --log-group-class STANDARD
aws logs put-retention-policy --log-group-name /orders/app --retention-in-days 30
aws logs create-log-stream --log-group-name /orders/app --log-stream-name web-1
aws logs put-log-events --log-group-name /orders/app --log-stream-name web-1 \
  --log-events '[{"timestamp":1760259600000,"message":"{\"level\":\"ERROR\",\"orderId\":\"42\"}"}]'

aws logs tail /orders/app --follow --since 15m --filter-pattern ERROR
aws logs start-live-tail \
  --log-group-identifiers arn:aws:logs:eu-west-1:111122223333:log-group:/orders/app \
  --log-event-filter-pattern ERROR
aws logs delete-retention-policy --log-group-name /orders/app    # back to "never expire"

Writing structured logs from code (the SDK batches nothing for you; prefer a library, the agent or the service integration in production):

import json, time
import boto3

logs = boto3.client("logs")
logs.put_log_events(logGroupName="/orders/app", logStreamName="web-1", logEvents=[{
    "timestamp": int(time.time() * 1000),
    "message": json.dumps({"level": "INFO", "orderId": "42", "latencyMs": 87})}])

• IAM for writers: logs:CreateLogGroup (if the code creates it), logs:CreateLogStream, logs:PutLogEvents, scoped to arn:aws:logs:<region>:<account>:log-group:/orders/app:*. Lambda gets these from AWSLambdaBasicExecutionRole; servers from CloudWatchAgentServerPolicy.
• Log JSON, not free text: JSON fields are discovered automatically by Logs Insights and can be used in metric filters and subscription filter patterns.

Traps

• Trap: "Logs expire after 30 days by default." → Reality: new log groups never expire until you set a retention policy; unmanaged groups grow storage cost forever.
• Trap: "Switch the noisy log group to Infrequent Access to save money and keep its alarms." → Reality: the class is fixed at creation, and Infrequent Access has no metric filters or subscription filters.
• Trap: "Add retry logic for InvalidSequenceTokenException when two threads log to one stream." → Reality: sequence tokens are ignored now; parallel writes to one stream are fine.
• Trap: "The agent is installed, so logs flow." → Reality: without an instance role allowing the logs actions (CloudWatchAgentServerPolicy) and network access to the CloudWatch Logs endpoint, nothing arrives.
• Trap: "A data protection policy masks the logs we already stored." → Reality: masking applies only to events ingested after the policy exists.
• Trap: "Use the CloudWatch Logs agent on new AL2023 instances." → Reality: it's deprecated and fails with IMDSv2-only instances; use the unified agent.

Practice questions

1. Compliance requires that an application's logs in CloudWatch Logs are kept for 400 days and then deleted automatically. The log group currently uses the default retention. Which solution has the LEAST operational overhead?

   • A. Export the log group to Amazon S3 every day, add an S3 Lifecycle expiration rule, and then delete the log group.
   • B. Schedule a Lambda function that deletes log streams older than 400 days.
   • C. Recreate the log group in the Infrequent Access log class, which expires events after 400 days.
   • D. Run aws logs put-retention-policy --log-group-name /payments/app --retention-in-days 400.

2. Applications on EC2 instances that require IMDSv2 must send /var/log/app/*.log and memory utilization to CloudWatch. The instances run the older CloudWatch Logs agent, and no logs arrive. Which TWO actions fix the problem? (Select TWO.)

   • A. Replace the CloudWatch Logs agent with the unified CloudWatch agent and configure the log files and memory metrics.
   • B. Enable EC2 detailed monitoring on the instances.
   • C. Change the log group retention to never expire.
   • D. Create a subscription filter on the target log group.
   • E. Attach the CloudWatchAgentServerPolicy managed policy to the instances' IAM role.

4. Logs Insights subscriptions and S3 export

Concept

• Once logs are in CloudWatch Logs you can use them in three ways, and the exam's deciding constraint is usually latency:
  • Query them on demand with CloudWatch Logs Insights (interactive analytics over stored logs, seconds to minutes).
  • Stream them in near real time with subscription filters to Kinesis Data Streams, Amazon Data Firehose, Lambda or OpenSearch Service (usually under 3 minutes).
  • Export them in batches to Amazon S3 with an export task (an export can lag ingestion by as much as 12 hours).
• Logs Insights reads what is already stored and answers when the query finishes; to watch new events arrive use Live Tail (§3), and to alert use metric filters or log alarms with an alarm (§2, §5).

Exam facts

• ⚠️ Changed: Logs Insights supports three query languages: Logs Insights QL (purpose-built, pipe syntax: fields, filter, stats, sort, limit, parse, display, dedup, pattern…), OpenSearch PPL and OpenSearch SQL (joins and sub-queries across log groups). Older material may say Logs Insights has only its own purpose-built language.
• Fields: @timestamp, @message, @logStream, @log always exist; JSON fields and fields of AWS service logs (Lambda, CloudTrail, VPC Flow Logs, Route 53) are discovered automatically. Plain-text values can be extracted with parse.
• Scope of a query: up to 50 log groups through logGroupNames/logGroupIdentifiers, or a SOURCE command that selects log groups by name prefix, account, log class or tags. A monitoring account in cross-account observability can query linked source accounts (pass log group ARNs). Field indexes (Standard class) cut the data scanned for large log-group sets.
• ⚠️ Changed: up to 100 concurrent Logs Insights QL queries per account, including dashboard queries (15 for OpenSearch PPL or SQL), since 2026-03-09 (most commercial Regions); a Logs Insights QL query can return up to 100,000 results (since 2026-05-15), paginated at 10,000 per GetQueryResults call. Older material may say 30 concurrent queries and 10,000 results.
• Queries time out after 60 minutes; results are stored for 7 days; you pay per GB of uncompressed data scanned, whatever the language. StartQuery and GetQueryResults are limited to 10 calls per second each. Queries can be saved (with parameters) and added to dashboards. Log events with timestamps older than the log group's creation can't be queried.
• Logs Insights works on Infrequent Access log groups too (most QL commands, plus PPL and SQL), so querying is the way to read IA logs.
• Subscription filters: a filter pattern plus a destination ARN per filter. ⚠️ Changed: up to 5 subscription filters per log group, plus one account-level subscription filter per Region (since January 2024). Older material may say 2 per log group.
• Subscriptions work only on Standard log groups. Events arrive base64-encoded and gzip-compressed, usually within 3 minutes of ingestion, at least once (duplicates possible), with messageType DATA_MESSAGE (or CONTROL_MESSAGE reachability checks), owner, logGroup, logStream, subscriptionFilters and logEvents. A Lambda destination receives {"awslogs": {"data": "<base64 gzip>"}}.
• Delivery failures: retryable errors (throttling, 5xx) are retried for up to 24 hours, then dropped. Non-retryable errors (AccessDeniedException, ResourceNotFoundException) disable the filter for up to 10 minutes, and events in that window are skipped. Watch DeliveryThrottling and DeliveryErrors; with Kinesis use --distribution Random (default ByLogStream can overload one shard) or on-demand streams.
• Permissions by destination: Kinesis Data Streams and Firehose need an IAM role that logs.amazonaws.com can assume (restrict it with aws:SourceArn) with kinesis:PutRecord or firehose:PutRecord; Lambda needs a resource-based policy allowing logs.amazonaws.com to call lambda:InvokeFunction, scoped to the log group ARN with --source-arn and --source-account. Never subscribe a function's own log group to that function (recursion).
• Cross-account delivery: the recipient account creates a CloudWatch Logs destination (PutDestination: target Kinesis or Firehose stream + role) and attaches a destination access policy (PutDestinationPolicy) allowing logs:PutSubscriptionFilter for the sender account ID or the whole organization (aws:PrincipalOrgID). The sender then creates a subscription filter whose --destination-arn is the destination. The log group and the destination must be in the same Region; the stream behind it may be elsewhere. For organization-wide replication without streams, CloudWatch also offers cross-account, cross-Region log centralization rules (since 2025-09-17).
• Export to S3: CreateExportTask with a time range (--from/--to in epoch milliseconds), bucket and optional prefix; same or different account; SSE-S3 or SSE-KMS buckets (not DSSE-KMS); Object Lock buckets allowed. Only one active export task per account; tasks time out after 24 hours (shorten the range); exported objects are gzip files and ordering inside them isn't guaranteed. AWS recommends subscriptions (Firehose) instead of repeated exports for continuous archiving. The bucket policy must allow the CloudWatch Logs service principal to write.
• ⚠️ Changed: the Firehose destination is now called Amazon Data Firehose (renamed 2024-02-09; same firehose API and CLI). Older material may say Kinesis Data Firehose. Kinesis Data Analytics for SQL, sometimes drawn as a Kinesis consumer, is discontinued; Amazon Managed Service for Apache Flink replaces it. Details: notes/sqs-sns-kinesis-guide.md#13-amazon-data-firehose.

Developer angle

Logs Insights QL error trend (JSON fields such as level are discovered automatically):

fields @timestamp, @message, @logStream
| filter level = "ERROR" and service = "checkout"
| stats count(*) as errors by bin(5m)
| sort errors desc
| limit 20

Run it from the CLI (times in epoch seconds), then fetch the results with the returned ID:

QUERY_ID=$(aws logs start-query --log-group-names /orders/app /orders/worker \
  --start-time 1760256000 --end-time 1760259600 \
  --query-string 'fields @timestamp, requestId, latencyMs | sort latencyMs desc | limit 20' \
  --query queryId --output text)
aws logs get-query-results --query-id "$QUERY_ID"

Subscription filters to Lambda (resource-based policy, no role) and to Kinesis (role, random distribution):

aws lambda add-permission --function-name log-processor --statement-id cwlogs-orders \
  --principal logs.amazonaws.com --action lambda:InvokeFunction \
  --source-arn "arn:aws:logs:eu-west-1:111122223333:log-group:/orders/app:*" \
  --source-account 111122223333
aws logs put-subscription-filter --log-group-name /orders/app --filter-name to-lambda \
  --filter-pattern '{ $.level = "ERROR" }' \
  --destination-arn arn:aws:lambda:eu-west-1:111122223333:function:log-processor

aws logs put-subscription-filter --log-group-name /orders/app --filter-name to-stream \
  --filter-pattern "" --distribution Random \
  --destination-arn arn:aws:kinesis:eu-west-1:111122223333:stream/orders-logs \
  --role-arn arn:aws:iam::111122223333:role/CWLtoKinesisRole

Cross-account: the recipient (999999999999) publishes a destination; the sender subscribes to it:

# recipient account
aws logs put-destination --destination-name central-logs \
  --target-arn arn:aws:kinesis:eu-west-1:999999999999:stream/central-logs \
  --role-arn arn:aws:iam::999999999999:role/CWLtoKinesisRole
aws logs put-destination-policy --destination-name central-logs --access-policy file://access.json
# sender account
aws logs put-subscription-filter --log-group-name /orders/app --filter-name to-central \
  --filter-pattern "" \
  --destination-arn arn:aws:logs:eu-west-1:999999999999:destination:central-logs

access.json (allow one sender account to subscribe):

{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"AWS": "111111111111"},
  "Action": "logs:PutSubscriptionFilter", "Resource": "arn:aws:logs:eu-west-1:999999999999:destination:central-logs"}]}

One-off export of a day of logs (epoch milliseconds):

aws logs create-export-task --log-group-name /orders/app \
  --from 1759276800000 --to 1759363200000 \
  --destination orders-log-archive --destination-prefix app/2025-10-01

Traps

• Trap: "Export to S3 every few minutes for near-real-time analytics." → Reality: exports can lag ingestion by as much as 12 hours and only one export task runs at a time; use a Firehose subscription.
• Trap: "Point the sender's subscription filter straight at the Kinesis stream in the other account." → Reality: cross-account delivery goes through a CloudWatch Logs destination in the recipient account plus its access policy.
• Trap: "Use Logs Insights to alert within seconds." → Reality: it's an on-demand query engine; alert with a metric filter or log alarm plus an alarm.
• Trap: "Subscribing Lambda only needs the subscription filter." → Reality: the function also needs a resource-based policy for logs.amazonaws.com; Kinesis and Firehose need a role instead.
• Trap: "Infrequent Access logs can't be read at all." → Reality: they can be queried with Logs Insights; they just can't stream (no subscription filters) or feed metric filters.
• Trap: "A Logs Insights query can run for hours over years of data." → Reality: it times out after 60 minutes; narrow the time range or the log-group set.

Practice questions

1. A security team needs every log event from a production account's /payments/app log group delivered within minutes to a Kinesis data stream in a separate security account, for intrusion detection. Which configuration meets these requirements?

   • A. Run CreateExportTask every hour to an S3 bucket in the security account and read new objects from there.
   • B. In the security account, create a CloudWatch Logs destination for the stream with an access policy that allows the production account; in the production account, create a subscription filter that targets the destination ARN.
   • C. Schedule a CloudWatch Logs Insights query every minute in the production account and write the results to the stream.
   • D. In the production account, create a subscription filter whose destination ARN is the stream in the security account, using a role in the production account.

2. A developer must find the 20 slowest requests of the last hour across three log groups from a shell script. Which TWO statements are correct? (Select TWO.)

   • A. A Logs Insights query can run for up to 24 hours before it times out.
   • B. Query results are stored for 90 days after the query completes.
   • C. aws logs start-query returns a queryId, and the rows are retrieved with aws logs get-query-results.
   • D. Logs Insights can query log groups that use the Infrequent Access log class.
   • E. Queries written in OpenSearch SQL are not billed for the data they scan.

5. CloudWatch alarms and Synthetics canaries

Concept

• A metric alarm watches one metric, a metric math expression, an anomaly detection band or a Metrics Insights query, compares it with a threshold over a number of periods, and moves between three states: OK, ALARM and INSUFFICIENT_DATA. Actions run when the state changes.
• A composite alarm watches the states of other alarms through a rule expression, so you page on "latency AND errors" instead of on every single symptom.
• A Synthetics canary is a script that CloudWatch runs on a timer against your API or web UI. Because it generates its own requests, an outage shows up at 3 a.m. with zero real traffic, and its metrics feed alarms.

Exam facts

• Evaluation: Period (seconds per data point), EvaluationPeriods (N most recent periods) and DatapointsToAlarm (M of those N must breach: "M out of N"). Period × EvaluationPeriods can't exceed 1 day for periods under 1 hour, or 7 days otherwise. A new alarm starts in INSUFFICIENT_DATA.
• ⚠️ Changed: an alarm period can be 10, 20 or 30 seconds (a high-resolution alarm, billed higher and meaningful only for metrics published with StorageResolution 1) or any multiple of 60 seconds. Older material may say only 10 and 30 seconds.
• Missing data (TreatMissingData): missing (default: the alarm looks further back or goes to INSUFFICIENT_DATA), notBreaching (gaps count as good), breaching (gaps count as bad: useful for "heartbeat stopped"), ignore (keep the current state). Alarms on AWS/DynamoDB metrics always ignore missing data.
• Statistics: Statistic (Average, Sum, Minimum, Maximum, SampleCount) or ExtendedStatistic (p99, tm90…); EvaluateLowSampleCountPercentile = ignore keeps a percentile alarm steady when there are too few samples. Specifying a Unit that the metric doesn't use leaves the alarm in INSUFFICIENT_DATA, so AWS recommends omitting it.
• ⚠️ Changed: alarm actions can notify an SNS topic, invoke a Lambda function directly (since 2023-12-22), run an EC2 action (stop, terminate, reboot, recover), run an EC2 Auto Scaling policy, create a Systems Manager OpsItem or Incident Manager incident, or start a CloudWatch investigation (the last three on ALARM only). Older material may say only SNS, EC2 and Auto Scaling actions.
• Up to 5 actions per state. Actions run once per state change; only Auto Scaling actions repeat every minute while the alarm stays in ALARM. CloudWatch doesn't validate that an action's target exists. Alarms based on anomaly detection can't have Auto Scaling actions. Every state change is also delivered (guaranteed) to EventBridge as a CloudWatch Alarm State Change event from source aws.cloudwatch, so any EventBridge target can react; creating, updating or deleting an alarm emits CloudWatch Alarm Configuration Change.
• Lambda action: the function's resource-based policy must allow the principal lambda.alarms.cloudwatch.amazonaws.com to call lambda:InvokeFunction, scoped with AWS:SourceAccount and AWS:SourceArn (the alarm ARN); the function can be in another account; the invocation is asynchronous and the event carries source aws.cloudwatch, alarmArn and alarmData (current and previous state, configuration).
• EC2 actions use the ARNs arn:aws:automate:<region>:ec2:stop|terminate|reboot|recover and the service-linked role AWSServiceRoleForCloudWatchEvents; recovery and status checks are EC2 topics (notes/ec2-mastery-guide.md#11-cloudwatch-for-ec2-metrics-agent-and-status-checks).
• Testing and maintenance: set-alarm-state sets a state until the next evaluation and runs that state's actions for real; disable-alarm-actions / enable-alarm-actions switch actions off and on; alarm mute rules (since 2026-02-10) silence actions during scheduled cron()/at() windows of 1 minute (PT1M) to 15 days (P15D) and re-run them at the end of the window if the alarm is still in that state; describe-alarm-history keeps 30 days of state changes.
• ⚠️ Changed: composite alarm rules use ALARM(), OK(), INSUFFICIENT_DATA(), TRUE, FALSE with AND, OR, NOT and parentheses, plus an AT_LEAST function for "at least N or N % of these alarms" (since 2025-11-11; exact syntax ⚠️ unverified); composite alarms can notify SNS, invoke Lambda, create OpsItems or incidents and start investigations, but still can't run EC2 or Auto Scaling actions. Older material may say composite alarms support only AND/OR and only SNS notifications.
• More composite facts: up to 100 child alarms per rule; one alarm can feed up to 150 composite alarms; an actions suppressor alarm (with wait and extension periods) silences the composite during, for example, a deployment; creating one requires cloudwatch:PutCompositeAlarm on *; composite alarms can't be cross-account.
• Other alarm sources: metric math (Metrics array with ReturnData), anomaly detection (ANOMALY_DETECTION_BAND + ThresholdMetricId), Metrics Insights queries (one alarm watching a whole tagged fleet), and log alarms on scheduled Logs Insights queries (since 2026-07-01). Quota warnings use AWS/Usage metrics with SERVICE_QUOTA() (cli-sdk-policies §8).
• ⚠️ Changed: canaries can be written in Node.js (Puppeteer or Playwright), Python (Selenium) or Java (since 2025-05-28, for API checks without a browser), and Node.js browser canaries can drive headless Chrome or Firefox (Python/Selenium canaries use Chrome only). Older material may say canaries are Node.js or Python scripts with headless Chrome only.
• How canaries run: as Lambda functions (cwsyn-<name>-…) in your account with the Synthetics runtime layer; a single run or a recurring schedule (rate() down to once a minute, or cron()); artifacts (screenshots, HAR files, logs, reports) go to an S3 bucket you name; logs to CloudWatch Logs. To test private endpoints, attach the canary to VPC subnets and security groups (it then needs routes to S3 and CloudWatch). Turn on active tracing to see canaries on the X-Ray trace map.
• Blueprints: heartbeat monitoring (load URLs, screenshots, HAR), API canary (multi-step HTTP calls, can import an API Gateway API), broken link checker, visual monitoring (compare screenshots with a baseline), canary recorder (Chrome extension that writes the script), GUI workflow builder (click, type, verify text). ⚠️ Changed: a Multi checks blueprint (since 2025-10-22) runs up to 10 HTTP, DNS, SSL and TCP checks from a JSON configuration, with authentication through Secrets Manager. Older material may say there are only the six original blueprints.
• Canary metrics (CloudWatchSynthetics namespace, dimension CanaryName, plus StepName for steps): SuccessPercent, Duration, Failed, 2xx, 4xx, 5xx. Alarm on SuccessPercent below 100 (or 90) to page on failures.
• Real users instead of scripts: CloudWatch RUM (browser and mobile sessions). Canaries need no real traffic.

Developer angle

A p99 latency alarm (3 of 5 one-minute points), a heartbeat-style alarm, and a composite that ignores deployments:

aws cloudwatch put-metric-alarm --alarm-name checkout-p99-high \
  --namespace OrdersApp --metric-name CheckoutLatency --dimensions Name=Service,Value=checkout \
  --extended-statistic p99 --period 60 --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 800 --comparison-operator GreaterThanThreshold --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-alerts

aws cloudwatch put-metric-alarm --alarm-name orders-heartbeat-missing \
  --namespace OrdersApp --metric-name OrdersPlaced --dimensions Name=Service,Value=checkout \
  --statistic Sum --period 300 --evaluation-periods 2 --threshold 1 \
  --comparison-operator LessThanThreshold --treat-missing-data breaching \
  --alarm-actions arn:aws:lambda:eu-west-1:111122223333:function:orders-remediate

aws cloudwatch put-composite-alarm --alarm-name checkout-degraded \
  --alarm-rule 'ALARM("checkout-p99-high") AND ALARM("checkout-5xx-high") AND NOT ALARM("deploy-in-progress")' \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:pager

Let the alarm invoke the remediation function, then test the whole chain:

aws lambda add-permission --function-name orders-remediate --statement-id cw-alarm \
  --action lambda:InvokeFunction --principal lambda.alarms.cloudwatch.amazonaws.com \
  --source-account 111122223333 \
  --source-arn arn:aws:cloudwatch:eu-west-1:111122223333:alarm:orders-heartbeat-missing

aws cloudwatch set-alarm-state --alarm-name orders-heartbeat-missing \
  --state-value ALARM --state-reason "end-to-end test of the remediation action"
aws cloudwatch describe-alarm-history --alarm-name orders-heartbeat-missing \
  --history-item-type StateUpdate

Create and start a canary whose script is already zipped in S3 (runtime version names change often, so the current one is passed in a variable):

aws synthetics create-canary --name checkout-api \
  --code Handler=index.handler,S3Bucket=canary-code-111122223333,S3Key=checkout-canary.zip \
  --artifact-s3-location s3://canary-artifacts-111122223333/checkout/ \
  --execution-role-arn arn:aws:iam::111122223333:role/CanaryExecutionRole \
  --schedule Expression="rate(5 minutes)" \
  --runtime-version "$SYNTHETICS_RUNTIME"
aws synthetics start-canary --name checkout-api
aws synthetics get-canary-runs --name checkout-api --max-results 5

• The canary's execution role needs to write artifacts to the S3 bucket, publish CloudWatchSynthetics metrics and write logs; whoever creates the canary also needs iam:PassRole on that role.
• aws cloudwatch describe-alarms --state-value ALARM lists what is firing now; --alarm-types CompositeAlarm filters composites.

Traps

• Trap: "The alarm is stuck in INSUFFICIENT_DATA, so CloudWatch is broken." → Reality: usually the dimensions, namespace or unit in the alarm don't match the published metric exactly, or the period is shorter than the metric's resolution.
• Trap: "A 10-second alarm on a standard-resolution metric reacts faster." → Reality: it receives data in only one period of six, lapses into INSUFFICIENT_DATA and costs more; publish with StorageResolution 1.
• Trap: "set-alarm-state is a dry run." → Reality: it executes the actions of the new state.
• Trap: "The alarm will keep emailing every minute while in ALARM." → Reality: notifications fire once per state change; only Auto Scaling actions repeat.
• Trap: "Use a composite alarm to reboot the instance when both checks fail." → Reality: composite alarms can't run EC2 or Auto Scaling actions; put the action on a metric alarm.
• Trap: "Adding the Lambda ARN to AlarmActions is enough." → Reality: the function needs a resource-based policy for lambda.alarms.cloudwatch.amazonaws.com.
• Trap: "A canary can't test an internal API." → Reality: run it in the VPC (subnets and security groups) with routes to S3 and CloudWatch.

Practice questions

1. An on-call team wants one page only when both the p99 latency alarm and the 5xx error-rate alarm of a service are in ALARM, and no page while the deploy-in-progress alarm is in ALARM. The solution must not need custom code. What should the developer configure?

   • A. A metric math alarm that averages the latency and error-rate metrics against one threshold.
   • B. A Lambda function subscribed to both alarms' SNS topics that checks the third alarm before paging.
   • C. A composite alarm with the rule ALARM("p99-high") AND ALARM("5xx-high") AND NOT ALARM("deploy-in-progress") and an SNS action.
   • D. Higher EvaluationPeriods values on both metric alarms so that they fire less often.

2. A developer must detect failures of a public checkout API every 5 minutes, even at night when there is no customer traffic, and automatically invoke a remediation Lambda function when failures occur. Which TWO steps meet these requirements? (Select TWO.)

   • A. Enable X-Ray active tracing on the API stage and alarm on the trace count.
   • B. Create a CloudWatch Synthetics canary from the API canary blueprint with a rate(5 minutes) schedule.
   • C. Create a metric filter on the API's access logs that counts 5xx responses.
   • D. Create a composite alarm whose action reboots the API's backend instances.
   • E. Create an alarm on the canary's SuccessPercent metric with the function ARN as the alarm action, and allow lambda.alarms.cloudwatch.amazonaws.com in the function's resource-based policy.

6. EventBridge rules event patterns and targets

Concept

• Amazon EventBridge (formerly CloudWatch Events) is a serverless event router. Producers put JSON events on an event bus: AWS services publish to the default bus automatically, your applications call PutEvents on the default or a custom bus, and SaaS partners publish to partner buses.
• On a rule-based bus, a rule holds an event pattern; each event that matches is delivered to up to five targets (Lambda, SQS, SNS, Step Functions, Kinesis, ECS tasks, API destinations, another bus…), optionally reshaped first.
• Around the bus: EventBridge Scheduler runs targets on a schedule, archive and replay keeps and re-sends past events, and EventBridge Pipes connects one source to one target point to point with filtering and enrichment.
• Since 2026-09-24 a second kind of custom bus exists (subscribers instead of rules); this section describes the rule-and-target model that the exam and most material use, now called Custom Event Bus - Classic. The new bus type: §7.

Exam facts

• Event envelope: version, id, detail-type, source, account, time, region, resources, detail. Patterns match on any of these fields.
• PutEvents: up to 10 entries per request. ⚠️ Changed: a request to the default bus or a Classic custom bus can carry up to 1 MB (1,048,576 bytes) in total, and one entry may use all of it (since 2026-01-29; not in every Region at launch). Older material may say 256 KB. For bigger payloads put the data in S3 and send its URL.
• Partial failure: the HTTP status can be 200 while some entries failed: check FailedEntryCount, then resend only the entries whose result has an ErrorCode (ErrorMessage explains it), with backoff. Publishing to a bus name that doesn't exist also returns 200, matches no rule and silently drops the event.
• Quotas (Classic): PutEvents 10,000 requests/s in us-east-1, us-west-2 and eu-west-1, down to 400/s in smaller Regions (adjustable); rule invocations throttle at 18,750/s in the largest Regions (delayed, not lost); 300 rules per bus in most Regions (adjustable); 5 targets per rule (fixed); event pattern up to 2,048 characters; 30 rules with wildcards per bus.
• Pattern matching: listed values are OR ("state": ["stopped", "terminated"]), separate fields are AND, matching is exact and case-sensitive unless you use an operator: prefix, suffix, anything-but (also with prefix, suffix, wildcard, equals-ignore-case), numeric ranges, cidr, exists, equals-ignore-case, wildcard (buses only, not Pipes) and $or across fields (up to 1,000 combinations). A field that appears twice in a pattern keeps only the last definition. Test with TestEventPattern.
• Target permissions: Lambda, SNS and SQS targets use a resource-based policy (principal events.amazonaws.com, aws:SourceArn = the rule ARN) or, ⚠️ Changed: since 2025-03-11 an IAM execution role on the target. Older material may say these targets always need a resource-based policy. CloudWatch Logs targets use a log resource policy only (don't set RoleArn); Kinesis, Step Functions, ECS, CodeBuild, CodePipeline, Systems Manager and cross-account buses use an IAM role trusted by events.amazonaws.com; API Gateway uses its resource policy (execute-api:Invoke).
• What the target receives: the whole event (default), part of it (InputPath), a constant (Input), or an input transformer (InputPathsMap extracts fields, InputTemplate builds the payload).
• Retries and DLQ: by default EventBridge retries a failed delivery for up to 24 hours and 185 attempts with backoff (RetryPolicy: MaximumEventAgeInSeconds 60–86,400, MaximumRetryAttempts 0–185). Errors such as missing permissions or a deleted target are not retried. Undeliverable events go to a dead-letter queue if configured: a standard SQS queue in the same Region (FIFO not supported) whose policy allows events.amazonaws.com; each message carries RULE_ARN, TARGET_ARN, ERROR_CODE (for example NO_PERMISSIONS, NO_RESOURCE, THROTTLING), ERROR_MESSAGE, EXHAUSTED_RETRY_CONDITION and RETRY_ATTEMPTS.
• Metrics (AWS/Events): MatchedEvents, TriggeredRules, Invocations, FailedInvocations, ThrottledRules, InvocationsSentToDLQ, InvocationsFailedToBeSentToDLQ. Matched but not delivered → look at FailedInvocations and the DLQ.
• ⚠️ Changed: scheduled rules are a legacy feature (default bus only, UTC only, 1-minute precision); AWS recommends EventBridge Scheduler. Older material may say EventBridge scheduled rules are the way to run cron jobs.
• EventBridge Scheduler: at(yyyy-mm-ddThh:mm:ss) one-time schedules, rate(5 minutes), cron(minutes hours day-of-month month day-of-week year) with ScheduleExpressionTimezone (daylight saving handled); FlexibleTimeWindow (OFF or FLEXIBLE with a maximum window) to spread load; templated targets (Lambda, SQS, SNS, Step Functions, EventBridge…) and universal targets for more than 6,000 API actions; every target needs an execution role (RoleArn); retry policy and DLQ per schedule; ActionAfterCompletion DELETE removes finished one-time schedules; built to handle millions of schedules.
• Archive and replay: an archive belongs to one source bus, optionally filters with a pattern, and keeps events indefinitely by default or for N days. A replay re-sends a time window to the source bus only (optionally to chosen rules), adds a replay-name field, doesn't preserve order, runs at most 10 at a time per account and Region, and shouldn't start until about 10 minutes after the newest event you need. Replayed events are kept out of archives automatically.
• Pipes: one source (SQS, Kinesis Data Streams, DynamoDB Streams, Amazon MSK, self-managed Kafka, Amazon MQ) → optional filter (you pay only for events that pass) → optional enrichment (Lambda, Step Functions Express, API Gateway, API destination) → one target, with an IAM execution role. Use buses for many-to-many routing, Pipes for point-to-point. API destinations make any HTTPS endpoint a target, with a connection that keeps its credentials (default limit 300 invocations/s).
• Useful AWS events: EC2 Instance State-change Notification, Object Created from S3 (after the bucket enables EventBridge notifications), CodePipeline/CodeBuild/CodeDeploy state changes, CloudWatch Alarm State Change, AWS Health events, and API calls recorded by CloudTrail (§10). Classic buses offer no ordering guarantee; ordered delivery (FIFO within an EventGroupId) is a feature of the new Custom Event Bus (§7).

Developer angle

Pattern for custom high-value failures (order-failed-pattern.json):

{
  "source": ["com.example.orders"],
  "detail-type": ["OrderFailed"],
  "detail": {"amount": [{"numeric": [">=", 100]}], "channel": [{"anything-but": "test"}]}
}

Rule on a custom bus, Lambda permission, target with retry policy, DLQ and an input transformer (targets.json below):

aws events create-event-bus --name orders
aws events put-rule --name order-failed --event-bus-name orders \
  --event-pattern file://order-failed-pattern.json --state ENABLED
aws lambda add-permission --function-name order-failed-handler \
  --statement-id eventbridge-order-failed --action lambda:InvokeFunction \
  --principal events.amazonaws.com \
  --source-arn arn:aws:events:eu-west-1:111122223333:rule/orders/order-failed
aws events put-targets --rule order-failed --event-bus-name orders --targets file://targets.json
aws events test-event-pattern --event-pattern file://order-failed-pattern.json --event file://sample-event.json

[
  {
    "Id": "handler",
    "Arn": "arn:aws:lambda:eu-west-1:111122223333:function:order-failed-handler",
    "RetryPolicy": {"MaximumRetryAttempts": 10, "MaximumEventAgeInSeconds": 3600},
    "DeadLetterConfig": {"Arn": "arn:aws:sqs:eu-west-1:111122223333:order-events-dlq"},
    "InputTransformer": {
      "InputPathsMap": {"order": "$.detail.orderId", "amount": "$.detail.amount"},
      "InputTemplate": "{\"orderId\": <order>, \"amount\": <amount>, \"action\": \"refund-check\"}"
    }
  }
]

Publishing from code and resending only the failed entries:

import json
import boto3

events = boto3.client("events")
entries = [{"Source": "com.example.orders", "DetailType": "OrderFailed", "EventBusName": "orders",
            "Detail": json.dumps({"orderId": "42", "amount": 250, "channel": "web"})}]
response = events.put_events(Entries=entries)
if response["FailedEntryCount"]:
    failed = [entry for entry, result in zip(entries, response["Entries"]) if "ErrorCode" in result]
    events.put_events(Entries=failed)  # production code: back off and cap the attempts

A daily job at 02:00 Berlin time with Scheduler, and an archive plus a replay of one bad hour:

aws scheduler create-schedule --name nightly-report \
  --schedule-expression "cron(0 2 * * ? *)" --schedule-expression-timezone "Europe/Berlin" \
  --flexible-time-window Mode=OFF \
  --target '{"Arn":"arn:aws:lambda:eu-west-1:111122223333:function:nightly-report","RoleArn":"arn:aws:iam::111122223333:role/SchedulerInvokeReport"}'

aws events create-archive --archive-name orders-archive --retention-days 30 \
  --event-source-arn arn:aws:events:eu-west-1:111122223333:event-bus/orders
aws events start-replay --replay-name orders-fix-1 \
  --event-source-arn arn:aws:events:eu-west-1:111122223333:archive/orders-archive \
  --event-start-time 2026-10-11T08:00:00Z --event-end-time 2026-10-11T09:00:00Z \
  --destination Arn=arn:aws:events:eu-west-1:111122223333:event-bus/orders

Traps

• Trap: "PutEvents returned 200, so every event was accepted." → Reality: read FailedEntryCount and resend the failed entries; a misspelled bus name also returns 200 and drops the event.
• Trap: "Run the job at 02:00 local time with a scheduled rule." → Reality: scheduled rules are UTC only; use Scheduler with ScheduleExpressionTimezone.
• Trap: "Add a sixth target to the rule." → Reality: a rule has at most 5 targets; add another rule with the same pattern or fan out through SNS.
• Trap: "The rule matches, so the Lambda function must run." → Reality: without the resource-based policy (or an execution role), deliveries fail (FailedInvocations, NO_PERMISSIONS in the DLQ) and aren't retried.
• Trap: "Use an SQS FIFO queue as the DLQ." → Reality: EventBridge DLQs must be standard queues in the same Region.
• Trap: "Replay the archive into a test bus." → Reality: replays go back to the source bus only; restrict them to chosen rules instead.
• Trap: "EventBridge preserves event order." → Reality: Classic rule-based buses don't; design consumers to tolerate reordering, or use the new Custom Event Bus with an EventGroupId and a FIFO subscriber.

Practice questions

1. A Lambda function must run every day at 02:00 Europe/Berlin local time, staying correct across daylight saving changes, with the LEAST operational overhead. What should the developer configure?

   • A. An EventBridge Scheduler schedule with cron(0 2 * * ? *), ScheduleExpressionTimezone set to Europe/Berlin and an execution role that can invoke the function.
   • B. An EventBridge scheduled rule on the default bus with cron(0 2 * * ? *).
   • C. Two EventBridge scheduled rules, one for summer time and one for winter time, enabled and disabled by hand.
   • D. A cron job on an EC2 instance that calls aws lambda invoke at 02:00.

2. An EventBridge rule on a custom bus targets a Lambda function that a teammate created with the CLI. The rule's MatchedEvents metric rises, but the function is never invoked and FailedInvocations increases. Which TWO actions should the developer take? (Select TWO.)

   • A. Allow events.amazonaws.com to invoke the function in its resource-based policy with the rule ARN as source ARN, or give the target an execution role that allows lambda:InvokeFunction.
   • B. Request a higher quota for the rule's event pattern size.
   • C. Configure a standard SQS queue as the target's dead-letter queue and read the ERROR_CODE and ERROR_MESSAGE attributes of the failed events.
   • D. Move the rule to the default event bus, which doesn't need target permissions.
   • E. Enable an archive on the bus so that failed deliveries are retried automatically.

7. EventBridge schema registry and cross account events

Concept

• A schema describes the structure of an event. The schema registry stores schemas so producers and consumers agree on them, and turns them into code bindings: typed classes you import instead of parsing raw JSON.
• An event bus has a resource-based policy that decides which other accounts or organizations may send events to it (or manage rules on it). With it you build a central bus that aggregates events from every account in an organization.
• Since 2026-09-24 EventBridge offers a second custom bus design built for sharing across teams and accounts: consumers attach their own subscribers, and the bus retains events.

Exam facts

• Registries: aws.events (schemas of all AWS service events, read-only), discovered-schemas (filled by schema discovery, which you start on a bus and which infers schemas from the events flowing through it, for a charge), and custom registries you create or upload to. Formats: OpenAPI 3 and JSONSchema Draft4. Schemas are versioned; discovery adds a new version when an event's structure changes. Discovered schema names combine the event source and detail type (pattern <source>@<DetailType> ⚠️ unverified; check with aws schemas list-schemas).
• Code bindings: Java (Java8), Python (Python36), TypeScript (TypeScript3) and Go (Go1), from the console, the Schema Registry API (PutCodeBinding, GetCodeBindingSource) or the AWS Toolkits for VS Code and JetBrains; SAM templates can start from them.
• Discovery infers schemas of the original event, not of an input transformer's output; nesting beyond 255 levels is ignored; events over 1,000 KiB are skipped; it includes cross-account events by default; it isn't supported on buses encrypted with a customer managed key; the first 5 million processed events per month are free. Default quotas per Region: 10 registries, 100 schemas per registry, 100 versions per schema, 10 discoverers, 200 discovered schemas (adjustable).
• Bus resource policy: grants events:PutEvents (send) and optionally events:PutRule/events:PutTargets (manage rules) to account IDs, or to "*" restricted with aws:PrincipalOrgID; managed with put-permission / remove-permission, visible in describe-event-bus; Classic policy size up to 10,240 characters. The default bus already accepts AWS service events.
• Sender side: a rule whose target is the receiver's bus ARN. Cross-account event bus targets created since 2023-03-02 need an IAM role (RoleArn, trusted by events.amazonaws.com, allowed events:PutEvents on the target bus); the console creates it, the CLI doesn't. The sender pays for the events as custom events; the receiver pays nothing for them.
• Receiver side: rules on the receiving bus route the events (add account to patterns when many accounts can send). Events forwarded once aren't forwarded again to a third account or Region, and a rule on one bus can't chain events through a second bus of the same account to a third.
• ⚠️ Changed: since 2025-01-21 a rule can also deliver directly to SQS, SNS, Lambda, Kinesis Data Streams or API Gateway targets in another account of the same Region, provided the target has an execution role (RoleArn) in the bus account, which EventBridge assumes, and the target's resource-based policy allows the bus account; no intermediary bus is needed. Older material may say cross-account delivery always goes through an event bus in the other account.
• Cross-Region: a rule can target a bus in another supported Region; global endpoints add Region failover for PutEvents with event replication.
• ⚠️ Changed: EventBridge now has two custom bus types (2026-09-24). The new Custom Event Bus retains every event 1–365 days (24 hours by default), lets consumer accounts create subscribers (filter + optional transform + exactly one target each) after the owner shares the bus through AWS RAM (for example the AWSRAMEventBridgeEventBusV2SubscribeOnly permission), replays by moving a subscriber's start position, orders events within an EventGroupId, deduplicates within 5 minutes, and accepts 1–100 entries per PutEvents or PutRawEvents call (JSON, Avro, Protobuf, raw bytes) through the aws eventsv2 CLI. Existing custom buses were renamed Custom Event Bus - Classic and keep their rules, targets and APIs; the default bus still routes with rules. At launch it was available in 14 Regions. Older material may say there is a single kind of custom bus.
• Exam questions written before that launch assume the Classic model: resource policy on the bus, rules and up to 5 targets, archives for replay.

Developer angle

Discover schemas on a bus and download Java bindings for one of them:

aws schemas create-discoverer \
  --source-arn arn:aws:events:eu-west-1:111122223333:event-bus/orders
aws schemas list-schemas --registry-name discovered-schemas
aws schemas put-code-binding --registry-name discovered-schemas \
  --schema-name "com.example.orders@OrderFailed" --language Java8
aws schemas get-code-binding-source --registry-name discovered-schemas \
  --schema-name "com.example.orders@OrderFailed" --language Java8 order-failed-bindings.zip

Central bus in account 999999999999 that accepts events from every account of the organization (bus-policy.json):

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowOrgAccountsToPutEvents",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "events:PutEvents",
    "Resource": "arn:aws:events:eu-west-1:999999999999:event-bus/central-bus",
    "Condition": {"StringEquals": {"aws:PrincipalOrgID": "o-a1b2c3d4e5"}}
  }]
}

# receiving account
aws events create-event-bus --name central-bus
aws events put-permission --event-bus-name central-bus --policy file://bus-policy.json
aws events describe-event-bus --name central-bus

# each sending account: forward EC2 state changes; the role allows events:PutEvents on central-bus
aws events put-rule --name forward-ec2-state \
  --event-pattern '{"source":["aws.ec2"],"detail-type":["EC2 Instance State-change Notification"]}'
aws events put-targets --rule forward-ec2-state --targets \
  '[{"Id":"central","Arn":"arn:aws:events:eu-west-1:999999999999:event-bus/central-bus","RoleArn":"arn:aws:iam::111122223333:role/EventBridgeToCentralBus"}]'

• Least privilege on the sender role: events:PutEvents on the central bus ARN only; on the bus policy: organization condition instead of a bare "*".
• For SQS, SNS, Lambda, Kinesis or API Gateway targets in another account (same Region), you can now target the resource ARN directly: give the target a RoleArn in the bus account (trusted by events.amazonaws.com, allowed the delivery action such as sqs:SendMessage on that resource only), and allow the bus account in that resource's policy.

Traps

• Trap: "Set the bus policy principal to * so all our accounts can send." → Reality: without an aws:PrincipalOrgID (or account) condition, any AWS account can put events on the bus.
• Trap: "The receiving bus will route our events without any configuration." → Reality: the receiver needs its own rules; the sender's rule only delivers to the bus.
• Trap: "A CLI-created rule can target another account's bus without a role." → Reality: new cross-account bus targets need RoleArn with events:PutEvents.
• Trap: "Chain account A → account B → account C through rules." → Reality: events don't take a second cross-account or cross-Region hop.
• Trap: "Generate C# classes from the schema registry." → Reality: code bindings exist for Java, Python, TypeScript and Go.
• Trap: "Schema discovery records the transformed payload our target sees." → Reality: it infers the original event's schema.
• Trap: "Use the EventBridge schema registry for Kafka Avro schemas." → Reality: that is the AWS Glue Schema Registry (which the new Custom Event Bus can reference for Avro and Protobuf).

Practice questions

1. A company runs workloads in 40 accounts of one AWS Organization. A central operations account must receive every EC2 instance state-change event in its own event bus, with the LEAST operational overhead and least privilege. What should the developer do?

   • A. Attach a resource-based policy to the central bus that allows events:PutEvents from principal * with no condition, and create the forwarding rules in each account.
   • B. Create an IAM user with access keys in the central account and run a polling script in every account that calls PutEvents with those keys.
   • C. Create an organization trail in CloudTrail and query the EC2 events from S3 with Athena every five minutes.
   • D. Attach a resource-based policy to the central bus that allows events:PutEvents from principal * with an aws:PrincipalOrgID condition, and in each account create a rule that targets the central bus with an IAM role allowed to put events on it.

2. A developer wants strongly typed Java classes for the events that several teams already publish on a custom bus, without writing the schemas by hand. Which TWO steps accomplish this? (Select TWO.)

   • A. Create an archive on the bus with indefinite retention.
   • B. Add an input transformer to every rule so that events have the same shape.
   • C. Register the event formats in the AWS Glue Schema Registry and import them into the bus.
   • D. Start schema discovery on the bus with aws schemas create-discoverer.
   • E. Download code bindings for the discovered schema with the Java8 language.

8. Distributed tracing concepts and instrumentation

Concept

• AWS X-Ray follows one request through every component it touches. Each component that handles the request sends a segment (its name, timing, request and response, errors); a segment can contain subsegments for finer detail (a call to DynamoDB, an HTTP request, a SQL query, a block of your own code). All segments that share a trace ID form a trace.
• From traces X-Ray builds the trace map (service graph): nodes for services, edges for calls, each coloured by latency, errors (4xx), throttles (429) and faults (5xx). Downstream services that don't send their own data (DynamoDB, S3, external APIs) appear as inferred nodes built from your subsegments.
• The trace context travels between services in the X-Amzn-Trace-Id HTTP header (or the W3C traceparent header with OpenTelemetry).
• Getting data in takes two parts: instrumentation in the code (the X-Ray SDK, or OpenTelemetry through ADOT or the upstream SDKs) and a collector that relays it to X-Ray (the X-Ray daemon, the CloudWatch agent or an OpenTelemetry Collector). Lambda, API Gateway and Elastic Beanstalk provide the collector side for you.

Exam facts

• Header: X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;Parent=53995c3f42cd8ad8;Sampled=1. The root ID is version 1, the request time as 8 hex digits of epoch seconds, and a 96-bit random ID (24 hex digits). Sampled=1/0 carries the sampling decision downstream; Lineage is added by Lambda and other services. A client can forge the header, so an entry service may strip it.
• Limits: a segment document is at most 64 KB; trace data and service graph data are retained 30 days; X-Ray indexes up to 50 annotations per trace.
• Annotations vs metadata: annotations are key-value pairs with string, number or Boolean values, indexed for filter expressions, groups and GetTraceSummaries (key up to 500 alphanumeric characters, value up to 1,000 characters). Metadata are key-value pairs of any type (objects, lists), not indexed, grouped in namespaces (default if none): store debug payloads there. A user ID recorded with set_user is searchable with the user keyword.
• Error classes: Error = 4xx client errors, Throttle = 429, Fault = 5xx server errors; exceptions are recorded with stack traces.
• X-Ray SDKs (Java, Python, Node.js, Go, .NET, Ruby) provide middleware for incoming HTTP requests, handlers that patch AWS SDK clients, instrumented HTTP and SQL clients, and plugins that add EC2, ECS or Beanstalk metadata; the Java SDK also has an auto-instrumentation agent. The SDK sends segments as UDP datagrams to the daemon at 127.0.0.1:2000 unless AWS_XRAY_DAEMON_ADDRESS points elsewhere.
• ⚠️ Changed: the X-Ray SDKs and the X-Ray daemon are in maintenance mode since 2026-02-25 (security fixes only, no new features, no end-of-support date announced); AWS recommends OpenTelemetry for new and existing applications. The X-Ray service, its APIs and the CloudWatch trace views continue. Older material may say the X-Ray SDK plus daemon is the standard, fully supported way to trace.
• ⚠️ Changed: the recommended path is OpenTelemetry instrumentation (upstream OTel SDKs or the AWS Distro for OpenTelemetry, ADOT) with the CloudWatch agent 1.300025.0+ or an OpenTelemetry Collector with the awsxray exporter replacing the daemon; in Lambda, the AWS Lambda Layer for OpenTelemetry (recommended) or the ADOT managed layer. Older material may say ADOT is only an optional alternative to the X-Ray SDK.
• OpenTelemetry mapping: server spans become segments, other spans subsegments; span attributes become metadata by default, and only attributes whose keys you list in aws.xray.annotations become annotations; X-Ray accepts W3C-format trace IDs (since 2023-10-27; needs CloudWatch agent 1.300030.0+ or ADOT Collector 0.34.0+). OTel adds zero-code auto-instrumentation (Java, Python, Node.js, .NET), many library instrumentations and export to several backends.
• Collectors: the X-Ray daemon listens on UDP 2000 for segments and TCP 2000 as a proxy for the sampling APIs, buffers segments and uploads them in batches with PutTraceSegments (the course's "batch every second" is ⚠️ unverified: current docs only say batches). Options: -n Region, -o local mode (skip the EC2 metadata check, for example ./xray -o -n eu-west-1 on a laptop), -r/--role-arn to upload into another account, -b/-t bind addresses. The CloudWatch agent replaces it (OTLP on 4317 gRPC / 4318 HTTP, X-Ray proxy on 2000); stop the daemon first to avoid port conflicts.
• IAM: whatever sends segments (instance profile, ECS task role, Beanstalk instance profile, Lambda execution role) needs xray:PutTraceSegments and xray:PutTelemetryRecords, plus xray:GetSamplingRules, xray:GetSamplingTargets and xray:GetSamplingStatisticSummaries for centralized sampling: the managed policy AWSXRayDaemonWriteAccess (the older AWSXrayWriteOnlyAccess still exists with the same actions). Viewers use AWSXrayReadOnlyAccess; AWSXrayFullAccess adds configuration; AWSXrayCrossAccountSharingConfiguration manages Observability Access Manager links.
• Encryption: traces are always encrypted at rest; optionally use the AWS managed key aws/xray (to audit key use) or a symmetric customer managed key (PutEncryptionConfig); if X-Ray loses access to the key it stops storing data.
• Service integrations: Lambda (active tracing, daemon run for you: notes/lambda-guide.md#12-observability-with-logs-metrics-and-traces); API Gateway REST stages (active tracing, uses sampling rules; HTTP APIs have none: notes/api-gateway-guide.md#12-logging-metrics-and-tracing); Step Functions and AppSync (active); Application Load Balancer (adds or updates X-Amzn-Trace-Id but sends no segments and isn't a node); SQS, SNS and EventBridge (pass the header on so consumers continue the trace); Elastic Beanstalk (runs the daemon when XRayEnabled is true); ECS, EKS and EC2 (you run the collector).
• Elastic Beanstalk: enable in the console or with .ebextensions option_settings → aws:elasticbeanstalk:xray → XRayEnabled: true; the environment's instance profile needs the write policy; the code still needs instrumentation; not provided on the Multicontainer Docker (ECS-based) platform. Beanstalk itself: notes/beanstalk-guide.md#14-health-logs-and-troubleshooting.
• ECS: run the daemon (or the CloudWatch agent) as a container: as a sidecar in each task (required on Fargate) or as one daemon per container instance on EC2. Map UDP 2000; in bridge mode link the app container and set AWS_XRAY_DAEMON_ADDRESS=xray-daemon:2000; in awsvpc mode containers share localhost, so no link or variable is needed. Permissions go on the task role.
• Cross-account tracing: let the daemon or agent assume a role in a central account (--role-arn), or share traces with CloudWatch cross-account observability (§1).

Developer angle

X-Ray SDK for Python (maintenance mode, still common in existing code): patch AWS SDK and HTTP clients, add a subsegment, an annotation and metadata:

from aws_xray_sdk.core import patch_all, xray_recorder

patch_all()  # instruments boto3, requests and other supported libraries

@xray_recorder.capture("price_order")
def price_order(order):
    xray_recorder.put_annotation("customer_tier", order["tier"])   # indexed: searchable
    xray_recorder.put_metadata("order", order, "checkout")         # not indexed: debug detail
    return sum(item["price"] for item in order["items"])

The same with the OpenTelemetry API (ADOT or upstream SDK configured to export to X-Ray):

from opentelemetry import trace

tracer = trace.get_tracer("checkout")

def price_order(order):
    with tracer.start_as_current_span("price_order") as span:
        span.set_attribute("customer_tier", order["tier"])  # metadata in X-Ray unless listed in aws.xray.annotations
        return sum(item["price"] for item in order["items"])

Turn tracing on where AWS runs the collector, and choose the encryption key:

aws lambda update-function-configuration --function-name checkout --tracing-config Mode=Active
aws apigateway update-stage --rest-api-id a1b2c3d4e5 --stage-name prod \
  --patch-operations op=replace,path=/tracingEnabled,value=true
aws xray put-encryption-config --type KMS --key-id alias/xray-traces

.ebextensions/xray-daemon.config for Elastic Beanstalk:

option_settings:
  aws:elasticbeanstalk:xray:
    XRayEnabled: true

ECS task definition (bridge mode) with the daemon as a sidecar container:

{
  "family": "checkout",
  "taskRoleArn": "arn:aws:iam::111122223333:role/CheckoutTaskRole",
  "containerDefinitions": [
    {"name": "xray-daemon", "image": "public.ecr.aws/xray/aws-xray-daemon:3.x", "essential": true,
     "portMappings": [{"containerPort": 2000, "protocol": "udp"}]},
    {"name": "app", "image": "111122223333.dkr.ecr.eu-west-1.amazonaws.com/checkout:1.4.2", "essential": true,
     "links": ["xray-daemon"],
     "environment": [{"name": "AWS_XRAY_DAEMON_ADDRESS", "value": "xray-daemon:2000"}]}
  ]
}

Traps

• Trap: "Store the order ID as metadata and search for it in the console." → Reality: metadata isn't indexed; only annotations (and the user field) work in filter expressions.
• Trap: "Put the whole order object in an annotation." → Reality: annotation values are strings, numbers or Booleans; objects go in metadata.
• Trap: "The load balancer will show up as a node on the trace map." → Reality: an ALB only adds the trace header; it sends no segments.
• Trap: "Open UDP 2000 in the security group so the app can reach the daemon." → Reality: the daemon runs on the same host or task (localhost or a linked container); it then calls the X-Ray API over HTTPS with the role's credentials.
• Trap: "Beanstalk traces appear once XRayEnabled is set." → Reality: that only runs the daemon; the code must be instrumented and the instance profile needs the write policy.
• Trap: "Start new projects on the X-Ray SDK, it's the AWS standard." → Reality: it's in maintenance mode; use OpenTelemetry (ADOT) and the CloudWatch agent or a Collector.
• Trap: "OpenTelemetry attributes become annotations automatically." → Reality: they become metadata unless listed in aws.xray.annotations.

Practice questions

1. Developers must find all traces for a given customer_id with a filter expression, and they also want the full order payload available in each trace for debugging. How should the application record these values?

   • A. Record both customer_id and the order payload as metadata.
   • B. Record customer_id as an annotation and the order payload as metadata.
   • C. Record both customer_id and the order payload as annotations.
   • D. Use customer_id as the name of a custom subsegment and log the payload to CloudWatch Logs.

2. A Java service on EC2 uses the X-Ray SDK for instrumentation, but no traces appear in X-Ray. The instance profile has only an S3 read policy. Which TWO actions address the most likely causes? (Select TWO.)

   • A. Enable active tracing in the Lambda console for the service.
   • B. Make sure a collector (the X-Ray daemon or the CloudWatch agent with trace collection) runs on the instance and listens on UDP port 2000.
   • C. Add the AWSXRayDaemonWriteAccess managed policy to the instance profile's role.
   • D. Allow inbound UDP 2000 from 0.0.0.0/0 in the instance's security group.
   • E. Create a subscription filter on the application's log group that targets X-Ray.

9. Trace sampling rules APIs and troubleshooting

Concept

• Tracing every request costs money and adds overhead, so X-Ray samples: the service at the start of a trace decides whether to record it, and every downstream service follows that decision (it travels in Sampled=1/0 in the trace header).
• Sampling rules live in the X-Ray service, so you can raise or lower tracing for a route, method or service without redeploying. SDKs, ADOT, API Gateway and other integrated services fetch the rules and report what they sampled.
• The X-Ray APIs let scripts and tools read summaries, full traces and service graphs, while filter expressions and groups slice traces by service, status, latency, user or annotation.

Exam facts

• Default rule: a reservoir of 1 request per second, then a fixed rate of 5 % of additional requests. It applies when no other rule matches; you can change its reservoir and rate but not delete it.
• Custom rule fields: RuleName, Priority (1–9999, lowest evaluated first, first match wins), ReservoirSize (requests per second for the rule, shared across all instances: X-Ray hands each instance a quota), FixedRate (0–1 in the API and in local JSON; a percentage in the console), and match criteria ServiceName, ServiceType (AWS::EC2::Instance, AWS::ECS::Container, AWS::ElasticBeanstalk::Environment, AWS::APIGateway::Stage…), Host, HTTPMethod, URLPath, ResourceARN (SDKs only support *) and Attributes (SDKs ignore rules that use them; API Gateway matches request headers); wildcards * and ?; Version 1.
• Centralized vs local rules: local JSON rules make every instance sample on its own (reservoirs add up across the fleet) and need a redeploy to change; centralized rules are managed by X-Ray. A service that hasn't received its quota yet borrows one trace per second.
• Parent-based: a rule only matters where a trace starts (API Gateway, the first instrumented service, a worker that starts new traces). A strict rule on a service that is always called by another one is never used.
• Lambda sampling is fixed (1 per second + 5 %) and not controlled by sampling rules; API Gateway stages do use them (service name api-name/stage).
• Adaptive sampling (since 2025-09-25): a rule's SamplingRateBoost (MaxRate, CooldownWindowMinutes) lets X-Ray raise sampling temporarily when it detects anomalies such as error spikes.
• Write and sampling APIs (called by SDKs, daemon and agent): PutTraceSegments (segment documents), PutTelemetryRecords (collector health counters such as SegmentsReceivedCount, SegmentsRejectedCount, BackendConnectionErrors), GetSamplingRules, GetSamplingTargets, GetSamplingStatisticSummaries.
• Read APIs: GetTraceSummaries (trace IDs, annotations and timing for a time range, filtered by an expression; use it first), BatchGetTraces (full segment documents for up to 5 trace IDs per call), GetServiceGraph (map for a time range), GetTraceGraph (map of specific traces), GetTimeSeriesServiceStatistics. Management: CreateSamplingRule, UpdateSamplingRule, CreateGroup, PutEncryptionConfig, insights APIs.
• Filter expressions: Boolean keywords ok, error, throttle, fault, partial, inferred, root; numbers responsetime, duration, http.status; strings http.url, http.method, user, name; complex keywords service("checkout") { fault }, edge("api", "orders") { error }, annotation[customer_tier] = "gold" (bracket syntax, required for keys with dots); combine with AND / OR.
• Groups: named filter expressions with their own trace map and CloudWatch metrics (published every minute); editing a group's expression affects only new traces. X-Ray Insights detects anomalies in fault rates and can notify through EventBridge.
• Transaction Search (CloudWatch, since 2024-11-21): stores 100 % of spans as structured logs in the aws/spans log group (CloudWatch Logs features and pricing apply) and indexes a percentage of them as X-Ray trace summaries (1 % free by default); switched on account-wide with UpdateTraceSegmentDestination = CloudWatchLogs. With it enabled, BatchGetTraces can't find traces; use StartTraceRetrieval and ListRetrievedTraces instead.
• Free tier: 100,000 traces recorded and 1,000,000 traces retrieved or scanned per month.
• "No traces" checklist: (1) the code is instrumented and AWS SDK clients are patched; (2) a collector runs and is reachable (AWS_XRAY_DAEMON_ADDRESS, UDP 2000, OTLP 4317/4318, no port clash); (3) the role actually used has the write permissions; (4) the request was simply not sampled, or an upstream service sent Sampled=0; (5) active tracing is on for Lambda functions and API Gateway stages; (6) you're looking in the Region the collector sends to (-n). Missing downstream nodes usually mean the AWS SDK or HTTP client isn't instrumented.

Developer angle

A temporary debugging rule that records every POST /checkout* call, without touching the code (debug-rule.json):

{
  "SamplingRule": {
    "RuleName": "debug-checkout-post",
    "Priority": 1,
    "ReservoirSize": 1,
    "FixedRate": 1.0,
    "ServiceName": "checkout",
    "ServiceType": "*",
    "Host": "*",
    "HTTPMethod": "POST",
    "URLPath": "/checkout*",
    "ResourceARN": "*",
    "Version": 1
  }
}

aws xray create-sampling-rule --cli-input-json file://debug-rule.json
aws xray get-sampling-rules --query 'SamplingRuleRecords[].SamplingRule.[RuleName,Priority,FixedRate,ReservoirSize]'
aws xray delete-sampling-rule --rule-name debug-checkout-post     # when the incident is over

Find failed gold-customer traces from the last hour, then fetch their segments:

aws xray get-trace-summaries --start-time 2026-10-12T08:00:00Z --end-time 2026-10-12T09:00:00Z \
  --filter-expression 'annotation[customer_tier] = "gold" AND fault' \
  --query 'TraceSummaries[].Id' --output text
aws xray batch-get-traces --trace-ids 1-5759e988-bd862e3fe1be46a994272793
aws xray get-service-graph --start-time 2026-10-12T08:00:00Z --end-time 2026-10-12T09:00:00Z
aws xray create-group --group-name gold-faults --filter-expression 'annotation[customer_tier] = "gold" AND fault'

from datetime import datetime, timedelta, timezone

import boto3

xray = boto3.client("xray")
end = datetime.now(timezone.utc)
ids = []
for page in xray.get_paginator("get_trace_summaries").paginate(
        StartTime=end - timedelta(hours=1), EndTime=end,
        FilterExpression='annotation[order_status] = "failed"'):
    ids.extend(summary["Id"] for summary in page["TraceSummaries"])

for start in range(0, len(ids), 5):                     # BatchGetTraces takes 5 IDs per call
    for trace in xray.batch_get_traces(TraceIds=ids[start:start + 5])["Traces"]:
        print(trace["Id"], len(trace["Segments"]), "segments")

• IAM for these scripts: xray:GetTraceSummaries, xray:BatchGetTraces, xray:GetServiceGraph (all in AWSXrayReadOnlyAccess); creating rules and groups needs xray:CreateSamplingRule and xray:CreateGroup, which support resource-level ARNs (arn:aws:xray:<region>:<account>:sampling-rule/<name>).

Traps

• Trap: "Edit the sampling JSON in the app and redeploy to trace more during the incident." → Reality: create a high-priority centralized rule; it takes effect without a deployment.
• Trap: "Raise the default rule's reservoir to trace one route." → Reality: that changes sampling for everything; add a specific rule with a lower priority number.
• Trap: "A strict rule on the downstream inventory service reduces its traces." → Reality: sampling is decided at the trace root; configure the entry service.
• Trap: "BatchGetTraces with a time range returns everything." → Reality: it takes trace IDs (up to 5 per call); get them from GetTraceSummaries first.
• Trap: "Change Lambda's sampling rate with a sampling rule." → Reality: Lambda's sampling is fixed; rules apply to API Gateway, SDK and ADOT entry points.
• Trap: "Use annotation.order.id in a filter." → Reality: use annotation[order.id]; brackets are required when the key contains dots.

Practice questions

1. During an incident, a team wants X-Ray to record every POST /checkout request handled by a fleet of 30 EC2 instances for the next hour, while all other requests keep the default sampling. The instances use the X-Ray SDK with centralized sampling. What should the developer do?

   • A. Change the local sampling rules file on every instance to a fixed rate of 100 % and redeploy the application.
   • B. Raise the default rule's reservoir to 1,000 requests per second.
   • C. Create a sampling rule with priority 1, reservoir 1, fixed rate 100 %, HTTP method POST and URL path /checkout*, and delete it after the incident.
   • D. Enable active tracing for the application in the Lambda console.

2. A developer writes a script that must print the full segment documents of every trace in the last hour that has the annotation order_status = "failed". Which TWO X-Ray API operations should the script call? (Select TWO.)

   • A. GetTraceSummaries with the filter expression annotation[order_status] = "failed".
   • B. PutTraceSegments for the time range of the last hour.
   • C. GetSamplingTargets for the service that records the annotation.
   • D. BatchGetTraces with the trace IDs returned by the first call, in groups of up to five.
   • E. GetServiceGraph for the last hour.

10. CloudTrail auditing and choosing the right tool

Concept

• AWS CloudTrail records API activity in your account (console, CLI, SDK and calls that AWS services make for you) as events: who called (userIdentity), what (eventSource, eventName, requestParameters), when (eventTime), from where (sourceIPAddress, userAgent, awsRegion) and with what result (responseElements, errorCode, errorMessage).
• Event history is always on and free: the last 90 days of management events in each Region. A trail delivers events continuously to S3 (optionally also CloudWatch Logs and EventBridge) for long retention, analysis and alerting; an organization trail covers every account.
• Event types: management (control plane, logged by default), data (data plane, high volume, opt-in and billed), network activity (calls through VPC endpoints) and Insights (unusual call or error rates).
• CloudTrail answers "who did what"; CloudWatch answers "how is it behaving"; X-Ray answers "where did this request spend its time or fail".

Exam facts

• Event history: management events only (no data, Insights or network activity events), 90 days, per Region, immutable; one lookup attribute plus a time range per search (EventId, EventName, EventSource, ReadOnly, ResourceName, ResourceType, Username, AccessKeyId); LookupEvents is limited to 2 requests per second per account and Region and 50 results per page; downloads up to 200,000 events per file.
• ⚠️ Changed: trails created in the console are always multi-Region (and start logging automatically); single-Region trails can be created only with the CLI or API, where single-Region is the default (--is-multi-region-trail changes it) and logging starts only after start-logging. Older material may say the console lets you apply a trail to all Regions or to a single Region.
• Trails: 5 per Region (fixed); one copy of management events to S3 is free (S3 storage is not); the bucket policy must allow cloudtrail.amazonaws.com; log and digest files are encrypted with SSE-KMS by default in the console, with SSE-S3 when the CLI or API omits --kms-key-id; gzip files up to 50 MB before compression; optional SNS notification per delivered file. Events typically arrive about 5 minutes after the API call (not guaranteed). Read and write management events are both logged by default; KMS and RDS Data API events can be excluded from a trail.
• Log file integrity validation (--enable-log-file-validation): SHA-256 hashes and SHA-256-with-RSA-signed digest files delivered every hour; check with aws cloudtrail validate-logs to prove files weren't modified or deleted.
• ⚠️ Changed: data events cover many resource types through advanced event selectors (S3 objects, Lambda Invoke, DynamoDB items, SNS Publish, SQS, Kinesis Data Streams, CloudWatch PutMetricData and many more), with filters on eventName, readOnly, resources.ARN and identity; basic selectors cover only S3 objects, Lambda functions and DynamoDB tables. Older material may say only S3 object-level activity and Lambda Invoke.
• ⚠️ Changed: Insights has two types, API call rate (needs write management events) and API error rate (read or write management events), built from a baseline of normal activity; since 2025-11-20 the same two types also exist for data events (trails only). Insights events go to the trail's S3 bucket, the console, CloudWatch Logs and EventBridge (AWS Insight via CloudTrail), and cost extra. Older material may say Insights only analyzes write management events.
• ⚠️ Changed: CloudTrail Lake (SQL over event data stores, retention up to 10 years) is closed to new customers since 2026-05-31; existing customers continue, and AWS points to CloudWatch for similar capabilities (CloudTrail events can be ingested into CloudWatch). For long retention plus SQL, the classic answer is a trail to S3 queried with Athena. Older material may say CloudTrail Lake is the recommended way to run SQL queries over events.
• To CloudWatch Logs: update-trail --cloud-watch-logs-log-group-arn … --cloud-watch-logs-role-arn … with a role that cloudtrail.amazonaws.com assumes (logs:CreateLogStream, logs:PutLogEvents; the console proposes CloudTrail_CloudWatchLogs_Role); then metric filters and alarms on specific calls (root activity, AccessDenied, security group changes). CloudTrail events are capped at 256 KB (event version 1.05+); larger ones aren't sent to CloudWatch Logs or EventBridge.
• To EventBridge: API calls arrive on the default bus as AWS API Call via CloudTrail (also AWS Console Signin via CloudTrail, AWS Insight via CloudTrail); per the EventBridge docs a trail with logging must be enabled; data events need a trail that logs them; forward to a Classic custom bus with a rule. Write management events match normally. ⚠️ Changed: read-only management events (Get*, List*, Describe*) are matched only by rules whose state is ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS (since 2023-11-17, set with the CLI or CloudFormation); when you forward them to another bus, both the forwarding rule and the destination rule need that state. Older material may say EventBridge never receives read-only API calls.
• First move when a resource vanished or changed: search Event history (lookup-events by EventName or ResourceName), then read userIdentity (type IAMUser, AssumedRole, Root, AWSService; arn; accessKeyId; sessionContext) and sourceIPAddress.
• Which tool answers which question:

Developer angle

Who deleted the table yesterday (no trail needed for the last 90 days)?

aws cloudtrail lookup-events --start-time 2026-10-11T00:00:00Z --max-results 50 \
  --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteTable \
  --query 'Events[].[EventTime,Username,CloudTrailEvent]'

A validated multi-Region trail with selected data events, Insights and CloudWatch Logs delivery:

aws cloudtrail create-trail --name audit --s3-bucket-name audit-logs-111122223333 \
  --is-multi-region-trail --enable-log-file-validation --kms-key-id alias/cloudtrail
aws cloudtrail start-logging --name audit
aws cloudtrail put-event-selectors --trail-name audit --advanced-event-selectors file://selectors.json
aws cloudtrail put-insight-selectors --trail-name audit \
  --insight-selectors '[{"InsightType":"ApiCallRateInsight"},{"InsightType":"ApiErrorRateInsight"}]'
aws cloudtrail update-trail --name audit \
  --cloud-watch-logs-log-group-arn arn:aws:logs:eu-west-1:111122223333:log-group:cloudtrail/audit:* \
  --cloud-watch-logs-role-arn arn:aws:iam::111122223333:role/CloudTrail_CloudWatchLogs_Role
aws cloudtrail validate-logs --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/audit \
  --start-time 2026-10-01T00:00:00Z

selectors.json (all management events, plus writes to one bucket's objects):

[
  {"Name": "Management events", "FieldSelectors": [{"Field": "eventCategory", "Equals": ["Management"]}]},
  {"Name": "Object writes in orders-uploads", "FieldSelectors": [
    {"Field": "eventCategory", "Equals": ["Data"]},
    {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
    {"Field": "readOnly", "Equals": ["false"]},
    {"Field": "resources.ARN", "StartsWith": ["arn:aws:s3:::orders-uploads/"]}]}
]

React to one dangerous call within minutes (EventBridge rule on the default bus, SNS target):

{
  "source": ["aws.dynamodb"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {"eventSource": ["dynamodb.amazonaws.com"], "eventName": ["DeleteTable"]}
}

Alarm on unauthorized calls through the trail's log group:

aws logs put-metric-filter --log-group-name cloudtrail/audit --filter-name unauthorized-calls \
  --filter-pattern '{ ($.errorCode = "*UnauthorizedOperation") || ($.errorCode = "AccessDenied*") }' \
  --metric-transformations metricName=UnauthorizedCalls,metricNamespace=Audit,metricValue=1,defaultValue=0

Traps

• Trap: "Create a trail now to find out who deleted the bucket last week." → Reality: a new trail only captures future events; Event history already holds 90 days of management events.
• Trap: "Event history keeps everything for 90 days, including S3 GetObject." → Reality: it holds management events only; data events need a trail or event data store configured for them.
• Trap: "Keep 7 years of audit logs in Event history." → Reality: 90 days; deliver a trail to S3 with lifecycle rules (or Object Lock) for long retention.
• Trap: "create-trail from the CLI starts recording all Regions." → Reality: the CLI default is single-Region and logging begins only after start-logging.
• Trap: "Insights alerts on a specific API call like DeleteBucket." → Reality: Insights flags unusual rates of calls or errors; for a specific call use an EventBridge rule or a metric filter.
• Trap: "Use X-Ray to see which IAM user changed the security group." → Reality: X-Ray traces requests in your application; API audit is CloudTrail.
• Trap: "A default-state EventBridge rule catches GetSecretValue calls." → Reality: read-only management events need the ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS rule state.

Practice questions

1. An Amazon DynamoDB table was deleted yesterday. The account has no CloudTrail trail. The team must identify the IAM principal that deleted it, with the LEAST effort. What should the developer do?

   • A. Search CloudTrail Event history (aws cloudtrail lookup-events) for the DeleteTable event name and read its userIdentity.
   • B. Open the X-Ray trace map for the application that used the table.
   • C. Run a CloudWatch Logs Insights query against the table's log group.
   • D. Create a trail now and query its S3 objects with Amazon Athena.

2. A security team wants an email within minutes whenever any IAM role's trust policy is changed (UpdateAssumeRolePolicy), and must keep the full record of such API calls for 7 years. Which TWO actions meet these requirements? (Select TWO.)

   • A. Rely on CloudTrail Event history, which can retain events for up to 7 years.
   • B. Enable CloudTrail Insights for API call rate on the account.
   • C. Create an EventBridge rule on the default bus that matches AWS API Call via CloudTrail events with eventSource iam.amazonaws.com and eventName UpdateAssumeRolePolicy, with an SNS topic as target.
   • D. Create an X-Ray group with a filter expression for IAM calls.
   • E. Create a multi-Region trail that delivers to an S3 bucket with a lifecycle policy (or Object Lock) that keeps objects for 7 years.

Quick decision rules

Metrics and metrics from logs (§1–§2)

• If you see "custom metric with p99 across instances", think raw values with Values/Counts and one shared dimension set, not statistic sets.
• If you see "sub-minute alarm" or "react within 30 seconds", think StorageResolution 1 plus a 10, 20 or 30-second alarm period.
• If you see "metric data older than two weeks", think it's rejected by PutMetricData; keep history in logs or S3.
• If you see "metrics from Lambda without extra API calls and with many dimensions", think embedded metric format printed to stdout.
• If you see "count errors in existing plain-text logs without code changes", think metric filter (≤ 3 dimensions, not retroactive) plus an alarm.
• If you see "per-request or per-user dimension", think cardinality explosion: keep IDs in logs or traces.
• If you see "one account must view metrics, logs and traces of many accounts", think CloudWatch cross-account observability (monitoring account + source account links).

Logs (§3–§4)

• If you see "logs never expire / storage cost grows", think put-retention-policy with an allowed value (1 day to 10 years).
• If you see "cheaper ingestion for rarely read logs, no alarms needed", think Infrequent Access log class, chosen at creation.
• If you see "logs stopped arriving from EC2", think instance role (CloudWatchAgentServerPolicy), unified agent (not the deprecated Logs agent), network path.
• If you see "mask credit card numbers or PII in logs", think a data protection policy; reading unmasked data needs logs:Unmask.
• If you see "ad hoc search, aggregate, top-N over stored logs", think Logs Insights (start-query + get-query-results, 60-minute timeout).
• If you see "near-real-time log delivery to Kinesis, Firehose or Lambda", think subscription filter (up to 5 per log group).
• If you see "logs from many accounts into one Kinesis stream", think destination + destination access policy in the recipient account, subscription filters in the senders.
• If you see "archive logs to S3 once" or "batch, hours of delay are fine", think CreateExportTask (one active task per account); for continuous archiving think Firehose.

Alarms and canaries (§5)

• If you see "page only when several symptoms coincide" or "suppress during deployments", think composite alarm (AND/OR/NOT, actions suppressor).
• If you see "run code when an alarm fires", think Lambda alarm action plus a resource policy for lambda.alarms.cloudwatch.amazonaws.com (or an EventBridge rule on CloudWatch Alarm State Change).
• If you see "alarm stuck in INSUFFICIENT_DATA", think mismatched dimensions or unit, period shorter than the data resolution, or TreatMissingData.
• If you see "alert when the heartbeat metric stops", think TreatMissingData = breaching.
• If you see "test the notification path end to end", think set-alarm-state (it really runs the actions).
• If you see "check an API or page every few minutes with no real traffic", think CloudWatch Synthetics canary plus an alarm on SuccessPercent.

EventBridge (§6–§7)

• If you see "cron in a local time zone", "one-time future invocation" or "millions of schedules", think EventBridge Scheduler, not scheduled rules.
• If you see "PutEvents succeeded but some events are missing", think FailedEntryCount and resend only the failed entries.
• If you see "rule matches but the target is never invoked", think target permissions (resource policy or execution role) and a standard SQS DLQ to read ERROR_CODE.
• If you see "reprocess yesterday's events after a bug fix", think archive and replay to the source bus.
• If you see "point-to-point from SQS, Kinesis or DynamoDB Streams with filtering and enrichment", think EventBridge Pipes.
• If you see "events from every account of an organization in one bus", think bus resource policy with aws:PrincipalOrgID and sender rules with an IAM role.
• If you see "typed event classes without hand-written schemas", think schema discovery plus code bindings (Java, Python, TypeScript, Go).

Tracing (§8–§9)

• If you see "search traces by a business value", think annotation; "store a debug payload in the trace", think metadata.
• If you see "no traces from EC2 or containers", think collector on UDP 2000 (daemon or CloudWatch agent), AWSXRayDaemonWriteAccess on the role, instrumented clients, sampling.
• If you see "Elastic Beanstalk tracing", think XRayEnabled: true in aws:elasticbeanstalk:xray plus instance profile permissions plus instrumented code.
• If you see "ECS or Fargate tracing", think a daemon or agent sidecar container on UDP 2000 and the task role.
• If you see "new tracing work" or "send traces to several backends", think OpenTelemetry (ADOT) with the CloudWatch agent or an OTel Collector.
• If you see "trace more of one route right now without redeploying", think a centralized sampling rule with a low priority number at the entry service.
• If you see "fetch full traces by condition from a script", think GetTraceSummaries (filter) then BatchGetTraces (5 IDs per call).

Audit and tool choice (§10)

• If you see "who deleted or changed a resource in the last 90 days", think CloudTrail Event history (lookup-events).
• If you see "keep API activity for years" or "query it with SQL", think a trail to S3 with lifecycle rules, queried with Athena.
• If you see "S3 object-level or Lambda invoke auditing", think data events with advanced event selectors (extra cost).
• If you see "react within minutes to a specific API call", think EventBridge rule on AWS API Call via CloudTrail; for read-only calls the rule state ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS.
• If you see "unusual burst of API calls or errors", think CloudTrail Insights.
• If you see "prove log files weren't tampered with", think log file integrity validation and validate-logs.

Changed vs older material

Learn the Current column; older practice material may still show the Course / older material value. Rows are proposed for notes/current-facts.md after confirmation (some already exist there under the ec2 or sqs-sns-kinesis modules).