EC2 — DVA-C02 Mastery Guide

• Sources: course chapter 02 (EC2), plus the IMDS and credential-chain slides of chapter 07 (CLI, SDK and policies) and the EC2 monitoring slides of chapter 13 (Monitoring). The course serves as the syllabus only; all wording here is original.
• Verified against AWS documentation on 2026-10-12. The marker ⚠️ Changed: <current>. Older material may say <old>. flags a value or behaviour that differs from the course or older practice tests: learn the current value. Every change is listed in Changed vs older material and in notes/current-facts.md.
• Tier 3 (supporting), bank target 40 questions. Domains: Troubleshooting and Optimization (primary), Development with AWS Services, Security, Deployment.
• How to use it: /explain <topic> teaches from these sections (concept → exam facts → developer angle → traps → 3 check questions); /quiz ec2 asks bank questions whose source_note links point at the ## N. headings below; /drill ec2 fires short recall prompts from notes/ec2-cheat-sheet.md. Suggested loop: read a section and answer its practice questions before opening the answers → /explain what is still unclear → /quiz ec2 → /drill ec2.

How EC2 appears on DVA-C02

DVA-C02 does not test EC2 as a system administration topic. It treats an instance as the place where your code runs and asks what a developer must get right around it:

• Credentials and security: how code on an instance gets AWS permissions (instance profile, credential chain, IMDSv2) and how keys and secrets stay off the instance.
• Bootstrapping and deployment: user data, golden AMIs, launch templates, and rolling a new version through an Auto Scaling group.
• Troubleshooting: reading the symptom (timeout, connection refused, AccessDenied, ALB 502/503/504, failed status or health checks) to find the layer at fault.
• Choosing under a constraint: the purchasing option, volume type, storage service, load balancer or scaling policy that meets "lowest cost", "most secure", "least operational overhead" or a stated IOPS or latency target.

Typical question shapes: a scenario with one deciding constraint; an error message or metric to diagnose; "Which TWO actions…" with five options; the exact API action, CLI command, header or setting that does the job.

Domain map: Troubleshooting and Optimization → §4, §5 (cost), §8, §10, §11 · Security → §1, §2, §6 (encryption), §9 (TLS) · Development with AWS Services → §1, §3, §7 · Deployment → §3, §6 (AMIs), §10.

Out of scope: OS administration depth (kernel, package, file-system or Windows administration beyond the commands shown here) and network infrastructure design (subnet, routing, NAT, peering or Transit Gateway architecture). NACLs, route tables and VPC endpoints appear only as blockers you must recognise.

1. IAM roles, instance profiles and the credential chain

Concept

• Every AWS API call from code on an instance must be signed. The secure pattern is an IAM role delivered through an instance profile: EC2 hands the instance short-lived role credentials, so no secret is stored on disk, in the AMI, in user data or in the repository.
• The instance profile is the object EC2 actually attaches; it wraps exactly one role. The role carries a trust policy (who may assume it: ec2.amazonaws.com) and permission policies (what the code may do, ideally scoped to specific resources).
• SDKs and the CLI locate credentials by walking a credential provider chain and stop at the first hit. The instance profile is the last stop, so a stray environment variable or credentials file on the box silently takes over.
• Payoff for a developer: no key distribution or rotation, automatic refresh, least privilege per workload, and a one-command diagnosis (aws sts get-caller-identity).

Exam facts

• Role vs profile: creating a role for EC2 in the IAM console (or "Create new IAM role" in the launch wizard) auto-creates an instance profile with the same name. With the CLI/API/SDK they are separate steps (aws iam create-instance-profile, aws iam add-role-to-instance-profile) and you launch with the instance profile name or ARN, which may differ from the role name.
• Limits: 1 role per instance profile (hard limit, cannot be raised); 1 role per instance at a time, shared by every process on it. A role can sit in many profiles and serve many instances.
• Live changes, no restart: associate-iam-instance-profile attaches a profile to a running or stopped instance that has none. If one is already attached, use replace-iam-instance-profile-association (instance must be running; needs the iip-assoc-… ID from describe-iam-instance-profile-associations). disassociate-iam-instance-profile detaches (running or stopped).
• Swapping the role inside a profile (remove-role-from-instance-profile + add-role-to-instance-profile) can take up to 1 hour to reach the instance; AWS recommends replacing the instance profile association instead (or disassociate/associate, or stop/start, to force it).
• Delivery: IMDS item /latest/meta-data/iam/security-credentials/<role-name> returns AccessKeyId (temporary, ASIA…), SecretAccessKey, Token, Expiration. AWS rotates them and publishes a new set at least 5 minutes before expiry; SDKs/CLI refresh automatically. If you cache them yourself, refresh hourly or at least 15 minutes before Expiration. Retrieval details and IMDSv2: see §2.
• EC2 role credentials are not bound by the role's MaxSessionDuration. The session principal is arn:aws:sts::<account>:assumed-role/<role-name>/<instance-id> (aws:userid = <role-id>:<instance-id>).
• /latest/meta-data/iam/info shows the instance profile ARN/ID; IMDS reveals the role name but never the policy documents.
• Don't call sts:AssumeRole on the role the instance already has; it only mints identical credentials. If a role must assume itself, its trust policy must explicitly allow that, otherwise AccessDenied.
• Never place access keys in code, AMIs, user data, .env files or ~/.aws/credentials on instances. ⚠️ Changed: outside AWS, AWS now also recommends temporary credentials: IAM Identity Center (aws configure sso), aws login with console credentials (AWS CLI 2.32.0 or later; 15-minute credentials auto-refreshed for up to 12 h), or IAM Roles Anywhere for servers; long-term keys only where roles are impossible. Older material may say "use environment variables or named profiles with access keys" when working outside AWS.
• AWS CLI v2 precedence (first match wins): 1 command-line options (--profile, --region) → 2 environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) → 3 assume-role profile → 4 assume role with web identity → 5 IAM Identity Center → 6 credentials file ~/.aws/credentials → 7 custom credential_process → 8 config file ~/.aws/config → 9 container credentials (ECS task role) → 10 EC2 instance profile via IMDS. ⚠️ Changed: 10 documented sources. Older material may list only 6 (options, environment, credentials file, config file, container, instance profile).
• SDK for Java 2.x (DefaultCredentialsProvider): Java system properties aws.accessKeyId / aws.secretAccessKey / aws.sessionToken → environment variables → web identity token (AWS_WEB_IDENTITY_TOKEN_FILE + AWS_ROLE_ARN) → shared credentials/config files → ECS container credentials → EC2 instance profile. ⚠️ Changed: the secret property is aws.secretAccessKey and the chain includes session token and web identity. Older material (SDK for Java 1.x, end of support 2025-12-31) may show aws.secretKey and a 5-step chain.
• boto3 follows the same shape: explicit parameters → environment → assume-role / web identity / Identity Center → shared files → container → IMDS last. Rule for every SDK: explicit > environment > profiles/files > container > instance profile.
• AWS_PROFILE only selects a profile; access keys in environment variables still override that profile's keys. An explicit --profile on the command line overrides environment variables.
• Precedence trap: keys left in environment variables or a credentials file on the instance beat the role → calls run as an IAM user → over-broad access or AccessDenied naming the wrong principal. Fix by removing them (or by selecting the IMDS provider explicitly in code).
• aws sts get-caller-identity needs no permission (works even under an explicit deny) and returns Account, Arn, UserId: the first command to run when "the role doesn't work".
• aws sts decode-authorization-message turns the encoded message some services (e.g., EC2 UnauthorizedOperation) return into the principal, action, resource, condition values and whether an explicit deny matched; the caller needs sts:DecodeAuthorizationMessage.
• iam:PassRole is a permission (not an API, so no CloudTrail event of its own) checked whenever a role is handed to a service. Launching with or attaching a role needs ec2:RunInstances / ec2:AssociateIamInstanceProfile / ec2:ReplaceIamInstanceProfileAssociation plus iam:PassRole on the role ARN (not the instance profile ARN); the console also needs iam:ListInstanceProfiles. Narrow it with iam:PassedToService = ec2.amazonaws.com. A launch without iam:PassRole fails with EC2's UnauthorizedOperation ("You are not authorized to perform this operation"; the decoded message names the missing permission). An instance profile with no role fails the launch too; the IAM troubleshooting guide files both cases under an AccessDenied error.
• IAM is eventually consistent: new roles and policy edits take time to apply everywhere. Keep IAM changes out of critical code paths and verify propagation before relying on them. Edits to a role's policies propagate to all instances using it.
• Least privilege: scope actions and resource ARNs (one bucket prefix, one table, one secret), add conditions, and use IAM Access Analyzer to generate a policy from the role's CloudTrail activity.

Developer angle

Create the role and profile, then launch or attach:

aws iam create-role --role-name OrdersAppRole --assume-role-policy-document file://trust.json
aws iam put-role-policy --role-name OrdersAppRole --policy-name orders-s3 --policy-document file://perms.json
aws iam create-instance-profile --instance-profile-name OrdersAppProfile
aws iam add-role-to-instance-profile --instance-profile-name OrdersAppProfile --role-name OrdersAppRole

# at launch: pass the PROFILE, not the role
aws ec2 run-instances --image-id ami-0abcdef1234567890 --instance-type t3.micro \
  --iam-instance-profile Name=OrdersAppProfile

# on a running instance (no reboot): attach if none, otherwise replace
aws ec2 associate-iam-instance-profile --instance-id i-0123456789abcdef0 \
  --iam-instance-profile Name=OrdersAppProfile
aws ec2 describe-iam-instance-profile-associations \
  --filters Name=instance-id,Values=i-0123456789abcdef0
aws ec2 replace-iam-instance-profile-association \
  --association-id iip-assoc-0044d817db6c0a4ba --iam-instance-profile Name=OrdersAppProfileV2

trust.json (EC2 may assume the role) and perms.json (resource-scoped permissions):

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "ec2.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }]
}

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:PutObject"],
    "Resource": "arn:aws:s3:::orders-bucket/incoming/*"
  }]
}

Policy for the person or pipeline that launches instances with this role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ec2:RunInstances", "ec2:AssociateIamInstanceProfile",
                 "ec2:ReplaceIamInstanceProfileAssociation",
                 "ec2:DescribeIamInstanceProfileAssociations"],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::111122223333:role/OrdersAppRole",
      "Condition": { "StringEquals": { "iam:PassedToService": "ec2.amazonaws.com" } }
    }
  ]
}

Diagnose "wrong identity / AccessDenied" on the instance:

aws sts get-caller-identity             # expect ...:assumed-role/OrdersAppRole/i-0123456789abcdef0
env | grep '^AWS_'                      # stray AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_PROFILE?
ls -la ~/.aws /root/.aws 2>/dev/null    # stray credentials or config files?
aws configure list                      # shows the source (type/location) of each value in use
aws sts decode-authorization-message --encoded-message "$ENCODED_MSG" \
  --query DecodedMessage --output text

• In code, build clients without keys (boto3.client("s3"), S3Client.create() in Java 2.x); the default chain finds the role and refreshes it.
• On ECS on EC2, give each task a task role: SDKs in the task resolve it from the container provider (169.254.170.2) before IMDS. Containers on EC2 hosts can still reach the instance role through IMDS, so for least privilege keep the instance role minimal and block task access to IMDS (how: see §2).

Traps

• Trap: "The instance has a role, so the app uses it." → Reality: environment variables, ~/.aws/credentials or a credential_process profile on the box are found first; the instance profile is the last provider in every default chain.
• Trap: "Add a second role to the instance profile." → Reality: a profile holds exactly one role and an instance one profile; merge the least-privilege permissions into one role, or use ECS task roles per container.
• Trap: "Changing the role needs a stop/start." → Reality: associate-iam-instance-profile (none attached) or replace-iam-instance-profile-association (one attached, running) applies live; SDKs pick up the new credentials on their next refresh.
• Trap: "Update permissions by swapping the role inside the profile." → Reality: that can take up to 1 hour to reach the instance; replace the association instead.
• Trap: "Grant iam:PassRole on the instance profile ARN." → Reality: PassRole is evaluated against the role ARN.
• Trap: "get-caller-identity fails when the user has no STS permissions." → Reality: it needs no permission and cannot be blocked by an explicit deny.
• Trap: "Any developer can decode an authorization failure message." → Reality: decoding requires sts:DecodeAuthorizationMessage, because the details can be privileged.
• Trap: "Keys in user data or the AMI are fine if the EBS volume is encrypted." → Reality: they are still long-term secrets readable on the instance (user data through IMDS); use a role.
• Trap: "A 1-hour MaxSessionDuration on the role expires the instance's credentials." → Reality: EC2 role credentials ignore that setting and are rotated automatically.
• Trap: "Use identity-credentials/ec2/security-credentials/ec2-instance for application calls." → Reality: that is the instance identity role; it has no permissions beyond identifying the instance to integrated AWS features (and sts:GetCallerIdentity).
• Trap: "A policy edit takes effect instantly everywhere." → Reality: IAM is eventually consistent; retry with backoff and verify before depending on it.

Practice questions

1. A developer deploys an inventory service to an EC2 instance launched from a shared team AMI. The instance profile's role allows only dynamodb:Query on one table, yet the service's calls fail with AccessDenied errors that name the IAM user ci-deployer as the caller. What is the MOST likely cause?

   • A. The role's trust policy does not allow ec2.amazonaws.com to assume it.
   • B. A credentials file left in the AMI is found before the instance profile in the chain.
   • C. IMDSv2 is required on the instance and the SDK cannot obtain a session token.
   • D. The role's maximum session duration expired the instance credentials after one hour.

2. A developer must give a running production instance the permissions of a new role, ReportsRoleV2, without stopping it. The instance already has the instance profile ReportsProfile (containing ReportsRole) attached, and a profile ReportsProfileV2 containing the new role exists. Which TWO actions are required? (Select TWO.)

   • A. Call replace-iam-instance-profile-association with the current association ID and ReportsProfileV2.
   • B. Stop the instance, call associate-iam-instance-profile, and then start it again.
   • C. Make sure the caller has iam:PassRole permission on the ReportsRoleV2 role ARN.
   • D. Add ReportsRoleV2 to ReportsProfile so that the profile holds both roles.
   • E. Reboot the instance so the SDK reloads credentials from the metadata service.

2. Instance metadata service (IMDSv2)

Concept

• IMDS is a link-local HTTP endpoint reachable only from inside the instance (169.254.169.254, optional IPv6 [fd00:ec2::254]). Code uses it to discover its own instance ID, Region/AZ, network details, tags (opt-in), user data, a signed identity document and, most importantly, the instance role's temporary credentials (§1).
• Because credentials live there, IMDS is a classic target of SSRF bugs, open reverse proxies and misconfigured firewalls. IMDSv2 makes access session-based: obtain a token with an HTTP PUT, then present it on every GET.
• Each control blocks one attack path: most SSRF/WAF/proxy paths can't send a PUT with custom headers; PUTs carrying X-Forwarded-For are refused; the token response has an IP TTL (hop limit) of 1 by default, so it cannot leave the instance; a token only works on the instance that issued it.
• Developers rarely call IMDS by hand (current SDKs and the CLI use IMDSv2 automatically), but you must know how to enforce it, find leftover IMDSv1 callers, and fix containers that cannot get a token.

Exam facts

• Paths: /latest/meta-data/… (e.g., instance-id, placement/region, iam/info, iam/security-credentials/<role>, tags/instance when enabled), /latest/user-data (404 if none), /latest/dynamic/instance-identity/document (JSON with account ID, Region, instance ID; signature/pkcs7 alongside). Use latest, not dated versions. IMDS requests are not billed.
• Token flow: PUT /latest/api/token with X-aws-ec2-metadata-token-ttl-seconds: <1–21600> (1 s to 6 h) → token; then GET with X-aws-ec2-metadata-token: <token>. Reuse a token until it expires; there is no practical limit on concurrent tokens. Only GET/HEAD are allowed for metadata reads; a PUT to a dated path (e.g., /2021-03-23/api/token) returns 403.
• HttpTokens: optional (v1 and v2 both work) or required (v2 only). With required, a request with no valid or an expired token gets 401 Unauthorized. Other codes: 400 invalid PUT, 403 request not allowed or IMDS turned off, 404 no such item, 503 retry.
• HttpPutResponseHopLimit: 1–64. It is the IP TTL of the token (PUT) response, not of every response. Default 1, or 2 when the AMI has ImdsSupport: v2.0, unless an account default overrides. A container on bridge networking adds a hop, so with 1 the token never arrives; AWS recommends 2 on container hosts when IMDSv2 is required.
• HttpEndpoint=disabled turns IMDS off: no role credentials for SDKs. If disabled at launch, user data is not available and does not run, and the launch key's public key is not fetched. It can be re-enabled at any time.
• IPv6 endpoint: Nitro instances in IPv6-capable subnets only, off by default (HttpProtocolIpv6=enabled). Tags in metadata: off by default (InstanceMetadataTags=enabled).
• Metadata options can be changed on a running or stopped instance; the change applies immediately, no restart.
• Defaults: ⚠️ Changed: IMDSv2-only is now the default on many launch paths. Amazon Linux 2023 AMIs carry ImdsSupport: v2.0 (token required, hop limit 2), console Quick Start launches default to "V2 only (token required)", accounts can set per-Region IMDS defaults (March 2024) and can enforce IMDSv2 so that v1-enabled launches fail (HttpTokensEnforced, February 2026). An instance whose launch settings, account default and AMI are all unset still gets optional; current docs define no instance-type default. Older material may say IMDSv1 is the default and both versions are always available, or (AWS's November 2023 roadmap) that newly released instance types are IMDSv2-only by default; check the instance's MetadataOptions rather than assuming.
• Precedence at launch (evaluated per option): launch parameters or launch template → account-level default (per Region) → AMI ImdsSupport (only HttpTokens and hop limit). Account defaults never change existing instances.
• Enforcement (HttpTokensEnforced=enabled, checked after precedence): a launch fails unless the launch parameters, account default or AMI resolve HttpTokens to required. On existing instances, modify-instance-metadata-options --http-tokens optional fails with UnsupportedOperation; instances that already allow IMDSv1 are left untouched (require v2 on them yourself). Protect the setting by restricting ec2:ModifyInstanceMetadataDefaults to an admin role.
• ImdsSupport: v2.0 on an AMI (register-image / modify-image-attribute --imds-support v2.0) cannot be undone; to "reset", create a new AMI from the snapshot.
• Find IMDSv1 users first: CloudWatch AWS/EC2 metric MetadataNoToken counts successful IMDSv1 calls while v1 is allowed; MetadataNoTokenRejected counts v1 calls refused after v2 is required. Per instance, only one of them emits. The open-source IMDS Packet Analyzer shows which process makes v1 calls. Recommended path: audit → update software → require on instances → set the account default → enforce.
• IAM/SCP condition keys: ec2:MetadataHttpTokens, ec2:MetadataHttpPutResponseHopLimit, ec2:MetadataHttpEndpoint constrain RunInstances and ModifyInstanceMetadataOptions (a mismatch returns UnauthorizedOperation); ec2:Attribute/HttpTokens blocks opting back out on modify. Scope these denies to arn:aws:ec2:*:*:instance/* as AWS's examples do: a negated condition (StringNotEquals) on Resource: "*" also matches the image, subnet and other launch resources, where the key is absent, and denies every launch. ec2:RoleDelivery (1.0 or 2.0) is present on requests signed with EC2 role credentials; a deny on NumericLessThan "2.0" makes credentials fetched over IMDSv1 useless.
• Auto Scaling: set MetadataOptions in a new launch template version; running instances keep their settings (modify them, or replace them with instance refresh, §10).
• SDKs: IMDSv2 is used automatically from AWS CLI 1.16.289, boto3 1.12.6 (botocore 1.13.25), Java 2.x 2.10.21, Java 1.x 1.11.678, JavaScript v2 2.722.0, Go 1.25.38, .NET 3.3.634.1, PHP 3.147.7, Ruby 3.79.0; current SDKs all do. SDKs try v2 first. If the token call gets no response (the container hop-limit symptom), some SDKs retry and then fall back to v1, which adds delay; an HTTP 403/404/405 on the token call also makes SDKs fall back. With AWS_EC2_METADATA_V1_DISABLED=true (ec2_metadata_v1_disabled), and always in the Kotlin and Rust SDKs, there is no fallback: the credential lookup fails once the token call times out. The CLI waits 1 s for IMDS by default (AWS_METADATA_SERVICE_TIMEOUT).
• Throttling: IMDS queries are throttled per instance (no published rate, plus a cap on simultaneous connections). Separately, all link-local traffic (IMDS, Route 53 Resolver DNS, NTP, Windows licensing) shares a 1,024 packets/s per network interface limit; drops show in the ENA metric linklocal_allowance_exceeded. Cache credentials (SDKs do) and retry with exponential backoff.

Developer angle

Token flow with curl (-f makes curl fail on HTTP errors instead of storing the error text in TOKEN):

TOKEN=$(curl -sf -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -sf -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id
curl -sf -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/region
curl -sf -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/
curl -sf -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/dynamic/instance-identity/document

# IMDSv1-style call (no token): prints 401 when HttpTokens=required
curl -s -o /dev/null -w "%{http_code}\n" http://169.254.169.254/latest/meta-data/instance-id

Audit, then enforce:

# instances that still allow IMDSv1
aws ec2 describe-instances --filters "Name=metadata-options.http-tokens,Values=optional" \
  --query "Reservations[*].Instances[*].InstanceId" --output text

# IMDSv1 calls over the last week (want Sum = 0 before switching)
aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name MetadataNoToken \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --start-time 2026-10-05T00:00:00Z --end-time 2026-10-12T00:00:00Z \
  --period 86400 --statistics Sum

# running instance: require v2 now (no restart); hop limit 2 for containers
aws ec2 modify-instance-metadata-options --instance-id i-0123456789abcdef0 \
  --http-tokens required --http-endpoint enabled --http-put-response-hop-limit 2
aws ec2 describe-instances --instance-ids i-0123456789abcdef0 \
  --query "Reservations[].Instances[].MetadataOptions"

# new instance
aws ec2 run-instances --image-id ami-0abcdef1234567890 --instance-type t3.micro \
  --metadata-options "HttpEndpoint=enabled,HttpTokens=required,HttpPutResponseHopLimit=2"

# account default for new launches in one Region, then hard enforcement
aws ec2 modify-instance-metadata-defaults --region eu-west-1 \
  --http-tokens required --http-put-response-hop-limit 2
aws ec2 modify-instance-metadata-defaults --region eu-west-1 --http-tokens-enforced enabled
aws ec2 get-instance-metadata-defaults --region eu-west-1

# AMI level (irreversible)
aws ec2 modify-image-attribute --image-id ami-0abcdef1234567890 --imds-support v2.0

Launch template data (aws ec2 create-launch-template-version … --launch-template-data file://lt.json):

{
  "MetadataOptions": {
    "HttpEndpoint": "enabled",
    "HttpTokens": "required",
    "HttpPutResponseHopLimit": 2
  }
}

Guardrail policy (identity policy or SCP):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RequireImdsV2AtLaunch",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": { "StringNotEquals": { "ec2:MetadataHttpTokens": "required" } }
    },
    {
      "Sid": "CapHopLimit",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": { "NumericGreaterThan": { "ec2:MetadataHttpPutResponseHopLimit": "3" } }
    },
    {
      "Sid": "RejectCredentialsFromImdsV1",
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": { "NumericLessThan": { "ec2:RoleDelivery": "2.0" } }
    }
  ]
}

• SDK/CLI switches: AWS_EC2_METADATA_V1_DISABLED=true (never fall back to v1), AWS_EC2_METADATA_DISABLED=true (never call IMDS, e.g., off-EC2), AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE=IPv6.
• Containers: on ECS prefer task roles (credentials from 169.254.170.2, which the SDKs find before IMDS, §1) and block tasks from the host's IMDS (ECS_AWSVPC_BLOCK_IMDS=true for awsvpc mode; an iptables DROP rule from docker+ to 169.254.169.254 for bridge mode). If containers legitimately need IMDS while v2 is required, set the hop limit to 2.

Traps

• Trap: "The hop limit applies to every metadata response." → Reality: it is the IP TTL of the token (PUT) response; with 1, containers behind a bridge never receive a token.
• Trap: "An IMDSv1 call to an instance that requires v2 returns 403." → Reality: 401 Unauthorized (missing or expired token); 403 means IMDS is turned off or the request is not allowed.
• Trap: "After setting HttpTokens=required, reboot the instance." → Reality: it applies immediately on a running instance.
• Trap: "Setting the account default migrates the whole fleet." → Reality: only new launches; existing instances need modify-instance-metadata-options or replacement. Even enforcement leaves instances that already allow IMDSv1 untouched; it only blocks IMDSv1 launches and switching v2-only instances back (UnsupportedOperation).
• Trap: "Disable IMDS for maximum security." → Reality: that removes role credentials (and, at launch, user data); require IMDSv2 instead.
• Trap: "Requiring IMDSv2 means rewriting every app that uses the SDK." → Reality: supported SDK/CLI versions switch automatically; only hand-written calls and old libraries need changes; find them with MetadataNoToken and IMDS Packet Analyzer first.
• Trap: "Keep watching MetadataNoToken after enforcement." → Reality: it stops emitting once v1 is disabled; watch MetadataNoTokenRejected.
• Trap: "A token can be relayed through a proxy or reused on another instance." → Reality: tokens are bound to the issuing instance and PUTs with X-Forwarded-For are refused.
• Trap: "The token TTL controls how long role credentials last." → Reality: the TTL (1–21,600 s) only scopes the IMDS session; role credentials carry their own Expiration.
• Trap: "ImdsSupport v2.0 can be turned off again on the AMI." → Reality: it is irreversible; build a new AMI from the snapshot.
• Trap: "ec2:MetadataHttpTokens stops the use of credentials stolen via IMDSv1." → Reality: it only constrains launch/modify calls; a deny on ec2:RoleDelivery < 2.0 makes v1-delivered credentials unusable.

Practice questions

1. A company runs Docker containers with bridge networking on EC2 instances. After the security team set HttpTokens=required, processes on the host still call AWS successfully, but SDK calls from inside the containers can no longer find credentials. IMDSv2 must stay enforced. What should a developer do?

   • A. Set the instance's HTTP PUT response hop limit to 2.
   • B. Set HttpTokens back to optional so the SDK can fall back.
   • C. Enable the IMDS IPv6 endpoint on the instances.
   • D. Request tokens with a TTL of 21,600 seconds in the containers.

2. A security team must eliminate IMDSv1 in an AWS account without breaking running applications, and future launches in each Region must default to IMDSv2. Which TWO actions should a developer take? (Select TWO.)

   • A. Confirm that each instance's MetadataNoToken metric is zero, then set HttpTokens to required on it.
   • B. Run modify-instance-metadata-defaults with --http-tokens required in every Region in use.
   • C. Set HttpEndpoint to disabled on every instance that still accepts IMDSv1 requests.
   • D. Stop and start each instance after changing its metadata options so the change applies.
   • E. Rely on the new account default to convert existing instances at their next reboot.

3. User data and bootstrapping

Concept

• User data is an opaque payload attached to an instance at launch. On Linux, cloud-init reads it from IMDS on the instance's first boot and executes it (a shell script or #cloud-config directives); on Windows the launch agent (EC2Launch v2) handles <powershell>, <script> or YAML content.
• It is the quickest way to make a generic AMI do something specific (install packages, pull an artifact, write config, register with a service), but every step lengthens boot time, and installs that run on each launch can pick up different versions over time (on AL2023, mainly from a changing AMI or unpinned sources; see Golden AMI vs bootstrap below).
• The alternatives trade speed and consistency against maintenance: bake a golden AMI (automated with EC2 Image Builder), configure after launch with Systems Manager (Run Command, State Manager), or let CloudFormation drive configuration with cfn-init/cfn-signal.
• What a developer must get right: once vs every boot, the root context, base64 handling per tool, where the logs are, and keeping secrets out of user data.

Exam facts

• When and as whom: runs on the first boot of each instance only (cloud-init "once per instance"); a reboot or stop/start does not run it again. It runs as root, so no sudo; it is non-interactive, so use flags such as -y; files it creates are owned by root.
• Formats (Linux): shell script whose first line is #! plus the interpreter (e.g., #!/bin/bash); #cloud-config YAML (packages, write_files, runcmd, …); MIME multipart to combine both; #cloud-boothook, which runs very early on every boot (make it idempotent; cloud-init-per helps run something once).
• Every boot: install a script in /var/lib/cloud/scripts/per-boot/ (cloud-init runs that directory on every boot), or override the scripts-user module frequency to always in a cloud-config part (cloud_final_modules: [[scripts-user, always]]). Windows: <persist>true</persist> (XML) or frequency: always (EC2Launch v2 YAML).
• Size: 16 KB raw, measured before base64 (encoded size = ceil(n/3) × 4). For bigger logic, keep a small script in user data that downloads the rest from S3 with the instance role. (Windows EC2Launch v2 also accepts zipped user data above 16 KB.)
• Base64: the API field must be base64. Encoded for you: console (unless you tick "already base64 encoded"), CLI run-instances --user-data file://…, boto3 run_instances(UserData=…) (do not pre-encode). You encode: raw API calls, modify-instance-attribute, launch template UserData (create-launch-template[-version], including boto3), CloudFormation (Fn::Base64). describe-instance-attribute returns base64 (decode it yourself); IMDS and the console return it decoded.
• Logs and debugging: /var/log/cloud-init-output.log (stdout/stderr of your script) and /var/log/cloud-init.log; cloud-init status --long shows completion and errors. Without SSH, aws ec2 get-console-output (console: "Get system log") shows the boot console output. Windows EC2Launch v2: C:\ProgramData\Amazon\EC2Launch\log\agent.log.
• cloud-init copies the script to /var/lib/cloud/instances/<instance-id>/ and leaves it there; delete it (and any secrets written to disk) before creating an AMI.
• Changing it: view any time; modify only when the instance is stopped and has an EBS root volume. After the next start the new user data is visible in IMDS but, on Linux, is not executed unless you configured per-boot behaviour. Launch template versions are immutable: create a new version; Auto Scaling uses it only for new instances (roll existing ones with instance refresh, §10). In CloudFormation, changing UserData on an EBS-backed AWS::EC2::Instance restarts the instance.
• Visibility: any process on the instance can read /latest/user-data, and principals with ec2:DescribeInstanceAttribute can read it through the API; it is not encrypted. Never put passwords, keys or tokens in user data. Fetch secrets at boot from Secrets Manager (secretsmanager:GetSecretValue) or Parameter Store SecureString (ssm:GetParameter with decryption), plus kms:Decrypt when a customer managed key is used, all through the instance role.
• A script that calls AWS (CLI/SDK) needs an instance profile with the right permissions (§1). If IMDS is disabled at launch, user data is not delivered and does not run (§2).
• An AMI does not carry the source instance's user data; each launch supplies its own.
• ⚠️ Changed: on Amazon Linux 2023, cloud-init no longer installs security updates at launch by default, a failed remote include (unreachable URL) now fails cloud-init ("fail closed"), and the package manager is DNF. Older material based on Amazon Linux 2 may assume launch-time security updates and yum.
• Golden AMI vs bootstrap: baking dependencies into an AMI shortens the time until a new instance can serve (faster Auto Scaling scale-out) and freezes versions (no drift), at the cost of a rebuild-and-patch pipeline. EC2 Image Builder automates build → validate → test → distribute (multiple Regions, accounts and KMS keys) on a schedule; Image Builder itself is free, you pay for the instances, snapshots and storage it uses. Common split: bake the slow, heavy installs; keep user data for light per-environment settings and secret retrieval. AMI mechanics: see §6.
• ⚠️ Changed: on AL2023, deterministic upgrades (versioned repositories, on by default) lock dnf installs from the Amazon Linux repositories to the AMI's release, so a fixed AMI ID installs the same versions every day. Drift comes from a changing AMI (for example the latest-AMI SSM parameter /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64), --releasever=latest upgrades, or unversioned sources (pip, npm, third-party repositories, downloads); a golden AMI freezes all of them. Older material based on Amazon Linux 2 may say every boot-time yum install pulls the newest packages.
• Lower-ops configuration without SSH: Systems Manager Run Command runs commands across a fleet on demand; State Manager associations keep instances in a defined state on a schedule (State Manager has no additional charge). Same prerequisites as Session Manager (§4): SSM Agent plus an instance role with AmazonSSMManagedInstanceCore, or Default Host Management Configuration.
• CloudFormation: cfn-init applies AWS::CloudFormation::Init metadata; cfn-signal -e $? reports success or failure to a CreationPolicy (ResourceSignal Count default 1, Timeout default PT5M, maximum 12 h). Without enough success signals in time, resource creation fails and the stack rolls back. Details belong to the CloudFormation module.

Developer angle

Bootstrap script for Amazon Linux 2023 that pulls a secret with the instance role instead of embedding it:

#!/bin/bash
set -euxo pipefail        # stop on errors; trace goes to /var/log/cloud-init-output.log
dnf install -y nginx
TOKEN=$(curl -sf -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
REGION=$(curl -sf -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/placement/region)
mkdir -p /etc/orders
# role needs secretsmanager:GetSecretValue on this secret (+ kms:Decrypt for a customer managed key)
aws secretsmanager get-secret-value --secret-id prod/orders/db --region "$REGION" \
  --query SecretString --output text > /etc/orders/db.json
chmod 600 /etc/orders/db.json
# Parameter Store alternative:
# aws ssm get-parameter --name /prod/orders/db-password --with-decryption \
#   --query Parameter.Value --output text --region "$REGION"
systemctl enable --now nginx

Same idea as cloud-config:

#cloud-config
package_update: true
packages:
  - nginx
runcmd:
  - systemctl enable --now nginx

Make something run on every later boot (written during the first boot):

#!/bin/bash
cat > /var/lib/cloud/scripts/per-boot/refresh-config.sh <<'EOF'
#!/bin/bash
/opt/orders/refresh-config.sh
EOF
chmod +x /var/lib/cloud/scripts/per-boot/refresh-config.sh

Pass, change and read user data with the CLI:

# launch: the CLI base64-encodes the file for run-instances
aws ec2 run-instances --image-id ami-0abcdef1234567890 --instance-type t3.micro \
  --iam-instance-profile Name=OrdersAppProfile --user-data file://bootstrap.sh

# change: stop first, encode yourself, start again (the new script will NOT run by itself)
aws ec2 stop-instances --instance-ids i-0123456789abcdef0
aws ec2 wait instance-stopped --instance-ids i-0123456789abcdef0
base64 bootstrap.sh > bootstrap.b64
aws ec2 modify-instance-attribute --instance-id i-0123456789abcdef0 \
  --attribute userData --value file://bootstrap.b64
aws ec2 start-instances --instance-ids i-0123456789abcdef0

# read back: the API returns base64
aws ec2 describe-instance-attribute --instance-id i-0123456789abcdef0 --attribute userData \
  --output text --query "UserData.Value" | base64 --decode

# launch template version: UserData must already be base64 (GNU coreutils base64 shown)
aws ec2 create-launch-template-version --launch-template-id lt-0abcd290751193123 \
  --source-version 1 --launch-template-data "{\"UserData\":\"$(base64 -w0 bootstrap.sh)\"}"

boto3 encodes UserData for run_instances (but not for launch templates):

import boto3

ec2 = boto3.client("ec2")
with open("bootstrap.sh") as f:
    ec2.run_instances(ImageId="ami-0abcdef1234567890", InstanceType="t3.micro",
                      MinCount=1, MaxCount=1,
                      IamInstanceProfile={"Name": "OrdersAppProfile"},
                      UserData=f.read())   # plain text: boto3 base64-encodes it

Debug a failed bootstrap:

sudo tail -n 50 /var/log/cloud-init-output.log
cloud-init status --long
aws ec2 get-console-output --instance-id i-0123456789abcdef0 --output text   # from your workstation

CloudFormation pointer (wait for the app before the stack continues):

WebServer:
  Type: AWS::EC2::Instance
  CreationPolicy:
    ResourceSignal:
      Count: 1
      Timeout: PT15M
  Metadata:
    AWS::CloudFormation::Init:
      config:
        commands:
          01_install:
            command: dnf install -y nginx && systemctl enable --now nginx
  Properties:
    ImageId: ami-0abcdef1234567890
    InstanceType: t3.micro
    IamInstanceProfile: !Ref OrdersInstanceProfile
    UserData:
      Fn::Base64: !Sub |
        #!/bin/bash
        dnf install -y aws-cfn-bootstrap
        /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource WebServer --region ${AWS::Region}
        /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource WebServer --region ${AWS::Region}

Traps

• Trap: "A reboot or stop/start runs user data again." → Reality: it runs once per instance by default; per-boot needs explicit configuration (per-boot directory, frequency override, boothook, or <persist> on Windows).
• Trap: "Edit the user data of a running instance." → Reality: stop it first (EBS root only), and even then the edited script does not run on start unless configured to.
• Trap: "Base64-encoding hides the password." → Reality: base64 is encoding, not encryption; user data is readable through IMDS and DescribeInstanceAttribute. Use Secrets Manager or Parameter Store via the instance role.
• Trap: "The 16 KB limit applies to the encoded text." → Reality: 16 KB of raw data, before base64.
• Trap: "Always base64-encode before aws ec2 run-instances --user-data or boto3 run_instances." → Reality: both encode for you (double-encoding breaks the script); you encode for modify-instance-attribute, launch template data, raw API calls and CloudFormation (Fn::Base64).
• Trap: "Prefix commands with sudo." → Reality: user data already runs as root; interactive prompts hang the script.
• Trap: "A failed script shows up as a failed status check." → Reality: status checks do not look at bootstrap; read /var/log/cloud-init-output.log or the console output, and in CloudFormation use cfn-signal so a failure rolls the stack back.
• Trap: "AWS CLI calls in user data work out of the box." → Reality: they need an instance profile with permissions for those calls.
• Trap: "Creating an AMI copies the user data into it." → Reality: it doesn't, but cloud-init's copy under /var/lib/cloud/instances/<id>/ and any secrets written to disk are baked in; clean up first.
• Trap: "Scale-out is slow, so raise the cooldown or choose a bigger instance." → Reality: shorten the bootstrap: golden AMI with dependencies preinstalled, minimal user data.
• Trap: "Updating the launch template's user data updates the running ASG instances." → Reality: only new launches use the new version; replace existing instances with instance refresh (§10).

Practice questions

1. A developer's launch template passes a database password to EC2 instances in user data so that a startup script can write the application's configuration file. A security review flags this design. Which approach is MOST secure while keeping the bootstrap fully automated?

   • A. Base64-encode the password before adding it to the user data script.
   • B. Keep it in Secrets Manager; retrieve it at boot with the instance role.
   • C. Store the password in an instance tag and read it from instance metadata tags.
   • D. Bake the password into a configuration file inside a private golden AMI.

2. An Auto Scaling group's new instances take about 12 minutes to pass health checks because user data compiles and installs dependencies at boot. The team wants faster, repeatable scale-out with the least ongoing manual work. Which TWO actions should a developer take? (Select TWO.)

   • A. Build an AMI with the dependencies preinstalled using an EC2 Image Builder pipeline.
   • B. Keep only short, environment-specific steps, such as fetching config, in user data.
   • C. Move the install commands into a #cloud-boothook so that they run earlier in boot.
   • D. Raise the group's default cooldown so instances have more time to finish booting.
   • E. Update the user data on running instances so that new instances inherit the change.

4. Security groups and connectivity troubleshooting

Concept

• A security group (SG) is a stateful, allow-only firewall attached to an elastic network interface (ENI), so the same tool protects instances, load balancers, RDS databases, VPC-attached Lambda functions and interface endpoints. Reply traffic for a connection you allowed is let back automatically.
• Network ACLs (NACLs) are a second, subnet-level layer: stateless, numbered, with explicit allow and deny rules. For DVA-C02 you only need to recognise them as a possible blocker.
• Read the error before changing anything: a timeout points at the network path, an instant connection refused means the host answered but nothing listens, and AccessDenied / UnauthorizedOperation is IAM, not networking.
• Network least privilege is built with SG references (ALB SG → app SG → DB SG), not wide CIDR ranges.
• For shell access the exam's preferred answer is Session Manager (no inbound ports, IAM-controlled, logged). EC2 Instance Connect, optionally through an EIC Endpoint, is the SSH-based alternative.

Exam facts

• Rule = protocol + port range + source (inbound) or destination (outbound). Source/destination can be a CIDR, a managed prefix list or an SG ID. There are no deny rules.
• A new custom SG has no inbound rules (all inbound blocked) and one outbound allow-all rule. The VPC's default SG allows inbound from its own members and all outbound; an instance launched without an SG gets the VPC default SG.
• Several SGs on one ENI: their rules are combined (union). Rule changes apply to every associated resource automatically and immediately, no restart. Nuance: an already-tracked connection survives the removal of the rule that allowed it until it goes idle; a NACL deny cuts it at once.
• SG reference = "allow traffic from the private IPs of any ENI that carries SG X" (same VPC, or a peered VPC in the same Region; for a peer in another account write 123456789012/sg-...; an inter-Region peer needs its CIDR instead; inbound references also work across a transit gateway). No rules are copied from the referenced SG. It keeps working while an Auto Scaling group replaces instances and IPs change, and counts as one rule.
• Default quotas per Region: 2,500 SGs; 60 inbound + 60 outbound rules per SG (counted separately for IPv4 and IPv6); 5 SGs per ENI (adjustable to 16); rules per SG × SGs per ENI ≤ 1,000.
• ⚠️ Changed: an SG is still Region-scoped, but it can now be associated with other VPCs in the same Region (Security Group VPC Associations) and shared through AWS Organizations. Older material may say an SG is locked to a single VPC.
• SGs and NACLs never filter traffic to IMDS (169.254.169.254), the Amazon DNS resolver, Amazon Time Sync or DHCP.
• NACL (awareness): one per subnet; stateless, so return traffic on ephemeral ports 1024–65535 must be allowed explicitly; rules numbered 1–32766, lowest number evaluated first, first match wins; the default NACL allows everything, a new custom NACL denies everything.
• Addresses: an auto-assigned public IPv4 is released on stop, hibernate or terminate and a new one is assigned at the next start (a reboot keeps it); the private IPv4 stays until termination; an Elastic IP is a static public address you own (5 per Region by default).
• ⚠️ Changed: since 1 Feb 2024 every public IPv4 address is billed $0.005 per IP-hour, in use or idle, including an Elastic IP attached to a running instance. Older material may say an Elastic IP is free while attached to a running instance.
• EC2 Instance Connect (EIC): the SendSSHPublicKey API places a one-time public key in instance metadata for 60 seconds; who may push keys is decided by IAM (ec2-instance-connect:SendSSHPublicKey, optionally limited with ec2:osuser); every request is logged in CloudTrail; no extra cost. It is still SSH, so the instance SG needs TCP 22 from the Region's EIC prefix list (browser client: com.amazonaws.<region>.ec2-instance-connect for IPv4, com.amazonaws.<region>.ipv6.ec2-instance-connect for IPv6) or from your own IP (CLI/SSH client), plus a public IPv4 or IPv6 address in a subnet with a route to the internet (your own SSH client can also reach a private IP over VPN, Direct Connect or peering).
• ⚠️ Changed: since September 2024 EC2 Instance Connect also connects to an instance's public IPv6 address, with a separate IPv6 prefix list for the browser client. Older material may say EC2 Instance Connect needs a public IPv4 address.
• ⚠️ Changed: EIC is preinstalled on the AL2023 standard AMI, Amazon Linux 2 (2.0.20190618 or later), Ubuntu 20.04 or later and recent macOS AMIs. Older material may say it only works out of the box on Amazon Linux 2.
• ⚠️ Changed: an EC2 Instance Connect Endpoint (June 2023) reaches instances by private IPv4 address (IPv6 or dual-stack endpoints since October 2025) with no public IP, internet gateway or bastion. The instance SG allows 22 (or 3389 for RDP) from the endpoint's SG; IAM action ec2-instance-connect:OpenTunnel; max 1 hour per connection, 20 concurrent connections per endpoint, 1 endpoint per VPC and per subnet, 5 per Region; no extra charge (cross-AZ data transfer applies). Older material may say EC2 Instance Connect always needs a public IP.
• Session Manager (Systems Manager) needs: (1) SSM Agent running (preinstalled on Amazon Linux 2, AL2023, recent Ubuntu LTS and Windows Server AMIs); (2) permissions, either an instance profile with AmazonSSMManagedInstanceCore or Default Host Management Configuration (account/Region setting, requires IMDSv2); (3) outbound HTTPS 443 to the ssm, ssmmessages and ec2messages endpoints through a NAT/internet gateway or interface VPC endpoints (whose SG must allow inbound 443 from the instances). No inbound rules, no SSH keys, no bastion. Who may connect is an IAM decision (ssm:StartSession); API calls land in CloudTrail and session output can stream to S3 or CloudWatch Logs (not for port-forwarding or SSH-tunnel sessions). Port forwarding works to the node itself or through it to a remote host such as an RDS endpoint.

Classic ports (all TCP)

Error-symptom matrix

Developer angle

Least-privilege tiers and path checks (IDs are examples):

ALB_SG=sg-0a1b2c3d4e5f6a7b8; APP_SG=sg-0b2c3d4e5f6a7b8c9; DB_SG=sg-0c3d4e5f6a7b8c9d0
aws ec2 authorize-security-group-ingress --group-id "$ALB_SG" --protocol tcp --port 443 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id "$APP_SG" --protocol tcp --port 8080 --source-group "$ALB_SG"
aws ec2 authorize-security-group-ingress --group-id "$DB_SG" --protocol tcp --port 5432 --source-group "$APP_SG"

# What is really attached to the instance?
aws ec2 describe-instances --instance-ids i-0123456789abcdef0 \
  --query 'Reservations[].Instances[].{SGs:SecurityGroups[].GroupId,PublicIp:PublicIpAddress,Subnet:SubnetId}'

# Reachability Analyzer names the blocking SG, NACL or route
aws ec2 create-network-insights-path --source igw-0123456789abcdef0 \
  --destination i-0123456789abcdef0 --protocol tcp --destination-port 443
aws ec2 start-network-insights-analysis --network-insights-path-id nip-0123456789abcdef0

Shell access without opening port 22 to the world:

# Instance side: the role needs this managed policy (or enable Default Host Management Configuration)
aws iam attach-role-policy --role-name app-instance-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

# Your side: AWS CLI + Session Manager plugin + IAM permission ssm:StartSession
aws ssm start-session --target i-0123456789abcdef0

# Tunnel to a private database through the instance (no SSH, no bastion)
aws ssm start-session --target i-0123456789abcdef0 \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["orders.cluster-abc123.eu-west-1.rds.amazonaws.com"],"portNumber":["5432"],"localPortNumber":["5432"]}'

# EC2 Instance Connect: push a 60-second key, then SSH within that window (port 22 must be allowed)
aws ec2-instance-connect send-ssh-public-key --instance-id i-0123456789abcdef0 \
  --instance-os-user ec2-user --ssh-public-key file://~/.ssh/id_rsa.pub
ssh -i ~/.ssh/id_rsa ec2-user@ec2-203-0-113-25.eu-west-1.compute.amazonaws.com

# Private instance through an EIC Endpoint (AWS CLI v2)
aws ec2 create-instance-connect-endpoint --subnet-id subnet-0123456789abcdef0 \
  --security-group-ids sg-0d4e5f6a7b8c9d0e1
aws ec2-instance-connect ssh --instance-id i-0123456789abcdef0 --os-user ec2-user --connection-type eice

Key and address checks (identity checks with sts get-caller-identity and sts decode-authorization-message: see §1):

chmod 400 my-key.pem                  # fixes "UNPROTECTED PRIVATE KEY FILE"
aws ec2 allocate-address --domain vpc # stable public IPv4 (billed hourly)
aws ec2 associate-address --instance-id i-0123456789abcdef0 --allocation-id eipalloc-0123456789abcdef0

Traps

• Trap: "The request times out, so the application must have crashed." → Reality: a crashed or non-listening app answers instantly with connection refused; a timeout means packets were dropped (SG, NACL, route or missing public IP).
• Trap: "Add a deny rule to the security group for the attacker's IP." → Reality: SGs can only allow; block specific CIDRs with a NACL deny (or AWS WAF for HTTP).
• Trap: "Allow the database port from the app subnet CIDR, or from 0.0.0.0/0 for simplicity." → Reality: reference the app tier's SG ID; it follows instances as they scale and is the least-privilege answer.
• Trap: "Response traffic needs an outbound rule on the SG." → Reality: SGs are stateful; only NACLs need explicit return rules on ephemeral ports.
• Trap: "An AccessDenied from S3 means port 443 is blocked." → Reality: the call reached S3; fix the IAM or bucket policy. A blocked path ends in a connect timeout instead.
• Trap: "Open SSH to 0.0.0.0/0 (or build a bastion host) so developers can debug." → Reality: Session Manager is the MOST secure and LEAST operational-overhead choice: no inbound port, no keys, IAM plus CloudTrail.
• Trap: "Session Manager needs inbound 443 on the instance." → Reality: SSM Agent connects outbound; only the interface VPC endpoint's SG needs inbound 443.
• Trap: "EC2 Instance Connect removes the need for port 22." → Reality: it is still SSH on 22, allowed from the EIC prefix list or your IP, or from the EIC Endpoint's SG.
• Trap: "The public IP survives a stop/start." → Reality: it changes (a reboot keeps it); use an Elastic IP, DNS or a load balancer.
• Trap: "Ping fails, so the instance is down." → Reality: ICMP is not allowed by default; add an ICMP rule or test the real port.
• Trap: "Permission denied (publickey) means the SG is wrong." → Reality: the TCP connection succeeded; check the username, key and key-file permissions.

Practice questions

1. A developer runs an API on port 8080 on an EC2 instance. Calls from another instance in the same VPC fail immediately with Connection refused. The API instance's security group allows TCP 8080 from the caller's security group. What is the MOST likely cause?

   • A. The subnet's network ACL denies inbound traffic on port 8080.
   • B. The API process is not listening on port 8080 on the instance's network interface.
   • C. The instance role does not allow the ec2:AuthorizeSecurityGroupIngress action.
   • D. The API instance's security group has no outbound rule for the response traffic.

2. A company runs EC2 instances in private subnets that have no NAT gateway and no internet gateway. Engineers need interactive shell access with no inbound ports open and with session API calls recorded in AWS CloudTrail. Which TWO steps will meet these requirements? (Select TWO.)

   • A. Attach an instance profile whose role includes the AmazonSSMManagedInstanceCore managed policy.
   • B. Create interface VPC endpoints for ssm, ssmmessages and ec2messages, with a security group that allows inbound HTTPS from the instances.
   • C. Add an inbound rule for TCP 22 from 0.0.0.0/0 to the instances' security group.
   • D. Allocate and associate an Elastic IP address with each instance.
   • E. Allow TCP 443 inbound on the instances' security group from the EC2 Instance Connect prefix list.

5. Purchasing options and Spot

Concept

• Every purchasing option runs the same instances; what changes is the price, the commitment, and whether capacity is guaranteed or can be taken back.
• Commitments buy discounts: Savings Plans commit dollars per hour, Reserved Instances (RIs) commit to an instance configuration, both for 1 or 3 years. Spot sells spare capacity at up to 90% off, but EC2 can reclaim it with a 2-minute warning.
• Dedicated Hosts / Dedicated Instances solve licensing and compliance (physical isolation). On-Demand Capacity Reservations and Capacity Blocks for ML guarantee capacity, not a lower price.
• Developer impact: Spot only suits code that is stateless, idempotent or checkpointed. Let an Auto Scaling group (ASG) or EC2 Fleet spread capacity over many instance types and AZs, and react to the interruption signals in code.

Exam facts

• On-Demand: no commitment, highest unit price, billed only while running (attached EBS volumes are still billed when stopped). Per-second billing with a 60-second minimum for Linux, Windows (including SQL Server editions), RHEL and Ubuntu Pro; SUSE Linux Enterprise Server is billed per full hour.
• ⚠️ Changed: RHEL and Ubuntu Pro are now billed per second (60 s minimum) and only SLES is still hourly. Older material may say only Linux and Windows are per-second and every other OS is hourly.
• Reserved Instances: 1- or 3-year term; All, Partial or No Upfront (longer term and more upfront = bigger discount).
  • Standard up to 72% off: cannot be exchanged; can be sold (and bought) on the RI Marketplace.
  • Convertible up to 66% off: can be exchanged during the term for another Convertible RI with a different family, type, platform, scope or tenancy (never for another Region); cannot be sold. Both classes allow some attributes to be modified, and both stay in the Region where you bought them.
  • Scope (same price): regional = no capacity reservation, discount floats across AZs and, for Linux/Unix with default tenancy, across sizes in the family; zonal = capacity reserved in one AZ, no size flexibility.
• Savings Plans: $/hour commitment for 1 or 3 years, same payment options, usage above the commitment billed at On-Demand rates; the commitment can't be changed or cancelled, only returned after a purchase error (next bullet).
  • ⚠️ Changed: since March 2024 a Savings Plan with an hourly commitment of $100 or less can be returned within 7 days of purchase and in the same calendar month (up to 10 returns per calendar year per management account; upfront payments refunded in full; ReturnSavingsPlan API). Older material may say a Savings Plan can never be cancelled or returned.
  • Compute Savings Plans up to 66% off: any instance family, size, AZ, Region, OS or tenancy, plus AWS Fargate and AWS Lambda.
  • EC2 Instance Savings Plans up to 72% off: one instance family in one Region; any size, OS or tenancy inside it.
  • SageMaker AI Savings Plans up to 64% off.
  • ⚠️ Changed: AWS now lists four Savings Plans types, adding Database Savings Plans (up to 35% off across RDS, Aurora, DynamoDB, ElastiCache and other database services). Older material may name only Compute, EC2 Instance and SageMaker plans.
  • Savings Plans reserve no capacity (pair them with Capacity Reservations) and never apply to Spot usage.
• Spot pricing: up to 90% off On-Demand. EC2 sets the price per capacity pool (instance type + AZ) and moves it gradually with long-term supply and demand; nobody bids. The max price defaults to the On-Demand price, AWS recommends not setting one (a max price only causes more interruptions), and you always pay the current Spot price.
• ⚠️ Changed: no bidding; interruptions happen mainly because EC2 needs the capacity back, and the default max price is the On-Demand price. Older material may describe Spot as an auction where you lose the instance when someone outbids you or the price passes your bid.
• Interruption causes: capacity reclaim (main reason), Spot price above your max price, launch-group or AZ-group constraints.
• Interruption behaviour: terminate (default), stop or hibernate. Stop and hibernate require a persistent request (or EC2 Fleet/Spot Fleet type maintain); a one-time request can only terminate. While stopped you pay only for EBS.
• 2-minute interruption notice before stop or terminate: instance metadata item latest/meta-data/spot/instance-action (HTTP 404 until scheduled; returns {"action": ..., "time": ...}) and EventBridge event EC2 Spot Instance Interruption Warning; check every 5 seconds. With the hibernate behaviour you still get the notice, but hibernation starts immediately, so there is no 2-minute grace period.
• Rebalance recommendation: an earlier, best-effort "elevated risk" signal at latest/meta-data/events/recommendations/rebalance and EventBridge EC2 Instance Rebalance Recommendation; it can also arrive together with the 2-minute notice. Use it to checkpoint, stop taking new work, or launch a replacement.
• Billing on interruption: if EC2 interrupts in the first instance-hour there is no charge; if you stop or terminate it yourself you pay per second used (SLES: full hours).
• Request types: one-time (default) or persistent (re-opened after an interruption until it expires or is cancelled). Terminating an instance launched by a persistent request re-opens the request and a new instance launches, so cancel the request first, then terminate. Cancelling an active request does not terminate its running instances (stopped instances of a cancelled request are terminated).
• ⚠️ Changed: RequestSpotInstances and RequestSpotFleet (Spot Fleet) are legacy APIs with no planned investment; AWS recommends ASGs (CreateAutoScalingGroup) or EC2 Fleet (CreateFleet: types instant, request, maintain; default maintain), or RunInstances with Spot market options for a single instance type. Defined-duration "Spot blocks" are deprecated. Older material may teach Spot Fleet and Spot blocks as the main tools.
• Allocation strategies: price-capacity-optimized (recommended: pools least likely to be interrupted, then lowest price), capacity-optimized / capacity-optimized-prioritized (high cost of interruption, e.g. long CI or rendering jobs), diversified (fleets only), lowest-price (not recommended: highest interruption risk). lowest-price is still the API default for the ASG SpotAllocationStrategy and for EC2 Fleet; in an ASG it can only be chosen through the CLI/API, and the console preselects price-capacity-optimized.
• ⚠️ Changed: price-capacity-optimized is the current recommendation. Older material may recommend lowest-price or capacity-optimized.
• Diversify: be flexible across at least 10 instance types and all AZs; attribute-based instance type selection and Spot placement scores (1–10 per Region/AZ) help.
• Dedicated Hosts: a whole physical server for your use; per-host billing; visibility of sockets, physical cores and host ID; host affinity and targeted placement → BYOL with per-socket, per-core or per-VM licences and strict compliance. On-Demand or Dedicated Host Reservations (up to 70% off). Capacity Reservations are not supported.
• Dedicated Instances: run on hardware dedicated to your account (may share it with your other instances), per-instance billing plus a $2/hour fee in each Region where at least one runs; no socket/core visibility or host affinity; Capacity Reservations are supported.
• On-Demand Capacity Reservations (ODCR): reserve capacity for an instance type, platform and tenancy in one AZ for any duration; billed at the On-Demand rate whether instances run or not; no discount on their own, but regional RIs or Savings Plans apply to them (zonal RI discounts do not). Immediate-use reservations carry no commitment and can be cancelled any time. They count against your On-Demand instance quota.
• ⚠️ Changed: future-dated Capacity Reservations (since Nov 2024) need a committed duration (minimum 32 vCPUs per request) and a cancellation charge may apply. Older material may say Capacity Reservations never involve a commitment.
• Capacity Blocks for ML (awareness): book GPU/Trainium capacity (P5, P4d, Trn1/Trn2 and others) for a future window starting up to 8 weeks ahead; 1–14 days in 1-day steps or up to 182 days in 7-day steps; up to 64 instances per block; paid upfront; cannot be cancelled.
• ASG mixed instances policy (MixedInstancesPolicy.InstancesDistribution):
  • OnDemandBaseCapacity (default 0): the first N capacity units are always On-Demand.
  • OnDemandPercentageAboveBaseCapacity (default 100, i.e. all On-Demand until you lower it): On-Demand share of capacity above the base; the rest is Spot.
  • SpotAllocationStrategy; OnDemandAllocationStrategy (prioritized = order of the launch template overrides, the default with manually chosen types; lowest-price = default with attribute-based selection); SpotInstancePools (only with lowest-price, default 2, range 1–20); SpotMaxPrice (leave unset).
  • Capacity Rebalancing (CapacityRebalance: true / --capacity-rebalance): on a rebalance recommendation the ASG launches a replacement Spot instance, waits for it to pass health checks, then terminates the at-risk one; it may exceed the max size by up to 10% of desired capacity while doing so. Lifecycle hooks (§10) give the old instance time to drain (keep the work under 2 minutes).
• Launch errors: InsufficientInstanceCapacity = AWS has no free capacity for that type in that AZ right now (retry later, another AZ or type; reserve ahead with an ODCR); VcpuLimitExceeded / InstanceLimitExceeded = your account's On-Demand quota (raise it in Service Quotas); MaxSpotInstanceCountExceeded = your Spot quota.

Developer angle

Watch for the 2-minute notice from inside a Spot instance (IMDSv2):

#!/bin/bash
while true; do
  TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
    -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
  CODE=$(curl -s -o /dev/null -w '%{http_code}' -H "X-aws-ec2-metadata-token: $TOKEN" \
    http://169.254.169.254/latest/meta-data/spot/instance-action)
  if [ "$CODE" = "200" ]; then
    /opt/worker/bin/checkpoint-to-s3   # persist progress, stop pulling new jobs
    break
  fi
  sleep 5
done

React centrally with EventBridge (the Lambda function also needs a resource policy that allows events.amazonaws.com to invoke it):

aws events put-rule --name spot-warnings --event-pattern \
  '{"source":["aws.ec2"],"detail-type":["EC2 Spot Instance Interruption Warning","EC2 Instance Rebalance Recommendation"]}'
aws events put-targets --rule spot-warnings \
  --targets Id=drain,Arn=arn:aws:lambda:eu-west-1:111122223333:function:drain-worker

ASG with an On-Demand base and Spot for the rest (aws autoscaling create-auto-scaling-group --cli-input-json file://ci-runners.json):

{
  "AutoScalingGroupName": "ci-runners",
  "MinSize": 2,
  "MaxSize": 40,
  "DesiredCapacity": 6,
  "VPCZoneIdentifier": "subnet-0a1b2c3d4e5f6a7b8,subnet-0b2c3d4e5f6a7b8c9",
  "CapacityRebalance": true,
  "MixedInstancesPolicy": {
    "LaunchTemplate": {
      "LaunchTemplateSpecification": { "LaunchTemplateName": "ci-runner", "Version": "$Latest" },
      "Overrides": [
        { "InstanceType": "m7i.large" },
        { "InstanceType": "m6i.large" },
        { "InstanceType": "m7a.large" },
        { "InstanceType": "c7i.xlarge" }
      ]
    },
    "InstancesDistribution": {
      "OnDemandBaseCapacity": 2,
      "OnDemandPercentageAboveBaseCapacity": 20,
      "SpotAllocationStrategy": "price-capacity-optimized"
    }
  }
}

Here 2 instances are always On-Demand; everything above them is 20% On-Demand and 80% Spot.

Single-type Spot launch and clean shutdown of a persistent request:

aws ec2 run-instances --launch-template LaunchTemplateName=batch-worker,Version=3 \
  --instance-market-options '{"MarketType":"spot","SpotOptions":{"SpotInstanceType":"one-time","InstanceInterruptionBehavior":"terminate"}}'

aws ec2 describe-spot-instance-requests --spot-instance-request-ids sir-08b93456 \
  --query 'SpotInstanceRequests[].{Type:Type,State:State,Instance:InstanceId}'
aws ec2 cancel-spot-instance-requests --spot-instance-request-ids sir-08b93456   # 1. stop relaunches
aws ec2 terminate-instances --instance-ids i-0123456789abcdef0                   # 2. then terminate

Traps

• Trap: "Bid above other customers to keep your Spot instances." → Reality: there is no bidding; leave the max price unset, diversify instance types and AZs, and use price-capacity-optimized.
• Trap: "A max price protects you from paying too much." → Reality: you always pay the current Spot price (capped at On-Demand by default); a max price only adds interruptions.
• Trap: "Terminate the instances of a persistent Spot request to stop paying." → Reality: the request re-opens and launches replacements; cancel the request first, then terminate.
• Trap: "Cancelling the Spot request also terminates its instances." → Reality: running instances keep running and billing; terminate them separately.
• Trap: "Every interruption comes with a 2-minute grace period." → Reality: with the hibernate behaviour hibernation begins immediately, and rebalance signals are best effort.
• Trap: "Turn on termination or stop protection so EC2 cannot reclaim the Spot Instance." → Reality: neither protection can be enabled on Spot Instances; design for interruption instead.
• Trap: "Run the primary database on Spot to cut cost." → Reality: Spot fits fault-tolerant, stateless or checkpointed work (batch, CI, rendering, containers), not critical stateful systems.
• Trap: "Savings Plans or regional RIs guarantee capacity." → Reality: only zonal RIs, On-Demand Capacity Reservations and Capacity Blocks reserve capacity.
• Trap: "An EC2 Instance Savings Plan still applies after moving to another family, Region, Fargate or Lambda." → Reality: only a Compute Savings Plan follows those moves.
• Trap: "Convertible RIs can be resold on the RI Marketplace." → Reality: only Standard RIs can be sold; Convertible RIs are exchanged.
• Trap: "Dedicated Instances satisfy per-socket or per-core licensing." → Reality: that needs Dedicated Hosts (socket/core visibility and host affinity).
• Trap: "A Capacity Reservation is cheaper than On-Demand." → Reality: it costs the On-Demand rate even when unused; the discount comes from Savings Plans or regional RIs applied to it.
• Trap: "The mixed instances ASG launches no Spot, so the Spot market must be empty." → Reality: check OnDemandPercentageAboveBaseCapacity; its default of 100 means all On-Demand.
• Trap: "lowest-price is the cheapest strategy overall." → Reality: it has the highest interruption rate; price-capacity-optimized is recommended.

Practice questions

1. A company runs a steady 24/7 workload on m6i instances in eu-west-1. Over the next three years it plans to move part of the workload to Graviton-based m7g instances and some services to AWS Fargate. Which purchase gives the largest discount that keeps applying automatically after these changes?

   • A. An EC2 Instance Savings Plan for the m6i family in eu-west-1
   • B. A Compute Savings Plan
   • C. Standard Reserved Instances for m6i in eu-west-1
   • D. On-Demand Capacity Reservations for m6i in eu-west-1

2. A developer's job-processing application runs on Spot Instances and must save its progress to Amazon S3 before an instance is reclaimed. Which TWO mechanisms give the application advance warning that it can act on? (Select TWO.)

   • A. Poll http://169.254.169.254/latest/meta-data/spot/instance-action with an IMDSv2 token every few seconds.
   • B. Create an Amazon EventBridge rule that matches the EC2 Spot Instance Interruption Warning event and invokes a draining function.
   • C. Create a CloudWatch alarm on the StatusCheckFailed_System metric of each instance.
   • D. Enable termination protection on the Spot Instances.
   • E. Set the Spot maximum price equal to the On-Demand price.

6. EBS volumes, snapshots, encryption and AMIs

Concept

• Amazon EBS is durable block storage that an instance reaches over the AWS network. A volume lives in exactly one Availability Zone, can outlive the instance, and is billed for what you provision (GiB, plus IOPS/throughput where configurable), not for what you write.
• Snapshots are the backup and portability layer: Region-scoped, incremental, stored by EBS in S3. They are the classic way to move EBS data to another AZ, Region or account; newer same-AZ volume copies (optionally from a volume shared through AWS RAM) skip the snapshot when the data stays in its AZ.
• Encryption is a KMS-backed property fixed when a volume or snapshot is created. Because it cannot be switched on in place, "encrypt an existing volume" and "share an encrypted snapshot/AMI with another account" are classic exam workflows.
• An AMI is the Regional launch image for the disk layer: snapshots + block device mapping + launch permissions. Developers bake golden AMIs, copy them to other Regions for deployment/DR, and resolve the latest AWS-provided AMI through SSM public parameters.
• What a developer is tested on: picking the right volume type under an IOPS/cost constraint, scripting snapshot/AMI pipelines with the CLI/SDK, and fixing KMS permission problems that make launches fail.

Exam facts

Volumes and attachment

• A volume and its instance must be in the same AZ. To use the data elsewhere: snapshot → (copy-snapshot for another Region) → create-volume in the target AZ.
• Normally one instance per volume. Multi-Attach (io1/io2 only) lets up to 16 Nitro-based instances within one AZ attach the same volume simultaneously, all with read/write access. It requires a cluster-aware file system (XFS/ext4 are not safe for concurrent writers) and an application that orders its writes. Not allowed for boot volumes, cannot be enabled at launch (console/RunInstances), and can be turned on/off only while the volume is detached. Windows supports io2 Multi-Attach only.
• A volume's performance is also capped by the instance type's own EBS bandwidth/IOPS limit; the lower of the two wins.
• DeleteOnTermination: root volume created at launch = true (deleted with the instance); volumes attached after launch = false (kept and still billed). Data volumes declared in the launch block device mapping follow the AMI/request value (console defaults them to No, the CLI/API mapping to Yes), so set the flag explicitly. A volume attached later as the root (for example after a swap) also defaults to keep.
• Default-type trap: the console proposes gp3, but the CreateVolume and CopyVolumes APIs (and CLI) default to gp2 when no volume type is given, so a copy of a gp3 volume silently becomes gp2 (300 IOPS baseline at 100 GiB). Always pass --volume-type.

Volume types (current limits)

• gp3 (General Purpose SSD, the default choice): 1 GiB–64 TiB. 3,000 IOPS and 125 MiB/s are included at every size; provision up to 80,000 IOPS (max 500 IOPS per GiB, so the full 80,000 from 160 GiB) and 2,000 MiB/s (max 0.25 MiB/s per provisioned IOPS, so ≥ 8,000 IOPS). No burst credits; about 20% cheaper per GiB than gp2. ⚠️ Changed: gp3 maximums are 64 TiB / 80,000 IOPS / 2,000 MiB/s (since 2025-09-26). Older material may say 16 TiB / 16,000 IOPS / 1,000 MiB/s.
• gp2 (previous General Purpose SSD): 1 GiB–16 TiB. Baseline IOPS tied to size at 3 IOPS per GiB (minimum 100, maximum 16,000 reached at 5,334 GiB). Volumes under 1 TiB burst to 3,000 IOPS from an I/O-credit bucket (5.4 million credits, ≥ 30 min at 3,000 IOPS; BurstBalance metric). Throughput 128–250 MiB/s depending on size.
• io2 Block Express (Provisioned IOPS SSD): 4 GiB–64 TiB, up to 256,000 IOPS (1,000 IOPS per GiB) and 4,000 MiB/s, 99.999% durability, average latency under 500 µs, Multi-Attach. Full performance needs a Nitro instance (others reach 32,000 IOPS). ⚠️ Changed: since 2025-04-30 every io2 volume is io2 Block Express. Older material may say io2 is limited to 64,000 IOPS and 16 TiB unless it is Block Express.
• io1: 4 GiB–16 TiB, up to 64,000 IOPS (50 IOPS per GiB) on Nitro and 32,000 elsewhere, 1,000 MiB/s, 99.8–99.9% durability. AWS recommends io2 instead (better durability and performance at a lower cost).
• st1 (Throughput Optimized HDD: logs, ETL, big sequential reads): 125 GiB–16 TiB, 500 MiB/s, 500 IOPS (1 MiB I/O). sc1 (Cold HDD, cheapest per GiB): 125 GiB–16 TiB, 250 MiB/s, 250 IOPS. Neither HDD type can be a boot volume.
• Choosing: ⚠️ Changed: the rule "more than 16,000 IOPS means io1/io2" no longer holds, because gp3 now covers up to 80,000 IOPS and 2,000 MiB/s. Use io2 Block Express for more than 80,000 IOPS or 2,000 MiB/s, consistent sub-millisecond latency, 99.999% durability, or Multi-Attach. Older material may say any requirement above 16,000 IOPS needs Provisioned IOPS.

Elastic Volumes (change a volume in place)

• modify-volume raises the size (never lowers it), changes the type, IOPS and gp3 throughput of an attached volume on current-generation instances without downtime. A request cannot be cancelled. To shrink, create a smaller volume and copy the data.
• States: modifying → optimizing → completed. Added capacity is usable once optimizing begins, but the OS keeps seeing the old size until you extend the partition and file system (growpart, then xfs_growfs for XFS or resize2fs for ext4).
• ⚠️ Changed: a new modification can start as soon as the previous one reaches completed, up to 4 modifications per volume in a rolling 24 hours (since 2026-01-15). Older material may say you must wait at least 6 hours between modifications of the same volume.
• gp2 → gp3 without specifying performance: EBS provisions whichever is higher, the gp2-equivalent performance or the gp3 baseline.

Encryption

• AES-256 envelope encryption: KMS generates a data key, which is stored encrypted with the volume metadata and decrypted onto the Nitro host at attach time. An encrypted volume protects data at rest, data moving between the instance and the volume, every snapshot taken from it and every volume restored from those snapshots. Encrypted and unencrypted volumes can be attached to the same instance. ⚠️ Changed: all EC2 instance types support EBS encryption. Older material may say only some instance types do.
• Keys: the AWS managed key aws/ebs (default, one per Region) or a symmetric customer managed key (your key policy, rotation, cross-account use).
• Encryption by default is an opt-in setting per account and per Region. New volumes and new snapshot copies are then always encrypted (with the Region's default key, which you can point at a customer managed key); existing volumes and snapshots are not touched.
• Nothing changes in place: you cannot encrypt an existing unencrypted volume or snapshot, remove encryption, or change the KMS key of an existing volume or snapshot. Path: snapshot → create-volume --encrypted (or copy-snapshot --encrypted, then create the volume) → swap volumes or build a new AMI. Re-encrypting under a different key produces a full, non-incremental copy.
• Disabling or losing access to the KMS key does not affect volumes that are already attached (the Nitro host holds the decrypted data key) but the next attach fails.
• Launch-failure signature: the instance goes pending → terminated with Client.InternalError: Client error on launch when the caller, or the Auto Scaling service-linked role, lacks KMS permissions on the key used by the AMI or volumes.

Snapshots

• Point-in-time and Region-scoped. The first snapshot of a volume is full; later ones store only changed blocks. You pay for stored data, not volume size. Deleting a snapshot removes only blocks that no other snapshot references.
• Asynchronous: the volume stays usable while the snapshot is pending. For consistency pause writes or unmount; stop the instance before snapshotting a root volume.
• copy-snapshot runs in the destination Region with --source-region. The first copy into a Region or account is full, later copies can be incremental. KMS keys are Regional, so name a key that exists in the destination Region (otherwise that Region's default EBS key is used). Up to 20 concurrent copy requests per destination. Optional time-based copy: completion target of 15 min–48 h in 15-minute steps, extra charge (copy-snapshot --completion-duration-minutes; for an AMI, copy-image --snapshot-copy-completion-duration-minutes, which applies the target to each of its snapshots).
• Sharing: modify-snapshot-attribute --attribute createVolumePermission. Encrypted snapshots can be shared only privately and only when encrypted with a customer managed key whose key policy lets the other account use it; never with aws/ebs. Only unencrypted snapshots can be public. The recipient should copy the snapshot under its own key.
• EBS Snapshots Archive: for rarely used snapshots kept 90 days or more; up to 75% lower storage cost; an archived snapshot is converted to a full snapshot; 90-day minimum billing; restoring to the standard tier takes up to 72 hours (temporary restore 1–180 days, or permanent) before you can create volumes from it, copy it or share it.
• Recycle Bin: Region-level or tag-level retention rules (optional exclusion tags and rule lock) keep deleted snapshots and EBS-backed AMIs for 1–365 days; restore with restore-snapshot-from-recycle-bin / restore-image-from-recycle-bin. A rule must already exist when the deletion happens. ⚠️ Changed: Recycle Bin also protects deleted EBS volumes (retention 1–7 days, since 2025-11-20). Older material may say it covers snapshots and AMIs only.
• First-read latency: a volume restored from a snapshot pulls blocks lazily from S3. Fast Snapshot Restore (FSR) makes new volumes fully initialized at creation; it is enabled per snapshot per AZ, limited to 5 snapshots per Region, and billed per AZ per minute (1-hour minimum). Manual alternative: read every block with dd or fio. ⚠️ Changed: you can also request a Provisioned Rate for Volume Initialization of 100–300 MiB/s per volume (since 2025-05-06). Older material may say FSR and manual pre-warming are the only options.
• ⚠️ Changed: an instant, crash-consistent copy of a volume in the same AZ is possible with copy-volumes, which can also encrypt or re-key the copy (since 2025-10-14). Older material may say a snapshot is the only way to duplicate a volume. Moving to another AZ still requires a snapshot. The copy can be attached at once but initializes in the background; until then it delivers the lowest of 3,000 IOPS / 125 MiB/s and the provisioned performance of the source and of the copy (more only when both are provisioned higher and the source has unused capacity). Only one copy per source volume runs at a time, and at most 5 copies can be in progress per Region (CopyVolumesLimitExceeded).
• ⚠️ Changed: a volume can be shared with other accounts through AWS RAM (permission AWSRAMPermissionEBSVolumeCopyAccess); those accounts can view it and copy it in the same AZ, but never attach, modify or snapshot it. Only unencrypted volumes and volumes encrypted with a customer managed key (shared with the account) qualify; aws/ebs volumes can't be shared. Older material may say a snapshot is the only way to give another account a volume's data.
• Amazon Data Lifecycle Manager (DLM): tag-targeted policies that schedule snapshot creation, retention (by count or age), cross-Region copy, cross-account sharing, FSR and archiving; AMI policies create, deprecate and deregister EBS-backed AMIs. No extra charge; runs with a service role (AWSDataLifecycleManagerDefaultRole) and manages only what it created. AWS Backup is the cross-service alternative.

AMIs

• Region-scoped. An EBS-backed AMI references one or more snapshots plus a block device mapping and launch permissions. Sources: AWS public AMIs, AWS Marketplace, your own, or AMIs shared with you.
• create-image: by default EC2 shuts the instance down and reboots it so the snapshots are consistent; --no-reboot keeps it running but file-system integrity is not guaranteed. Every attached EBS volume is snapshotted; instance store contents are never captured (§7).
• copy-image runs in the destination Region with --source-region and returns a new AMI ID per Region. Launch permissions are not copied; user tags only with --copy-image-tags. You can encrypt or re-key the copy but never decrypt it. The first copy into a Region copies the snapshots in full. Update launch templates and Auto Scaling groups with the new ID.
• Sharing: modify-image-attribute --launch-permission with account IDs, organizations or OUs, one Region at a time (copy first for other Regions). AMIs backed by encrypted snapshots require a customer managed key that the target account may use (key policy + IAM policy in the target account); AMIs encrypted with aws/ebs cannot be shared. Launching needs only the launch permission plus key access; to let the other account copy the AMI, also grant read on the snapshots (createVolumePermission).
• Auto Scaling with customer managed keys: the key policy must allow the AWSServiceRoleForAutoScaling service-linked role. When the key lives in another account, the key owner must allow the Auto Scaling account kms:CreateGrant without the kms:GrantIsForAWSResource condition (that condition blocks direct grant calls), and the Auto Scaling account then runs aws kms create-grant for its own service-linked role, which must already exist.
• Retiring an AMI: deprecate (hidden from other accounts' listings, still launchable by ID; public AMIs are deprecated 2 years after creation by default) → disable (no new launches, made private) → deregister (deleted; running instances are unaffected). Deregistration protection blocks deletion, with an optional 24-hour cooldown after it is turned off. ⚠️ Changed: snapshots are still kept by default after deregistration, but deregister-image --delete-associated-snapshots removes them in the same call (since 2025-05-28). Older material may say the snapshots always have to be deleted separately.
• Golden AMI: an image with OS hardening, agents and the application pre-installed, giving faster and more predictable boots than long user-data scripts. Build and distribute it with EC2 Image Builder (pipeline, cost and the bake-vs-bootstrap trade-off: see §3; effect on scale-out: see §10).
• SSM public parameters always point to the latest AWS AMI in the current Region, for example /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 (or -arm64). Use resolve:ssm:<parameter> as the image ID in RunInstances or a launch template, or the CloudFormation parameter type AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>. Each AL2023 AMI is deprecated 90 days after release, so resolving dynamically avoids stale IDs. ⚠️ Changed: Amazon Linux 2 reached end of support on 2026-06-30, so use the AL2023 parameters. Older material may say to use /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2.

Developer angle

Encrypt an existing unencrypted volume (the classic snapshot path):

# 1) snapshot the unencrypted volume
SNAP=$(aws ec2 create-snapshot --volume-id vol-01234567890abcdef \
  --description "pre-encryption" --query SnapshotId --output text)
aws ec2 wait snapshot-completed --snapshot-ids "$SNAP"

# 2) restore it as an encrypted volume in the SAME AZ as the instance
NEW=$(aws ec2 create-volume --snapshot-id "$SNAP" --availability-zone us-east-1a \
  --volume-type gp3 --encrypted --kms-key-id alias/app-ebs \
  --query VolumeId --output text)
aws ec2 wait volume-available --volume-ids "$NEW"

# 3) swap it in (for a root volume: stop first and reuse the RootDeviceName)
aws ec2 describe-instances --instance-ids i-1234567890abcdef0 \
  --query "Reservations[].Instances[].RootDeviceName" --output text
aws ec2 stop-instances --instance-ids i-1234567890abcdef0
aws ec2 wait instance-stopped --instance-ids i-1234567890abcdef0
aws ec2 detach-volume --volume-id vol-01234567890abcdef
aws ec2 wait volume-available --volume-ids vol-01234567890abcdef
aws ec2 attach-volume --volume-id "$NEW" --instance-id i-1234567890abcdef0 --device /dev/xvda
aws ec2 start-instances --instance-ids i-1234567890abcdef0

Newer shortcuts. Both paths are valid today. The snapshot path is the classic answer, and questions written before October 2025 offer only that path; choose copy-volumes when it is listed and the copy stays in the same AZ.

# same-AZ encrypted copy: name the type (and match the source's performance), or the copy is gp2
aws ec2 copy-volumes --source-volume-id vol-01234567890abcdef \
  --volume-type gp3 --iops 6000 --throughput 250 \
  --encrypted --kms-key-id alias/app-ebs

# swap a prepared, detached volume from the same AZ in as the root:
# automatic reboot, same instance ID and IPs (add --delete-replaced-root-volume to drop the old root)
aws ec2 create-replace-root-volume-task --instance-id i-1234567890abcdef0 \
  --volume-id vol-0fedcba9876543210
aws ec2 describe-replace-root-volume-tasks \
  --filters Name=instance-id,Values=i-1234567890abcdef0 --query "ReplaceRootVolumeTasks[].TaskState"

# the old root's DeleteOnTermination does not carry over: set it on the new root
aws ec2 modify-instance-attribute --instance-id i-1234567890abcdef0 \
  --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"DeleteOnTermination":true}}]'

⚠️ Changed: create-replace-root-volume-task --volume-id swaps in an existing volume that is available (attached to nothing) and in the instance's AZ (since 2026-07-09). Older material may say root replacement can only restore the launch state, a snapshot of the root or an AMI. You can go from an unencrypted root to an encrypted one, but EC2 rejects an encrypted root → unencrypted volume swap.

Share an encrypted golden AMI from account 111122223333 with account 444455556666. Statements to add to the customer managed key's policy (keep the existing statements). The conditioned grant statement covers direct run-instances launches and copy-image, where EC2 creates the grant on the caller's behalf. Add the last statement only if an Auto Scaling group in the target account launches the shared AMI directly, because that account must call CreateGrant itself for its service-linked role:

{
  "Statement": [
    {
      "Sid": "AllowUseOfTheKeyByTargetAccount",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::444455556666:root"},
      "Action": ["kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey"],
      "Resource": "*"
    },
    {
      "Sid": "AllowGrantsForDirectLaunchesAndCopies",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::444455556666:root"},
      "Action": ["kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant"],
      "Resource": "*",
      "Condition": {"Bool": {"kms:GrantIsForAWSResource": true}}
    },
    {
      "Sid": "AllowTargetToGrantItsAutoScalingRole",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::444455556666:root"},
      "Action": "kms:CreateGrant",
      "Resource": "*"
    }
  ]
}

# owner: an aws/ebs-encrypted AMI must first be re-keyed with a customer managed key
SHARED=$(aws ec2 copy-image --region us-east-1 --source-region us-east-1 \
  --source-image-id ami-0abcdef1234567890 --name app-v42-cmk --encrypted \
  --kms-key-id arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab \
  --query ImageId --output text)
aws ec2 wait image-available --image-ids "$SHARED"

# owner: launch permission (plus snapshot read so the target can also copy the AMI)
aws ec2 modify-image-attribute --image-id "$SHARED" \
  --launch-permission "Add=[{UserId=444455556666}]"
for s in $(aws ec2 describe-images --image-ids "$SHARED" \
    --query "Images[].BlockDeviceMappings[].Ebs.SnapshotId" --output text); do
  aws ec2 modify-snapshot-attribute --snapshot-id "$s" \
    --attribute createVolumePermission --operation-type add --user-ids 444455556666
done

# target (needs an IAM policy allowing the same kms:* actions on the owner's key ARN):
# take ownership by copying under its own key
aws ec2 copy-image --region us-east-1 --source-region us-east-1 \
  --source-image-id "$SHARED" --name app-v42-local --encrypted --kms-key-id alias/team-ebs

# target, only if its Auto Scaling group launches the shared AMI directly (no local copy):
# grant the owner's key to its own service-linked role (IAM must allow kms:CreateGrant on that key ARN)
aws kms create-grant --region us-east-1 \
  --key-id arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab \
  --grantee-principal arn:aws:iam::444455556666:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling \
  --operations Encrypt Decrypt ReEncryptFrom ReEncryptTo GenerateDataKey GenerateDataKeyWithoutPlaintext DescribeKey CreateGrant

Cross-Region copy for DR with boto3 (call the API in the destination Region):

import boto3

ec2_dr = boto3.client("ec2", region_name="eu-west-1")
resp = ec2_dr.copy_image(
    SourceRegion="us-east-1",
    SourceImageId="ami-0abcdef1234567890",
    Name="web-v42-dr",
    Encrypted=True,
    KmsKeyId="alias/dr-ebs",   # a key that exists in eu-west-1
    CopyImageTags=True,
)
ec2_dr.get_waiter("image_available").wait(ImageIds=[resp["ImageId"]])
print(resp["ImageId"])          # new ID: update the DR launch template with it

Keep the root volume when the instance is terminated, then verify:

aws ec2 modify-instance-attribute --instance-id i-1234567890abcdef0 \
  --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"DeleteOnTermination":false}}]'
aws ec2 describe-instances --instance-ids i-1234567890abcdef0 \
  --query "Reservations[].Instances[].BlockDeviceMappings[].[DeviceName,Ebs.DeleteOnTermination]"

Grow and tune a gp3 volume live, then extend the file system:

aws ec2 modify-volume --volume-id vol-01234567890abcdef --size 200 --iops 12000 --throughput 500
aws ec2 describe-volumes-modifications --volume-ids vol-01234567890abcdef \
  --query "VolumesModifications[].[ModificationState,Progress]"
# on the instance (Nitro, XFS root partition 1)
sudo growpart /dev/nvme0n1 1 && sudo xfs_growfs -d /

Always launch the latest AL2023 AMI:

aws ssm get-parameter --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \
  --query Parameter.Value --output text
aws ec2 run-instances --instance-type t3.micro \
  --image-id resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64

Parameters:
  LatestAmiId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64
Resources:
  AppInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t3.micro
      ImageId: !Ref LatestAmiId

Protection and cost controls:

aws ec2 enable-ebs-encryption-by-default --region eu-west-1
aws ec2 modify-ebs-default-kms-key-id --kms-key-id alias/app-ebs --region eu-west-1
aws rbin create-rule --resource-type EBS_SNAPSHOT --description "prod snapshots 14 days" \
  --retention-period RetentionPeriodValue=14,RetentionPeriodUnit=DAYS \
  --resource-tags ResourceTagKey=env,ResourceTagValue=prod
aws ec2 modify-snapshot-tier --snapshot-id snap-0abcdef1234567890 --storage-tier archive
aws ec2 restore-snapshot-tier --snapshot-id snap-0abcdef1234567890 --temporary-restore-days 5
aws ec2 enable-fast-snapshot-restores --availability-zones us-east-1a us-east-1b \
  --source-snapshot-ids snap-0abcdef1234567890
aws ec2 deregister-image --image-id ami-0abcdef1234567890 --delete-associated-snapshots

Traps

• Trap: "Use modify-volume (or encryption by default) to encrypt the existing volume." → Reality: encryption can't be added in place, and encryption by default affects only new volumes and snapshot copies: snapshot → encrypted volume or encrypted snapshot copy → swap.
• Trap: "Edit the aws/ebs key policy so the other account can use the shared snapshot/AMI." → Reality: AWS managed key policies can't be changed, so aws/ebs resources can't be shared. Re-encrypt with a customer managed key, then share the key and the snapshot/AMI.
• Trap: "Sharing the AMI is enough for the other account's Auto Scaling group." → Reality: launches also need KMS access to the customer managed key (key policy + IAM). Across accounts, the key policy must allow that account an unconditioned kms:CreateGrant, and it must create a grant for its AWSServiceRoleForAutoScaling; otherwise instances terminate with Client.InternalError.
• Trap: "copy-volumes produces an identical volume." → Reality: without --volume-type (and --iops/--throughput) the copy is a gp2 volume; it also initializes in the background, and only one copy per source runs at a time.
• Trap: "After a root volume replacement, the new root is deleted with the instance like the old one." → Reality: when you swap in an existing volume, DeleteOnTermination does not carry over; set it with modify-instance-attribute, or the volume outlives the instance and keeps billing.
• Trap: "Anything above 16,000 IOPS needs io1/io2." → Reality: gp3 delivers up to 80,000 IOPS and 2,000 MiB/s at gp3 prices; io2 Block Express is for > 80,000 IOPS, > 2,000 MiB/s, sub-millisecond latency, 99.999% durability or Multi-Attach.
• Trap: "A small gp3 volume has few IOPS, like gp2." → Reality: every gp3 volume gets 3,000 IOPS and 125 MiB/s; extra IOPS are limited only by 500 IOPS per GiB and 80,000 overall.
• Trap: "After modify-volume the extra space shows up in df." → Reality: extend the partition and file system (growpart, xfs_growfs/resize2fs) once the modification is optimizing or completed.
• Trap: "Attach the volume to an instance in another AZ" or "launch from the AMI ID in another Region." → Reality: volumes are AZ-bound; snapshots and AMIs are Regional. Snapshot/copy first; a copied AMI has a new ID.
• Trap: "Deregistering the AMI stops all its storage charges." → Reality: the snapshots stay (and are billed) unless you pass --delete-associated-snapshots or delete them yourself.
• Trap: "A deprecated AMI can no longer be launched." → Reality: it is only hidden from listings; launch templates, Auto Scaling groups and callers that know the ID keep using it. Disable or deregister to stop launches.
• Trap: "Recycle Bin can bring back anything deleted last week." → Reality: only resources matching a retention rule that existed at deletion time, within its retention period.
• Trap: "Create a volume straight from an archived snapshot." → Reality: restore it to the standard tier first (up to 72 hours).
• Trap: "Multi-Attach a gp3 volume to instances in two AZs and format it ext4." → Reality: io1/io2 only, up to 16 Nitro instances in one AZ, with a cluster-aware file system.
• Trap: "Extra data volumes always survive termination." → Reality: true for volumes attached after launch; volumes defined in the launch block device mapping follow that mapping (the CLI default is delete). Check DeleteOnTermination per device.

Practice questions

1. A developer needs one 500 GiB EBS data volume for a self-managed database on a Nitro-based instance. The volume must sustain 40,000 IOPS and 1,000 MiB/s. Sub-millisecond latency and Multi-Attach are not required. Which volume configuration meets the requirements at the LOWEST cost?

   • A. A gp2 volume enlarged until its baseline reaches 40,000 IOPS
   • B. A gp3 volume provisioned with 40,000 IOPS and 1,000 MiB/s
   • C. An io2 Block Express volume provisioned with 40,000 IOPS
   • D. An st1 volume sized for 1,000 MiB/s of throughput

2. Account A wants an Auto Scaling group in account B to launch instances from a golden AMI. The AMI's snapshots are encrypted with the AWS managed key aws/ebs. Which TWO actions are required? (Select TWO.)

   • A. In account A, copy the AMI and encrypt the copy with a customer managed KMS key.
   • B. Add account B to the key policy of the aws/ebs key in account A.
   • C. Allow account B to use the customer managed key and add a launch permission for account B on the new AMI.
   • D. Make the AMI public so that account B can launch it.
   • E. Turn on EBS encryption by default in account B.

7. Choosing storage: EBS vs EFS vs instance store

Concept

• EC2 offers three storage attachments with different contracts. Instance store is local disk on the physical host, as fast as it gets but tied to the instance's current host. EBS is a durable network block volume in one AZ, normally owned by one instance. EFS is an elastic NFS file system that many clients in many AZs share. Objects (files addressed by key through an API) belong in S3.
• Decide with five questions: must the data survive stop/terminate? Is it shared by several instances or services? Does it have to span AZs? How many IOPS / how much latency matter? Linux or Windows clients?
• Cost models differ: instance store is included in the instance price, EBS bills provisioned capacity (and performance), EFS bills what you store plus throughput depending on mode, with cheaper storage classes for cold files.
• Developer touchpoints: mounting EFS during bootstrap (§3) with TLS and IAM, opening NFS port 2049 between security groups (§4), giving Lambda or containers a shared file system through access points, and choosing EFS modes that don't throttle the application.

Exam facts

Instance store

• Disks physically attached to the host; no extra charge. Count and size are fixed by the instance type: look for a d in the type name (for example m5d) or storage-optimized families (I, D); many types have none. Example: i3.16xlarge has 8 × 1,900 GB NVMe SSD with about 3.3 million random-read IOPS.
• Attached only at launch. On NVMe instance store types (m5d, c6id, i3, i4i and other current families) every volume is attached and enumerated automatically, with no block device mapping needed; older non-NVMe types (C1, C3, M1, M2, M3, R3, D2, H1, I2, X1, X1e) need ephemeralN mappings in the launch request or the AMI. Instance store can never be added after launch, detached or moved to another instance, and it has no EBS snapshots. An EBS-backed AMI does not capture instance store data, and new instances get blank instance store volumes.
• Data survives a reboot (from the API/console or from inside the OS). It is lost (every block cryptographically erased) on stop, hibernate or terminate (including a shutdown started inside the OS), an instance-type change, a scheduled retirement, automatic recovery, or failure of the underlying disk.
• NVMe instance store is always encrypted at rest with hardware XTS-AES-256; keys are destroyed at stop/terminate and you cannot supply your own.
• Volumes arrive unformatted, and again after every stop/start: create the file system and mount it at boot, then rebuild or rehydrate the data (for example from S3).
• Fits caches, buffers, scratch/shuffle space and data replicated elsewhere (for example nodes of a distributed database).

EBS (details in §6)

• Persistent block storage in one AZ, normally attached to a single instance (io1/io2 Multi-Attach is the exception), backed up with snapshots. The natural choice for boot volumes and single-instance databases.

EFS

• Managed NFS (NFSv4.0 and v4.1) with POSIX permissions and file locking. Linux clients only (Windows is not supported). Mountable concurrently by EC2, ECS, EKS, Fargate, Lambda and on-premises servers over Direct Connect/VPN. Grows and shrinks automatically up to petabytes; you pay for what is stored, with no capacity to provision.
• File system types: Regional (data stored across ≥ 3 AZs; recommended) or One Zone (one AZ, lower cost, automatic backups on by default, no Max I/O; access from other AZs incurs data charges). ⚠️ Changed: the multi-AZ type is now called Regional, and "Standard" names the SSD storage class. Older material may say EFS Standard (and One Zone-IA) for the availability options.
• Mount targets: one per AZ, placed in one subnet of that AZ; each has an IP address and its own security groups. The file system DNS name resolves to the mount target in the client's AZ. Mount targets exist in only one VPC at a time.
• Network rule: the mount target's security group allows inbound TCP 2049 from the clients' security group; clients need outbound 2049. Read a failed mount by its message (§4):
  • Hangs for 1–2 minutes, then mount.nfs: Connection timed out → the mount target's security group has no inbound TCP 2049 from the client's group (or a network ACL blocks it, or the mount-target IP is wrong).
  • Failed to resolve server <fs-id>.efs.<region>.amazonaws.com → no mount target in the client's AZ, the client is in another VPC, or VPC DNS resolution/DNS hostnames are off. DNS records can also take up to 90 seconds to appear after a mount target is created, so scripted mounts should wait.
  • access denied by server → the identity policy or the file system policy denies the client (or nothing allows it); with IAM authorization, mount with -o tls,iam.
• Storage classes: Standard (SSD, lowest latency), Infrequent Access (a few accesses per quarter, tens of ms, per-GB read charges) and Archive (a few accesses per year; 90-day minimum storage; requires Elastic throughput; Regional only). IA and Archive bill at least 128 KiB per file; metadata always stays in Standard. ⚠️ Changed: there are three classes, and files smaller than 128 KiB are also tiered (billed as 128 KiB) for lifecycle policies set since 2023-11-26. Older material may say there are only Standard and IA classes and that small files never leave Standard.
• Lifecycle management: TransitionToIA, TransitionToArchive and TransitionToPrimaryStorageClass (AFTER_1_ACCESS). Console default: IA after 30 days and Archive after 90 days without access, no automatic move back. Listing a directory is not an access.
• Performance mode (set at creation, cannot be changed): General Purpose (default, lowest latency; watch PercentIOLimit) or Max I/O. ⚠️ Changed: Max I/O is a previous-generation mode, not supported with Elastic throughput or One Zone, and AWS recommends General Purpose for all file systems. Older material may say to choose Max I/O for highly parallel big-data or media workloads.
• Throughput mode (can be switched later):
  • Elastic (recommended): scales automatically and bills per GB read/written; best for spiky or unpredictable loads (average-to-peak ratio ≤ 5%).
  • Provisioned: a fixed MiB/s independent of stored size; for steady, known demand. After switching to Provisioned or changing the provisioned amount (up or down), for 24 hours you can't switch to Elastic or Bursting or decrease the amount.
  • Bursting: baseline 50 KiB/s per GiB in Standard (= 50 MiB/s per TiB); bursts to 100 MiB/s per TiB (at least 100 MiB/s) while credits last; watch BurstCreditBalance. Move to Elastic or Provisioned when throughput-bound.
• ⚠️ Changed: Elastic throughput reaches 20–60 GiB/s read and 1–5 GiB/s write per file system (Region-dependent), and up to 1,500 MiB/s per client with amazon-efs-utils 2.0 or later (500 MiB/s otherwise). Older material may say 3 GiB/s read and 1 GiB/s write.
• ⚠️ Changed: the console creates file systems with Elastic throughput, General Purpose mode and encryption at rest; the CreateFileSystem API/CLI still defaults to bursting and leaves encryption off unless you request it. Older material may say Bursting is the default everywhere.
• Encryption at rest: KMS (default aws/elasticfilesystem or a customer managed key), chosen only at creation. To encrypt an existing file system, create a new encrypted one and replicate/copy the data. Enforce with the IAM condition key elasticfilesystem:Encrypted on CreateFileSystem.
• Encryption in transit: mount helper option -o tls (TLS tunnel to port 2049). Enforce with a file system policy that denies requests where aws:SecureTransport is false.
• IAM for NFS clients: mount with -o tls,iam so the instance role (§1) is evaluated; client actions are elasticfilesystem:ClientMount (read), ClientWrite and ClientRootAccess. With no file system policy, any client that can reach a mount target gets full access.
• Access points: application entry points that force a POSIX UID/GID and a root directory (auto-created with the owner and permissions you set); restrict with the condition key elasticfilesystem:AccessPointArn. Lambda mounts EFS only through an access point, from a function attached to the VPC (subnets in AZs that have mount targets), at a path under /mnt/, with ClientMount/ClientWrite on the execution role (managed policy AmazonElasticFileSystemClientReadWriteAccess) plus VPC access permissions.

Other storage (awareness)

• FSx for Windows File Server: managed SMB shares with NTFS and Active Directory; the answer when Windows instances need a shared file system.
• FSx for Lustre: high-performance parallel file system for HPC/ML that can present an S3 bucket as files (scratch or persistent deployments).
• S3: object storage used through the API/SDK, not a block device; best for static assets, user uploads stored as objects (for example sent straight from clients with presigned URLs), backups and data lakes. ⚠️ Changed: Amazon S3 Files (built on EFS) can mount a bucket as an NFS 4.1/4.2 file system with POSIX permissions and file locking from EC2, Lambda, ECS and EKS (awareness only). Older material may say S3 can't be mounted as a POSIX file system with locking.

Decision table

Developer angle

Create an EFS file system that follows current guidance, add a mount target per AZ and open NFS only from the app tier:

FS=$(aws efs create-file-system --creation-token web-shared-01 \
  --performance-mode generalPurpose --throughput-mode elastic --encrypted \
  --tags Key=Name,Value=web-shared --query FileSystemId --output text)
# wait until LifeCycleState is "available"
aws efs describe-file-systems --file-system-id "$FS" --query "FileSystems[0].LifeCycleState"

# one mount target per AZ (repeat with a subnet in each AZ)
aws efs create-mount-target --file-system-id "$FS" \
  --subnet-id subnet-0123456789abcdef0 --security-groups sg-0123456789abcdef0

# mount-target SG: allow TCP 2049 only from the web servers' SG
aws ec2 authorize-security-group-ingress --group-id sg-0123456789abcdef0 \
  --protocol tcp --port 2049 --source-group sg-0fedcba9876543210

# lifecycle: IA after 30 days, Archive after 90, back to Standard on first access
aws efs put-lifecycle-configuration --file-system-id "$FS" \
  --lifecycle-policies '[{"TransitionToIA":"AFTER_30_DAYS"},{"TransitionToArchive":"AFTER_90_DAYS"},{"TransitionToPrimaryStorageClass":"AFTER_1_ACCESS"}]'

Mount with TLS and IAM authorization (for example from user data, §3), and persist it across reboots:

sudo dnf install -y amazon-efs-utils
sudo mkdir -p /mnt/shared
sudo mount -t efs -o tls,iam fs-0123456789abcdef0:/ /mnt/shared
echo 'fs-0123456789abcdef0:/ /mnt/shared efs _netdev,tls,iam 0 0' | sudo tee -a /etc/fstab

File system policy that rejects unencrypted connections and gives the app role read/write (aws efs put-file-system-policy --file-system-id fs-0123456789abcdef0 --policy file://fs-policy.json):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnencryptedTransport",
      "Effect": "Deny",
      "Principal": {"AWS": "*"},
      "Action": "elasticfilesystem:Client*",
      "Resource": "arn:aws:elasticfilesystem:us-east-1:111122223333:file-system/fs-0123456789abcdef0",
      "Condition": {"Bool": {"aws:SecureTransport": "false"}}
    },
    {
      "Sid": "AppRoleReadWrite",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::111122223333:role/web-app-ec2-role"},
      "Action": ["elasticfilesystem:ClientMount", "elasticfilesystem:ClientWrite"],
      "Resource": "arn:aws:elasticfilesystem:us-east-1:111122223333:file-system/fs-0123456789abcdef0"
    }
  ]
}

Give a VPC-attached Lambda function a shared directory through an access point:

AP_ARN=$(aws efs create-access-point --file-system-id fs-0123456789abcdef0 \
  --posix-user Uid=1001,Gid=1001 \
  --root-directory 'Path=/thumbnails,CreationInfo={OwnerUid=1001,OwnerGid=1001,Permissions=750}' \
  --query AccessPointArn --output text)
aws lambda update-function-configuration --function-name thumbnailer \
  --file-system-configs Arn="$AP_ARN",LocalMountPath=/mnt/thumbnails

Use instance store at boot (it is blank after every stop/start, so this belongs in a per-boot step, §3):

lsblk                                   # find the unformatted NVMe device, e.g. nvme1n1
sudo mkfs -t xfs /dev/nvme1n1
sudo mkdir -p /scratch && sudo mount /dev/nvme1n1 /scratch

Traps

• Trap: "Attach one EBS volume to all web servers across AZs to share files." → Reality: EBS is AZ-bound and Multi-Attach is io1/io2-only, same-AZ and needs a cluster-aware file system; use EFS with a mount target in every AZ.
• Trap: "Instance store data survives a stop/start because the instance keeps its ID." → Reality: only a reboot preserves it; stop, hibernate, terminate, type change, retirement and recovery wipe it.
• Trap: "Snapshot the instance store volume or bake it into an AMI." → Reality: neither is possible; copy the data to S3, EBS or EFS yourself.
• Trap: "Pick Max I/O for the best EFS performance." → Reality: Max I/O is previous-generation, has higher latency and doesn't work with Elastic throughput; use General Purpose + Elastic.
• Trap: "Spiky, unpredictable EFS traffic → Provisioned throughput sized for the peak." → Reality: Elastic throughput is the recommended, pay-per-use fit; Provisioned suits steady, known demand (and has a 24-hour change restriction).
• Trap: "EFS created with the CLI/SDK/IaC gets the same defaults as the console." → Reality: the API defaults to Bursting and no encryption at rest; set --throughput-mode elastic --encrypted explicitly.
• Trap: "Turn on encryption at rest for an existing EFS file system." → Reality: only possible at creation; create an encrypted file system and replicate/copy the data.
• Trap: "The EFS mount times out, so the IAM policy must be wrong." → Reality: a hang that ends in Connection timed out is the network: TCP 2049 blocked between the security groups (or a wrong mount-target IP). A missing mount target in the client's AZ shows up as Failed to resolve server instead, and IAM or file system policy problems fail fast with access denied by server.
• Trap: "Add ephemeral0 block device mappings, or the instance store disks won't appear." → Reality: NVMe instance store is attached automatically at launch; only old non-NVMe types need mappings. Either way, instance store can't be added after launch.
• Trap: "Use EFS for Windows file shares." → Reality: EFS is NFS/Linux-only; Windows needs FSx for Windows File Server (SMB).
• Trap: "Lambda can mount EFS from any function by file system ID." → Reality: it needs an access point, VPC configuration with reachable mount targets, a /mnt/... path and ClientMount/ClientWrite permissions.

Practice questions

1. A developer keeps a rebuildable cache on the NVMe instance store of an EBS-backed m6id.xlarge instance that was launched with hibernation enabled. During maintenance, which operation keeps the cached data on the instance store volumes?

   • A. Changing the instance type to m6id.2xlarge
   • B. Running aws ec2 reboot-instances for the instance
   • C. Hibernating the instance overnight and resuming it in the morning
   • D. Running sudo shutdown -h now inside the OS, then starting the instance again

2. A team is creating an Amazon EFS file system for Linux web servers that run in three Availability Zones. Traffic is spiky and hard to forecast, and file operations must stay low latency. Which TWO choices follow current AWS guidance? (Select TWO.)

   • A. Create a mount target in each Availability Zone and allow inbound TCP 2049 from the web servers' security group.
   • B. Choose the Max I/O performance mode to get the highest throughput.
   • C. Choose the Elastic throughput mode.
   • D. Choose a One Zone file system so that the data is stored in all three Availability Zones.
   • E. Choose the Bursting throughput mode so that throughput grows with the amount of stored data.

8. Elastic Load Balancing: ALB, NLB and GWLB

Concept

• Elastic Load Balancing (ELB) is a managed, Regional load balancer: AWS runs, scales, patches and replaces the load balancer nodes in every Availability Zone you enable. You model traffic as listener → rules → target group → targets, and every target group has its own health check.
• Clients get one stable DNS name and requests go only to healthy targets in several AZs; an Auto Scaling group registers and deregisters its instances in the target group automatically (§10).
• Choose by OSI layer: ALB (layer 7) understands HTTP and routes on request content, including to Lambda; NLB (layer 4) forwards TCP/UDP/TLS/QUIC flows at very high speed with static IPs; GWLB (layer 3) passes traffic transparently through third-party inspection appliances. CLB is the legacy previous generation.
• Developer impact: your app exposes a health endpoint, reads client details from X-Forwarded-* headers, accepts traffic only from the load balancer's security group (§4), and you troubleshoot with load-balancer-vs-target error metrics and access logs.

Exam facts

Health checks (per target group)

• ALB defaults: HTTP on the traffic port, path /, interval 30 s, timeout 5 s, healthy threshold 5 (consecutive successes that bring an unhealthy target back), unhealthy threshold 2, success code 200. Ranges: interval 5–300 s, timeout 2–120 s, thresholds 2–10, success codes 200–499 (gRPC 0–99, default 12). Lambda target groups: health checks off by default (35 s interval / 30 s timeout when enabled).
• NLB defaults: TCP (HTTP or HTTPS optional), interval 30 s, timeout 10 s for TCP/HTTPS and 6 s for HTTP, thresholds 5/2, HTTP success codes 200-399 (range 200–599).
• A newly registered ALB target needs to pass only one health check, not the healthy threshold, before it gets traffic; an NLB starts routing to a new target once registration completes and the initial health checks pass, which can take a few minutes. Target states: initial, healthy, unhealthy, unused, draining, unavailable (health checks disabled). If every target is unhealthy, ALB and NLB fail open and route to all of them.
• Health check requests use User-Agent: ELB-HealthChecker/2.0 and are excluded from ALB metrics.

Security groups

• Standard chain: load balancer SG allows 80/443 from clients (e.g. 0.0.0.0/0); instance SG allows the app port and health check port only from the load balancer's SG ID. The load balancer SG must also allow outbound traffic to the targets on those ports.
• ⚠️ Changed: NLBs support security groups (since August 2023), but only if at least one is attached at creation; an NLB created without security groups can never get one. Older material may say NLBs have no security groups. NLB health checks are subject to the NLB SG outbound rules, not its inbound rules. Trade-off: QUIC and TCP_QUIC listeners need an NLB with no security groups, so choose at creation.

ALB (layer 7)

• Front end: HTTP, HTTPS, HTTP/2 (HTTPS listeners only), WebSockets, gRPC (HTTPS listener + target group protocol version GRPC). Needs subnets in at least 2 AZs. The DNS name is stable but node IPs are service-managed and change as the ALB scales, so always resolve the name (Route 53 alias); you cannot attach your own Elastic IPs to an ALB.
• ⚠️ Changed: an internet-facing IPv4 ALB can take its public IPv4 addresses from an Amazon VPC IPAM pool, such as your BYOIP range (aws elbv2 modify-ip-pools --ipam-pools Ipv4IpamPoolId=...), so partners can allow-list that CIDR. Not supported for internal ALBs, IPv6 pools or dualstack-without-public-ipv4; when the pool runs out the ALB falls back to AWS-managed IPs. Older material may say an ALB's public IPs are always AWS-chosen. For exact, fixed per-AZ IPs still use an NLB (optionally with the ALB as an alb target).
• Listener rules run in priority order (lowest number first); the default rule runs last. Conditions: host-header, path-pattern, http-header, http-request-method, query-string, source-ip (CIDR of the connecting client, never the X-Forwarded-For value). Max 5 match evaluations per rule. Path matching is case-sensitive; host, header and query-string matching is not.
• ⚠️ Changed: host-header, path-pattern and http-header conditions also accept regular expressions (RegexValues). Older material may say only * and ? wildcards.
• Actions: forward (one or several weighted target groups), redirect (HTTP_301 or HTTP_302, e.g. HTTP→HTTPS; HTTPS→HTTP is not allowed), fixed-response (2XX/4XX/5XX with optional body), authenticate-cognito / authenticate-oidc (HTTPS listeners only; Cognito is covered in the Advanced Identity module), jwt-validation (verifies JWT access tokens).
• Weighted forward: up to 5 target groups per action, weight 0–999 each → blue/green or canary shifting inside the load balancer. There is no automatic failover to the other group when one has no healthy targets.
• Target types (fixed at creation): instance; ip (private only: the VPC's subnets, RFC 1918, 100.64.0.0/10 → peered VPCs and on-premises servers over VPN/Direct Connect; never public IPs); lambda (one function per target group, same account and Region, request body and response JSON max 1 MB each).
• Headers for the target: X-Forwarded-For (client IP; processing mode append by default, or preserve/remove), X-Forwarded-Proto (http/https used by the client), X-Forwarded-Port, and X-Amzn-Trace-Id (Root=1-<8 hex epoch>-<24 hex id>, added or updated for request tracing and written to access logs). Header limits: request line 16 KB, single header 16 KB, all request headers 64 KB.
• Connection idle timeout 60 s by default (1–4000 s). Routing algorithms: round_robin (default), least_outstanding_requests, weighted_random (optional anomaly mitigation with automatic target weights). Slow start: 30–900 s, default 0 = off; not combinable with least outstanding requests or weighted random.
• Quotas: 100 rules per ALB (excluding default rules), 50 listeners, 1,000 targets per ALB, 5 target groups per action.

NLB (layer 4)

• ⚠️ Changed: listener protocols are TCP, TLS, UDP, TCP_UDP, QUIC and TCP_QUIC (QUIC passthrough since November 2025). Older material may list only TCP, TLS and UDP.
• QUIC limits: QUIC and TCP_QUIC listeners require an ipv4 NLB without security groups; for QUIC the deregistration delay is fixed at 300 s and Proxy Protocol v2 is not supported.
• One static IP per enabled AZ; an internet-facing NLB can use one Elastic IP per AZ, chosen at creation → partners can allow-list fixed IPs. Built for very high throughput (millions of requests per second) with very low latency.
• Targets: instance, ip (same private ranges as ALB), alb (an ALB behind an NLB gets static IPs; requires a TCP listener). No Lambda targets.
• Client IP preservation: on for instance targets; for ip targets off by default with TCP/TLS (turn on preserve_client_ip.enabled or use Proxy Protocol v2) and always on for UDP, TCP_UDP, QUIC and TCP_QUIC. The NLB never parses HTTP, so it adds no X-Forwarded-For header.
• A TLS listener terminates TLS on the NLB (certificate + security policy, §9); a TCP listener on 443 passes the encrypted stream through untouched.
• ⚠️ Changed: the TCP idle timeout is configurable 60–6000 s (default still 350 s, since September 2024). Older material may say it is fixed at 350 s.

GWLB (layer 3)

• Inserts a fleet of third-party inspection appliances (firewalls, IDS/IPS, deep packet inspection) transparently into the traffic path, a "bump in the wire": route tables send traffic to a GWLB endpoint, the GWLB spreads flows across healthy appliances, and inspected traffic continues to its original destination.
• Listens for all IP packets on all ports; talks to appliances with GENEVE on port 6081; targets instance or ip; flow stickiness 5-tuple (default), 3-tuple or 2-tuple; MTU 8,500 bytes.

CLB (legacy): HTTP, HTTPS, TCP and SSL listeners; instances registered directly (no target groups); one certificate per listener (no SNI, §9). Use ALB or NLB for new work.

Cross-zone load balancing

• ALB: always on at the load balancer level (can be switched off per target group); no inter-AZ data charge.
• NLB and GWLB: off by default; inter-AZ data transfer is charged once you turn it on.
• ⚠️ Changed: a CLB's default depends on how you create it: off through the API/CLI, on through the console; never an inter-AZ charge. Older material may say "CLB: disabled by default".

Deregistration delay (connection draining)

• Target group attribute deregistration_delay.timeout_seconds: default 300 s, range 0–3600 s (0 disables). The target enters draining: no new requests, in-flight requests may finish. On an ALB the value is an upper bound for that draining: a target with no in-flight requests and no active connections finishes deregistering at once, yet it still shows draining for the full delay and only then turns unused. An NLB turns the target unused after the full delay; existing non-idle connections can keep carrying traffic unless deregistration_delay.connection_termination.enabled closes them at the end of the delay. The CLB name is connection draining (timeout 1–3600 s, default 300 s).
• Auto Scaling waits for draining before it terminates an instance (§10). The full delay applies whenever requests or connections are still open (long requests, long-lived connections) and to any step that waits for the target to leave draining (per the ALB docs, an Auto Scaling instance can be terminated and replaced once its target is unused). Short requests → lower the delay (e.g. 30 s) to speed up deployments; set it too low and long requests are cut off (502).

Error codes generated by the ALB (counted in HTTPCode_ELB_4XX_Count / HTTPCode_ELB_5XX_Count)

• 403: AWS WAF blocked the request. 408: the client sent no data before the idle timeout. 413: Lambda request body over 1 MB. 460: the client closed the connection before the load balancer answered (client timeout shorter than the idle timeout). 463: X-Forwarded-For holds more than 30 IP addresses. 464: request protocol incompatible with the target group protocol version.
• 500: AWS WAF execution error, or the IdP of an authenticate action is unreachable, slower than 5 s, or returns claims over 11 KB.
• 502 Bad Gateway: the target reset or closed the connection (classic cause: its keep-alive timeout is shorter than the ALB idle timeout), a malformed response or invalid headers, a TLS handshake error with the target, a Lambda function error, throttle, function timeout, invalid response or response over 1 MB, or the deregistration delay expired mid-request.
• 503 Service Unavailable: the target group has no registered targets, or all of them are unused (e.g. registered in an AZ that is not enabled).
• 504 Gateway Timeout: no TCP connection to the target within 10 s (security group, NACL ephemeral ports, overloaded target) or no response before the idle timeout; or the Lambda service did not respond before the connection timeout.
• Status codes your application returns are counted separately in HTTPCode_Target_2XX/3XX/4XX/5XX_Count.

Monitoring and logs

• AWS/ApplicationELB, 60-second data, published only while traffic flows: RequestCount (Sum), TargetResponseTime (Average or pNN; seconds until the target starts sending response headers), HealthyHostCount / UnHealthyHostCount (per target group; alarm on Minimum UnHealthyHostCount > 0), HTTPCode_ELB_5XX_Count vs HTTPCode_Target_5XX_Count, RequestCountPerTarget (statistic Sum, which is already the per-target average), TargetConnectionErrorCount, RejectedConnectionCount, LambdaUserError. NLB metrics are in AWS/NetworkELB (ActiveFlowCount, NewFlowCount, TCP_Target_Reset_Count, HealthyHostCount).
• Access logs are off by default: compressed files per load balancer node every 5 minutes to an S3 bucket in the same Region (SSE-S3 only, best effort, you pay S3 storage only). Fields include client:port, target:port, request/target/response processing times, elb_status_code vs target_status_code, trace_id and error_reason.
• ⚠️ Changed: the recommended bucket policy grants s3:PutObject to the service principal logdelivery.elasticloadbalancing.amazonaws.com, and ALB logs can also be delivered as CloudWatch vended logs (CloudWatch Logs, Data Firehose or S3), alongside connection logs and health check logs. Older material may show a Region-specific ELB account ID as the principal and S3 as the only destination.

Developer angle

Target group with an application health endpoint, then content-based rules on the HTTPS listener:

aws elbv2 create-target-group --name orders-tg --protocol HTTP --port 8080 \
  --vpc-id vpc-0abc1234def567890 --target-type instance \
  --health-check-path /health --health-check-interval-seconds 15 \
  --healthy-threshold-count 3 --unhealthy-threshold-count 2 --matcher HttpCode=200

# /orders/* -> orders-tg (priority 10 is evaluated before higher numbers)
aws elbv2 create-rule --listener-arn "$LISTENER_ARN" --priority 10 \
  --conditions Field=path-pattern,Values='/orders/*' \
  --actions Type=forward,TargetGroupArn="$ORDERS_TG_ARN"

# host + header condition, canary split 90/10 between two target groups
aws elbv2 create-rule --listener-arn "$LISTENER_ARN" --priority 20 \
  --conditions file://api-conditions.json --actions file://canary-forward.json

api-conditions.json and canary-forward.json:

[
  { "Field": "host-header", "HostHeaderConfig": { "Values": ["api.example.com"] } },
  { "Field": "http-header", "HttpHeaderConfig": { "HttpHeaderName": "X-Client", "Values": ["mobile*"] } }
]

[
  {
    "Type": "forward",
    "ForwardConfig": {
      "TargetGroups": [
        { "TargetGroupArn": "arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/api-blue/0123456789abcdef", "Weight": 90 },
        { "TargetGroupArn": "arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/api-green/fedcba9876543210", "Weight": 10 }
      ]
    }
  }
]

Lambda behind an ALB: the CLI does not add the invoke permission for you (the console does).

TG_ARN=$(aws elbv2 create-target-group --name thumbs-tg --target-type lambda \
  --query 'TargetGroups[0].TargetGroupArn' --output text)
aws lambda add-permission --function-name thumbs:live --statement-id alb-invoke \
  --principal elasticloadbalancing.amazonaws.com --action lambda:InvokeFunction \
  --source-arn "$TG_ARN"
aws elbv2 register-targets --target-group-arn "$TG_ARN" \
  --targets Id=arn:aws:lambda:eu-west-1:111122223333:function:thumbs:live

The function receives the request as a JSON event (httpMethod, path, queryStringParameters, headers, body, isBase64Encoded) and must return:

{ "statusCode": 200, "statusDescription": "200 OK", "isBase64Encoded": false,
  "headers": { "Content-Type": "application/json" }, "body": "{\"ok\": true}" }

Reading client details behind an ALB (Flask example):

from flask import request

xff = request.headers.get("X-Forwarded-For", "")      # "203.0.113.7" or "spoofed-value, 203.0.113.7"
peer_ip = xff.split(",")[-1].strip()                   # last entry = the address the ALB actually saw
scheme = request.headers.get("X-Forwarded-Proto")      # "https" when the client used TLS
trace = request.headers.get("X-Amzn-Trace-Id")         # correlate with access logs / X-Ray

Static IPs for partners, deployments and troubleshooting:

# NLB with one Elastic IP per AZ; security groups only possible if attached now
# (omit --security-groups if you will need QUIC or TCP_QUIC listeners)
aws elbv2 create-load-balancer --name partner-nlb --type network --scheme internet-facing \
  --subnet-mappings SubnetId=subnet-0aaa1111bbbb2222c,AllocationId=eipalloc-0aaa1111bbbb2222c \
                    SubnetId=subnet-0ddd3333eeee4444f,AllocationId=eipalloc-0ddd3333eeee4444f \
  --security-groups sg-0123456789abcdef0

# Faster deployments: drain for 30 s instead of 300 s
aws elbv2 modify-target-group-attributes --target-group-arn "$ORDERS_TG_ARN" \
  --attributes Key=deregistration_delay.timeout_seconds,Value=30

# Why is a target out of service? (Target.ResponseCodeMismatch, Target.Timeout, ...)
aws elbv2 describe-target-health --target-group-arn "$ORDERS_TG_ARN" \
  --query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State,TargetHealth.Reason]'

# Access logs to S3 (bucket in the same Region, SSE-S3, bucket policy below)
aws elbv2 modify-load-balancer-attributes --load-balancer-arn "$ALB_ARN" \
  --attributes Key=access_logs.s3.enabled,Value=true Key=access_logs.s3.bucket,Value=my-alb-logs

# Alarm on errors generated by the load balancer itself
aws cloudwatch put-metric-alarm --alarm-name orders-alb-elb-5xx \
  --namespace AWS/ApplicationELB --metric-name HTTPCode_ELB_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/orders-alb/50dc6c495c0c9188 \
  --statistic Sum --period 60 --evaluation-periods 3 --threshold 10 \
  --comparison-operator GreaterThanThreshold --treat-missing-data notBreaching

Access-log bucket policy (current form):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "logdelivery.elasticloadbalancing.amazonaws.com" },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-alb-logs/AWSLogs/111122223333/*"
    }
  ]
}

Traps

• Trap: "Give partners the ALB's IP addresses or attach Elastic IPs to it." → Reality: ALB addresses are service-managed and change, and you cannot attach your own Elastic IPs; use an NLB (static IP or Elastic IP per AZ), optionally with the ALB registered behind it as an alb target. An IPAM pool on an internet-facing ALB only keeps its IPs inside a CIDR you own; it does not give fixed per-AZ addresses.
• Trap: "ALB 503 means every target failed its health check." → Reality: with all targets unhealthy the ALB fails open; 503 means the target group has no registered (or usable) targets.
• Trap: "HTTPCode_ELB_5XX_Count is rising, so the application code is throwing errors." → Reality: ELB 5XX codes (502/503/504) are generated by the load balancer: connection resets, timeouts, missing targets. Application errors show up in HTTPCode_Target_5XX_Count.
• Trap: "Intermittent 502s: raise the deregistration delay." → Reality: the usual cause is a target keep-alive timeout shorter than the ALB idle timeout (or malformed responses); make the app's keep-alive longer than the 60 s idle timeout.
• Trap: "504: the target returned an error." → Reality: 504 means no answer in time: no connection within 10 s (SG/NACL) or no response before the idle timeout; speed up the target or raise the idle timeout.
• Trap: "Log the TCP source address to get the user's IP behind an ALB." → Reality: that is the ALB node's private IP; read X-Forwarded-For. Behind an NLB with instance targets, the source address already is the client.
• Trap: "A source-ip rule condition matches the X-Forwarded-For value." → Reality: it matches the connection's source address; use an http-header condition on X-Forwarded-For.
• Trap: "Add a security group to an existing NLB later." → Reality: only possible if the NLB was created with at least one security group, and an NLB that has security groups cannot use QUIC or TCP_QUIC listeners.
• Trap: "A Lambda target registered with the CLI just works." → Reality: add a resource-based permission (lambda add-permission, principal elasticloadbalancing.amazonaws.com); without it requests fail (LambdaAccessDenied, counted in LambdaUserError).
• Trap: "Cross-zone load balancing is free everywhere." → Reality: free on ALB and CLB; NLB and GWLB charge inter-AZ data when it is on (and it is off by default).
• Trap: "The health check path can be the app's home page." → Reality: if / redirects to a login page (302), the default ALB matcher 200 marks every target unhealthy; expose a dedicated /health that returns 200 or widen the matcher.
• Trap: "Use a GWLB to filter HTTP requests by path or SQL injection." → Reality: GWLB is a transparent layer-3 hop for third-party appliances (GENEVE 6081); HTTP-level filtering belongs to AWS WAF on the ALB.

Practice questions

1. A company exposes a TCP-based pricing API to partner companies. Each partner must allow-list a fixed set of destination IP addresses in its firewall, the API must absorb sudden spikes of millions of requests per second with minimal latency, and the application must log each partner's real source IP address. Which solution meets these requirements?

   • A. An Application Load Balancer with an Elastic IP address in each subnet; the application reads X-Forwarded-For.
   • B. A Network Load Balancer with an Elastic IP address per Availability Zone and instance-type targets.
   • C. A Gateway Load Balancer with the API instances registered as appliances behind a GWLB endpoint.
   • D. A Classic Load Balancer with cross-zone load balancing and proxy protocol enabled.

2. An application behind an Application Load Balancer returns intermittent HTTP 502 errors. CloudWatch shows HTTPCode_ELB_5XX_Count increasing while HTTPCode_Target_5XX_Count stays at zero. Which TWO causes are consistent with these symptoms? (Select TWO.)

   • A. The web server's keep-alive timeout is shorter than the load balancer's idle timeout, so it closes connections the ALB is still reusing.
   • B. The target group currently has no registered targets.
   • C. The application sometimes writes malformed HTTP response headers.
   • D. An AWS WAF web ACL attached to the load balancer blocks some requests.
   • E. Clients disconnect before the load balancer sends the response.

9. Sticky sessions, TLS certificates and SNI

Concept

• Sticky sessions (session affinity) pin a client to one target so session data kept in that server's memory keeps working. It is a workaround: load becomes uneven and the session is lost when that target fails, is replaced or is scaled in. The scalable design is a stateless fleet with session state in ElastiCache or DynamoDB.
• TLS termination (offload): the load balancer holds the X.509 server certificate, decrypts client traffic, and forwards it to targets in plain HTTP or re-encrypted, inside the VPC. "SSL" is the legacy name; today's protocol is TLS 1.2/1.3.
• AWS Certificate Manager (ACM) issues public certificates for free (when used with integrated services such as ELB, CloudFront and API Gateway), renews them automatically and never exposes the private key of a standard (non-exportable) certificate. Exportable public certificates are a separate paid option (below). Certificates are Regional resources.
• SNI (Server Name Indication) lets the client name the host it wants during the TLS handshake, so one listener on 443 can serve many domains, each with its own certificate (ALB, NLB TLS listeners, CloudFront; not CLB).

Exam facts

Sticky sessions

• ALB and NLB: configured per target group. CLB: per listener. ALB and CLB stickiness relies on cookies, so the client must accept cookies.
• ALB duration-based (stickiness.type = lb_cookie): the load balancer generates the cookie AWSALB plus AWSALBCORS (same data with SameSite=None; Secure, for browsers that need it on cross-origin requests) and sends both on every response, CORS or not; if a request carries both, AWSALBCORS wins. Duration 1 s to 7 days (604,800 s), default 1 day (86,400 s).
• ALB application-based (app_cookie): your application sets its own cookie, whose name you configure per target group; the load balancer then adds its own encrypted cookie, which the client sees as AWSALBAPP-0 (plus AWSALBAPP-1…-3 for cookies over 4 KB, max 16 KB). Clients must send back both cookies, so stickiness ends when the app deletes or expires its cookie. Duration 1 s to 7 days, default 1 day (86,400 s). Cookie names starting with AWSALB, AWSALBAPP or AWSALBTG are reserved for the load balancer.
• Target-group stickiness for weighted forward actions (blue/green, §8): cookie AWSALBTG (+ AWSALBTGCORS), duration 1 s to 7 days.
• CLB: duration-based cookie AWSELB (+ AWSELBCORS) or application-controlled stickiness that follows your cookie's lifetime.
• NLB: stickiness by source IP (stickiness.type = source_ip), no cookies; not supported on TLS or QUIC listeners; every client behind the same NAT lands on the same target.
• ALB limits: stickiness needs cross-zone load balancing on (the target group default); not combinable with the weighted_random algorithm; application-based stickiness does not work with weighted target groups; WebSocket connections are inherently sticky.
• Load balancer cookies are encrypted with a rotating key; the ALB resets their expiry after every request. If the sticky target is unhealthy or deregistered, the load balancer picks a new healthy target and in-memory session data is gone.
• Metrics: NonStickyRequestCount (ALB had to choose a new target), AppCookieNonStickinessCount.

ACM certificates

• Validation: DNS (recommended; renews automatically while the CNAME record stays in DNS and the certificate is in use) or email (renewal needs someone to approve). HTTP validation exists only for certificates created through CloudFront distribution tenants (usable only with CloudFront, no wildcards). Wildcards cover one label level only and not the apex (*.example.com does not match example.com or a.b.example.com).
• ⚠️ Changed: ACM public certificates are valid for 198 days, and DNS-validated certificates enter managed renewal 45 days before expiry (older 395-day certificates renew 60 days before expiry). Older material may say 13 months (395 days) with renewal from 60 days. Under the CA/Browser Forum schedule the maximum public validity falls to 100 days for certificates issued from 15 March 2027 and to 47 days from 15 March 2029; ACM has not yet documented its own future values.
• Regional: the certificate must be in the same Region as the ALB/NLB (the same domain in two Regions = two certificates, renewed independently). Certificates for CloudFront viewers must be requested or imported in us-east-1.
• Renewal keeps the same certificate ARN, so listeners need no change. Imported certificates (third-party CA) are not renewed by ACM: re-import before expiry. IAM server certificates are the legacy alternative store.
• ⚠️ Changed: since June 2025 ACM also issues exportable public certificates (certificate, private key and chain) for EC2, containers or on-premises servers; they are paid, $7 per FQDN and $79 per wildcard name at issuance and at each renewal (since February 2026, down from $15/$149), and you redeploy each renewal yourself. Standard ACM public certificates still cannot be exported. Older material may say ACM public certificates can never leave ACM.

HTTPS / TLS listeners

• An HTTPS (ALB) or TLS (NLB) listener needs exactly one default certificate, plus an optional certificate list for SNI. Quota: 25 certificates per ALB excluding the default (adjustable).
• The default certificate is served when the client sends no SNI hostname or nothing in the list matches. If several certificates match, the ALB picks the best one the client supports (ECDSA before RSA, unexpired first, stronger hash, longer key). If certificates match but none is compatible with the client, the handshake fails: no fallback to the default certificate.
• The console adds the default certificate to the certificate list; listeners created with the CLI, CloudFormation or CDK start with an empty list, so the default certificate joins SNI selection only if you add it with add-listener-certificates (in CloudFormation: the Certificates property holds exactly one default certificate, AWS::ElasticLoadBalancingV2::ListenerCertificate holds the list).
• A security policy sets the TLS versions and ciphers. ⚠️ Changed: the console default for new ALB HTTPS listeners is ELBSecurityPolicy-TLS13-1-2-Res-PQ-2025-09 (TLS 1.3 + 1.2 with hybrid post-quantum key exchange); listeners created through the CLI, CloudFormation or CDK still default to ELBSecurityPolicy-2016-08. Older material may say ELBSecurityPolicy-2016-08 (or ELBSecurityPolicy-TLS13-1-2-2021-06) is the default everywhere. Choose an older policy such as ELBSecurityPolicy-TLS13-1-0-2021-06 only for legacy clients that need TLS 1.0/1.1.
• Access log fields domain_name (SNI host) and chosen_cert_arn show which certificate was served.

Offload vs end-to-end encryption

• ALB: HTTPS listener → HTTP targets (offload) or HTTPS targets (re-encryption). The ALB does not validate target certificates, so self-signed ones work. The ALB always terminates TLS; it cannot pass a TLS session through.
• NLB: TLS listener terminates TLS (optionally re-encrypting to a TLS target group; ALPN policy HTTP2Preferred etc., default None); a TCP listener on 443 passes the encrypted stream through so the target holds the certificate. NLBs do not support mutual TLS, so client-certificate checks behind an NLB need TCP passthrough.
• ALB mutual TLS: passthrough mode forwards the client certificate chain in the X-Amzn-Mtls-Clientcert header for the app to validate; verify mode makes the ALB validate clients against a trust store (CA bundle uploaded from S3, optional revocation lists) and pass X-Amzn-Mtls-Clientcert-Subject, -Issuer, -Serial-Number, -Validity and -Leaf headers.
• CLB: one certificate per listener, no SNI → several domains need a SAN/wildcard certificate or several CLBs.

Developer angle

Request a certificate, attach it to an HTTPS listener with an explicit modern policy, then add more domains for SNI:

# Same Region as the ALB (use --region us-east-1 instead for a CloudFront distribution)
aws acm request-certificate --domain-name shop.example.com \
  --subject-alternative-names www.shop.example.com --validation-method DNS --region eu-west-1
aws acm describe-certificate --certificate-arn "$SHOP_CERT_ARN" --region eu-west-1 \
  --query 'Certificate.DomainValidationOptions[].ResourceRecord'   # CNAME to create in DNS

# HTTPS listener: default certificate + explicit policy (the CLI default is ELBSecurityPolicy-2016-08)
aws elbv2 create-listener --load-balancer-arn "$ALB_ARN" --protocol HTTPS --port 443 \
  --certificates CertificateArn="$SHOP_CERT_ARN" \
  --ssl-policy ELBSecurityPolicy-TLS13-1-2-Res-PQ-2025-09 \
  --default-actions Type=forward,TargetGroupArn="$WEB_TG_ARN"

# More domains on the same listener (SNI certificate list)
aws elbv2 add-listener-certificates --listener-arn "$HTTPS_LISTENER_ARN" \
  --certificates CertificateArn="$BLOG_CERT_ARN" CertificateArn="$API_CERT_ARN"

# Port 80 only redirects to HTTPS
aws elbv2 create-listener --load-balancer-arn "$ALB_ARN" --protocol HTTP --port 80 \
  --default-actions '[{"Type":"redirect","RedirectConfig":{"Protocol":"HTTPS","Port":"443","StatusCode":"HTTP_301"}}]'

# Which protocols/ciphers does a policy allow?
aws elbv2 describe-ssl-policies --names ELBSecurityPolicy-TLS13-1-2-Res-PQ-2025-09

Check which certificate a hostname receives (SNI on the wire):

openssl s_client -connect shop.example.com:443 -servername api.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -dates

Stickiness on an ALB target group:

# Duration-based: load balancer cookie AWSALB, 1 hour
aws elbv2 modify-target-group-attributes --target-group-arn "$WEB_TG_ARN" --attributes \
  Key=stickiness.enabled,Value=true Key=stickiness.type,Value=lb_cookie \
  Key=stickiness.lb_cookie.duration_seconds,Value=3600

# Application-based: follow the app's own session cookie (name must not start with AWSALB)
aws elbv2 modify-target-group-attributes --target-group-arn "$WEB_TG_ARN" --attributes \
  Key=stickiness.enabled,Value=true Key=stickiness.type,Value=app_cookie \
  Key=stickiness.app_cookie.cookie_name,Value=SESSIONID \
  Key=stickiness.app_cookie.duration_seconds,Value=3600

The scalable alternative: keep the session outside the instance (DynamoDB table with TTL on expires_at):

import time
import uuid
import boto3

sessions = boto3.resource("dynamodb").Table("web-sessions")

def save_session(cart):
    session_id = str(uuid.uuid4())
    sessions.put_item(Item={"session_id": session_id, "cart": cart,
                            "expires_at": int(time.time()) + 3600})
    return session_id

Mutual TLS in verify mode:

aws elbv2 create-trust-store --name partner-ca \
  --ca-certificates-bundle-s3-bucket my-pki-bucket --ca-certificates-bundle-s3-key partner-ca-bundle.pem
aws elbv2 modify-listener --listener-arn "$HTTPS_LISTENER_ARN" \
  --mutual-authentication Mode=verify,TrustStoreArn="$TRUST_STORE_ARN"

Traps

• Trap: "Users lose their carts, so stickiness is the MOST scalable fix." → Reality: stickiness is the quick fix; sessions still die with the instance and load skews. Most scalable and resilient: stateless instances plus an ElastiCache or DynamoDB session store.
• Trap: "Name the application cookie AWSALBAPP or AWSALBsession." → Reality: names starting with AWSALB, AWSALBAPP or AWSALBTG are reserved; use your own name such as SESSIONID.
• Trap: "Duration-based stickiness keeps the user on one server for the configured duration whatever happens." → Reality: if the target fails health checks or is deregistered, the ALB moves the user to a healthy target.
• Trap: "NLB stickiness uses a cookie." → Reality: it hashes the source IP (no cookies) and is unavailable on TLS and QUIC listeners.
• Trap: "One CLB HTTPS listener can serve three domains with three certificates." → Reality: CLB has one certificate per listener and no SNI; use an ALB or NLB certificate list, or a single SAN/wildcard certificate.
• Trap: "Request the ACM certificate in the ALB's Region for the CloudFront distribution too." → Reality: CloudFront viewer certificates must be in us-east-1; ALB/NLB certificates must be in the load balancer's Region.
• Trap: "ACM renews every certificate it stores." → Reality: only ACM-issued ones (DNS-validated automatically, email-validated after approval); imported certificates must be re-imported.
• Trap: "A listener created with CloudFormation automatically gets the modern TLS 1.3 policy." → Reality: the API default is still ELBSecurityPolicy-2016-08 (allows TLS 1.0/1.1); set SslPolicy / --ssl-policy explicitly.
• Trap: "For end-to-end TLS where the load balancer must not decrypt, use an ALB with HTTPS targets." → Reality: the ALB always decrypts (and re-encrypts); only a layer-4 passthrough listener leaves TLS untouched: an NLB TCP listener (or a legacy CLB TCP listener; use the NLB for new work).
• Trap: "Targets behind an HTTPS target group need certificates from a public CA." → Reality: the ALB does not validate target certificates; self-signed ones work.
• Trap: "*.example.com also covers example.com." → Reality: wildcards cover exactly one subdomain level and never the apex; add the apex as a SAN.

Practice questions

1. A company runs a web application on EC2 instances in an Auto Scaling group behind an Application Load Balancer. Shopping carts are kept in each instance's memory, and users report empty carts after scale-in events and instance failures. The company wants the MOST scalable and resilient solution. What should a developer do?

   • A. Enable duration-based stickiness on the target group with a 7-day duration.
   • B. Store session data in Amazon ElastiCache or Amazon DynamoDB and keep the instances stateless.
   • C. Enable application-based stickiness that follows the application's session cookie.
   • D. Turn on scale-in protection for every instance in the Auto Scaling group.

2. A developer must serve shop.example.com and blog.example.org over HTTPS on port 443 from a single internet-facing load balancer in eu-west-1. Each domain needs its own publicly trusted certificate, and operational overhead must be as low as possible. Which TWO actions should the developer take? (Select TWO.)

   • A. Use an Application Load Balancer with one HTTPS listener: one certificate as the default and the other in the listener's certificate list, so SNI selects the right one.
   • B. Use a Classic Load Balancer with one HTTPS listener and attach both certificates to it.
   • C. Request both public certificates from ACM in eu-west-1 with DNS validation.
   • D. Request both certificates from ACM in us-east-1, because load balancer certificates are global.
   • E. Import certificates from a third-party CA into IAM and rotate them manually before they expire.

10. Auto Scaling: policies, health checks and instance refresh

Concept

• An Auto Scaling group (ASG) keeps the number of instances between minimum and maximum, steering toward desired capacity; it replaces unhealthy instances, balances instances across AZs and registers them in attached target groups (§8). The ASG itself is free; you pay for the instances.
• What it launches comes from a versioned launch template: AMI, instance type, user data (§3), instance profile (§1), security groups, key pair, storage, metadata options (§2), optionally mixed instance types and Spot (§5).
• When it scales comes from scaling policies driven by CloudWatch metrics (§11): target tracking, step, simple, scheduled and predictive.
• How you change a running fleet: instance refresh rolls a new launch template version through the group with health gates, checkpoints and rollback; lifecycle hooks let your code run while instances launch (bootstrap) or terminate (drain, copy logs).

Exam facts

Capacity, launch templates, health checks

• min ≤ desired ≤ max; scaling policies, scheduled actions and set-desired-capacity change desired capacity, never beyond min/max (a scheduled action can also change min/max).
• Launch templates are versioned: reference a specific number, $Default or $Latest. Updating a template does not touch running instances; only new launches (or an instance refresh) use it.
• ⚠️ Changed: launch configurations get no new instance types since 1 January 2023; accounts created on or after 1 June 2023 cannot create them in the console, and accounts created on or after 1 October 2024 cannot create them by any method (console, API, CLI, CloudFormation). They are immutable. Older material may call them merely "legacy but available".
• Health check types: EC2 status checks and scheduled events (default, always on; any state other than running → replaced immediately). ELB, VPC Lattice and EBS health checks must be turned on (--health-check-type takes a comma-separated value such as ELB or ELB,EBS; EC2 cannot be disabled). Custom checks: set-instance-health. ⚠️ Changed: VPC Lattice and EBS health checks now exist alongside EC2 and ELB. Older material may list only EC2 and ELB.
• ⚠️ Changed: EC2 application status checks extend the default EC2 check to the application. You create them in EC2 (aws ec2 create-application-status-check --protocol https --port 443 --path /health) and associate them with the group's instances by tag (aws ec2 associate-application-status-check --target-tag-associations Key=aws:autoscaling:groupName,Value=web-asg). They send HTTP/HTTPS requests every 60 s (fixed), with failure and success thresholds of 2, a 6 s timeout, matcher 200 and a 300 s initialization grace period. When an instance's overall application status is impaired, Auto Scaling replaces it even with health check type EC2 and no group change. They do not follow redirects, so a path that answers 301/302 or 401/403 fails. Cost: $0.01 per managed ENI-hour per AZ. Older material may say only ELB health checks detect application failures.
• Health check grace period: console default 300 s, CLI/SDK default 0 s (0 = off). It delays health-check-based replacement after an instance becomes InService (also after leaving standby or manual attach).
• Unhealthy instances are replaced: by default the new instance is provisioned while the old one terminates, at most 10% of desired capacity at a time (instances that are no longer running are replaced immediately). An instance maintenance policy (MinHealthyPercentage 0–100, MaxHealthyPercentage 100–200) can make it launch and warm up the replacement first. With connection draining, the ASG waits for deregistration before terminating (§8).
• Group metrics (AWS/AutoScaling, e.g. GroupInServiceInstances, GroupDesiredCapacity) are free at 1-minute granularity but must be enabled (enable-metrics-collection).

Scaling policies

• Target tracking: choose a metric and target value; Auto Scaling creates and owns the CloudWatch alarms (TargetTracking-…-AlarmHigh/AlarmLow; never edit them). Predefined metrics: ASGAverageCPUUtilization, ASGAverageNetworkIn, ASGAverageNetworkOut, ALBRequestCountPerTarget (needs ResourceLabel). Custom metrics and metric math are allowed; high-resolution metrics down to a 10 s period. Scale-in can be disabled. With several target tracking policies, the ASG scales out if any policy asks and scales in only if all agree.
• The metric must move in proportion to capacity: RequestCount, latency or raw ApproximateNumberOfMessagesVisible do not work; use per-instance values such as SQS backlog per instance = visible messages ÷ GroupInServiceInstances, with target = acceptable latency ÷ average processing time per message. ⚠️ Changed: target tracking computes this with metric math, so no custom metric has to be published. Older material may say you must publish backlog per instance yourself with PutMetricData.
• Step scaling: you create the alarm; step adjustments scale by breach size (ChangeInCapacity, ExactCapacity, PercentChangeInCapacity + MinAdjustmentMagnitude); keeps reacting while instances warm up.
• Simple scaling: one adjustment per alarm, then waits for the cooldown. AWS recommends target tracking or step instead.
• ⚠️ Changed: the default cooldown (300 s) gates simple scaling policies. Target tracking and step scaling scale out without waiting for it (they use instance warmup), and scheduled actions and unhealthy-instance replacements ignore it, but scale-in can still be delayed while the group is in a cooldown. Older material may say the cooldown pauses every scaling activity of the group.
• Default instance warmup: off unless you set it (AWS strongly recommends it; start around 300 s). New instances do not feed the group's aggregated metrics until warmup ends, and scale-in waits for it. Used by target tracking, step scaling, instance refresh and instance maintenance. If unset, scaling policies fall back to the default cooldown and instance refresh to the health check grace period. Pre-baked (golden) AMIs (§3, §6) shorten boot time, so warmup and grace period can be shorter.
• Scheduled actions: one-time or cron recurrence, UTC unless you pass a time zone (IANA name, DST-aware); can set min, max and desired; up to 125 per ASG; may start up to 2 minutes late.
• Predictive scaling: needs 24 h of metric history (analyses up to 14 days), forecasts hourly capacity for the next 48 h, refreshes the forecast every 6 h. Starts in ForecastOnly mode; ForecastAndScale only adds capacity (never scales in), so pair it with a dynamic policy. SchedulingBufferTime launches instances ahead of the forecast. Fits recurring daily/weekly patterns and slow-booting apps.
• EC2 metrics arrive every 5 minutes by default; enable detailed monitoring (1 minute) for faster reactions (§11).

Instance refresh (StartInstanceRefresh)

• Replaces instances in batches so the group converges on a desired configuration (launch template version and/or mixed instances policy); new scale-outs during the refresh already use it.
• MinHealthyPercentage default 90% (0–100) and MaxHealthyPercentage default 100% (100–200; max − min ≤ 100) when no instance maintenance policy exists. ⚠️ Changed: with MinHealthyPercentage 100% the refresh launches replacements before terminating old instances, and a higher MaxHealthyPercentage replaces more at once. Older material may say an instance refresh always terminates first, dropping capacity to the minimum healthy percentage.
• InstanceWarmup: defaults to the default instance warmup, else the health check grace period.
• CheckpointPercentages (e.g. [25, 100]) with CheckpointDelay (default 3600 s) pause the rollout in phases; each checkpoint except the final 100% one emits an EventBridge event. Percentages count new instances in the group; end the list with 100 or the refresh stops at the last checkpoint.
• SkipMatching leaves instances that already match the desired configuration (CLI default off, console default on); it does not notice new code pulled by user data.
• AutoRollback (default off) reverts to the previous configuration if the refresh fails or if any of up to 10 CloudWatch alarms in AlarmSpecification enters ALARM. Requires a desired configuration; the previous launch template version must be a specific number (not $Latest/$Default, not an SSM parameter AMI alias). Manual undo while in progress: rollback-instance-refresh.
• BakeTime (default 0) waits after the last replacement before declaring success. ScaleInProtectedInstances / StandbyInstances: CLI default Wait, console default Ignore. A refresh can run at most 14 days and keeps retrying for 1 hour before failing on a blocking problem.
• Strategies: Rolling (default) or ReplaceRootVolume (keeps network interfaces and data volumes; needs a mixed instances policy with ImageId overrides; no ELB support).

Lifecycle hooks

• Launch hook (autoscaling:EC2_INSTANCE_LAUNCHING): Pending → Pending:Wait → Pending:Proceed → InService. Termination hook (autoscaling:EC2_INSTANCE_TERMINATING): Terminating → Terminating:Wait → Terminating:Proceed → Terminated.
• Heartbeat timeout default 3600 s (30–7200 s); extend with record-lifecycle-action-heartbeat; global limit 48 h or 100 × heartbeat, whichever is smaller. DefaultResult defaults to ABANDON (launch: the instance is terminated and replaced; termination: by default it terminates either way).
• ⚠️ Changed: an instance lifecycle policy can retain an instance when its termination hook ends in ABANDON (TerminateHookAbandon): it moves to Terminating:Retained, keeps billing, no longer counts toward desired capacity (a replacement launches), and you terminate it yourself with terminate-instance-in-auto-scaling-group. Older material may say a termination hook can never stop the termination.
• Finish early with complete-lifecycle-action (CONTINUE or ABANDON); code on the instance can poll IMDS autoscaling/target-lifecycle-state.
• ⚠️ Changed: notification targets are EventBridge (the console default), or directly SNS or SQS (IAM role ARN; FIFO queues are not supported) or Lambda (resource-based policy that allows the AWSServiceRoleForAutoScaling service-linked role); direct targets are set with the CLI, CloudFormation or an SDK. Older material may list only EventBridge, SNS and SQS.
• Hooks do not run for attach/detach, standby moves or force delete; termination hooks are best effort and cannot delay a Spot interruption (2-minute notice).

Termination, protection, troubleshooting, cost

• Default termination order: balance AZs first (AZ with the most unprotected instances), then the allocation strategy (mixed groups), then outdated configuration (launch configuration → other template → oldest template version), then closest to the next billing hour, then random. Other policies: AllocationStrategy, OldestLaunchTemplate, OldestLaunchConfiguration, ClosestToNextInstanceHour, NewestInstance, OldestInstance, or a custom Lambda. Unhealthy instances bypass the termination policy.
• Scale-in protection stops only scale-in terminations, not health-check replacement, Spot interruption or manual termination.
• Suspend processes to troubleshoot: Launch, Terminate, AddToLoadBalancer, AlarmNotification, AZRebalance, HealthCheck, InstanceRefresh, ReplaceUnhealthy, ScheduledActions. Or move one instance to Standby (no health checks, out of the load balancer).
• Warm pools keep pre-initialised instances Stopped (pay only EBS and Elastic IPs), Hibernated or Running for fast scale-out; optional ReuseOnScaleIn; not with Spot or weighted mixed groups.
• Capacity Rebalancing (Spot): replaces a Spot Instance that receives a rebalance recommendation before its 2-minute interruption notice, launching the replacement first (mechanics and the 10% overshoot above max: see §5); pair it with the price-capacity-optimized allocation strategy.

Developer angle

Wire the group to the load balancer, use ELB health checks, a realistic grace period and a default warmup:

aws autoscaling attach-traffic-sources --auto-scaling-group-name web-asg \
  --traffic-sources Identifier="$WEB_TG_ARN",Type=elbv2
aws autoscaling update-auto-scaling-group --auto-scaling-group-name web-asg \
  --health-check-type ELB --health-check-grace-period 300 --default-instance-warmup 120
aws autoscaling enable-metrics-collection --auto-scaling-group-name web-asg --granularity "1Minute"

Target tracking on requests per target, and a scheduled weekly peak:

aws autoscaling put-scaling-policy --auto-scaling-group-name web-asg \
  --policy-name rps-per-target --policy-type TargetTrackingScaling \
  --target-tracking-configuration '{
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ALBRequestCountPerTarget",
      "ResourceLabel": "app/web-alb/50dc6c495c0c9188/targetgroup/web-tg/73e2d6bc24d8a067"
    },
    "TargetValue": 500.0
  }'

aws autoscaling put-scheduled-update-group-action --auto-scaling-group-name web-asg \
  --scheduled-action-name friday-peak --recurrence "0 17 * * 5" --time-zone "Europe/Berlin" \
  --min-size 10 --max-size 30

Queue workers: target tracking on SQS backlog per instance with metric math. GroupInServiceInstances exists only when group metrics are enabled; without it m2 has no data, the policy's alarms stay in INSUFFICIENT_DATA and the group never scales:

aws autoscaling enable-metrics-collection --auto-scaling-group-name worker-asg --granularity "1Minute"
aws autoscaling put-scaling-policy --auto-scaling-group-name worker-asg \
  --policy-name sqs-backlog --policy-type TargetTrackingScaling \
  --target-tracking-configuration file://sqs-backlog.json

sqs-backlog.json:

{
  "CustomizedMetricSpecification": {
    "Metrics": [
      { "Id": "m1", "ReturnData": false,
        "MetricStat": { "Metric": { "Namespace": "AWS/SQS", "MetricName": "ApproximateNumberOfMessagesVisible",
          "Dimensions": [ { "Name": "QueueName", "Value": "image-jobs" } ] }, "Stat": "Sum" } },
      { "Id": "m2", "ReturnData": false,
        "MetricStat": { "Metric": { "Namespace": "AWS/AutoScaling", "MetricName": "GroupInServiceInstances",
          "Dimensions": [ { "Name": "AutoScalingGroupName", "Value": "worker-asg" } ] }, "Stat": "Average" } },
      { "Id": "e1", "Expression": "m1 / m2", "Label": "Backlog per instance", "ReturnData": true }
    ]
  },
  "TargetValue": 100
}

Roll out launch template version 8 safely (aws autoscaling start-instance-refresh --cli-input-json file://refresh.json):

{
  "AutoScalingGroupName": "web-asg",
  "DesiredConfiguration": {
    "LaunchTemplate": { "LaunchTemplateName": "web-lt", "Version": "8" }
  },
  "Preferences": {
    "MinHealthyPercentage": 100,
    "MaxHealthyPercentage": 110,
    "InstanceWarmup": 120,
    "SkipMatching": true,
    "CheckpointPercentages": [25, 100],
    "CheckpointDelay": 900,
    "AutoRollback": true,
    "AlarmSpecification": { "Alarms": ["web-alb-elb-5xx-high"] }
  }
}

aws autoscaling describe-instance-refreshes --auto-scaling-group-name web-asg
aws autoscaling rollback-instance-refresh --auto-scaling-group-name web-asg   # only while in progress

Launch hook that waits for bootstrapping; the last lines of the user data script release the instance (IMDSv2, §2):

aws autoscaling put-lifecycle-hook --auto-scaling-group-name web-asg \
  --lifecycle-hook-name wait-for-bootstrap \
  --lifecycle-transition autoscaling:EC2_INSTANCE_LAUNCHING \
  --heartbeat-timeout 600 --default-result ABANDON

# ... at the end of user data, after the app passed a local smoke test:
TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)
aws autoscaling complete-lifecycle-action --auto-scaling-group-name web-asg \
  --lifecycle-hook-name wait-for-bootstrap --instance-id "$INSTANCE_ID" \
  --lifecycle-action-result CONTINUE

The instance profile role needs permission for that call:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["autoscaling:CompleteLifecycleAction", "autoscaling:RecordLifecycleActionHeartbeat"],
      "Resource": "arn:aws:autoscaling:eu-west-1:111122223333:autoScalingGroup:*:autoScalingGroupName/web-asg"
    }
  ]
}

Debug an instance that keeps being replaced:

aws autoscaling describe-scaling-activities --auto-scaling-group-name web-asg --max-items 5
aws autoscaling suspend-processes --auto-scaling-group-name web-asg --scaling-processes ReplaceUnhealthy
# ... inspect the instance, fix the health check path or security group ...
aws autoscaling resume-processes --auto-scaling-group-name web-asg --scaling-processes ReplaceUnhealthy

# Alternative: take one instance out of service (deregistered, no health checks, still billed)
aws autoscaling enter-standby --auto-scaling-group-name web-asg \
  --instance-ids i-0123456789abcdef0 --should-decrement-desired-capacity
aws autoscaling exit-standby --auto-scaling-group-name web-asg --instance-ids i-0123456789abcdef0

Traps

• Trap: "The ALB marks targets unhealthy, so the ASG replaces them." → Reality: by default the ASG only uses EC2 status checks; set the health check type to ELB.
• Trap: "Instances are terminated in a loop right after launch, so raise the cooldown." → Reality: cooldowns don't gate health replacements; the grace period is too short (0 s by default from the CLI) or the health check path/security group is wrong. Raise the grace period or use a launch lifecycle hook, and suspend ReplaceUnhealthy while debugging.
• Trap: "EC2 application status checks can use the home page, like a browser would." → Reality: they do not follow redirects, so a page that redirects to a sign-in page (301/302) or demands authentication (401/403) fails every check and Auto Scaling replaces healthy instances; use an unauthenticated health path that returns 200 (the same rule as ELB health checks).
• Trap: "Tune the CloudWatch alarms that target tracking created." → Reality: Auto Scaling owns and may replace them; change the target value or metric instead.
• Trap: "Target tracking on ApproximateNumberOfMessagesVisible (or RequestCount) keeps workers right-sized." → Reality: the metric must be per instance: SQS backlog per instance via metric math, or ALBRequestCountPerTarget.
• Trap: "Predictive scaling alone handles nights and weekends." → Reality: it only scales out; keep a dynamic policy for scale-in.
• Trap: "A cron expression in a scheduled action runs in the Region's local time." → Reality: UTC unless you set a time zone.
• Trap: "Publishing launch template version 8 updates the fleet." → Reality: running instances keep their version; start an instance refresh (or wait for replacements).
• Trap: "Auto rollback works even if the group used $Latest." → Reality: rollback needs a desired configuration and a numbered previous version.
• Trap: "Skip matching guarantees every instance runs the new code." → Reality: it compares launch template version and instance types only; code fetched by user data is not detected, so turn skip matching off.
• Trap: "Scale-in protection keeps an instance alive through health failures." → Reality: it blocks only scale-in; health replacement, Spot interruption and manual terminate still remove it.
• Trap: "If nobody completes the launch hook, the instance continues after the timeout." → Reality: DefaultResult is ABANDON unless you set CONTINUE; the instance is terminated and replaced.
• Trap: "Create a launch configuration for the new group." → Reality: new accounts (since 1 October 2024) cannot; use a launch template.

Practice questions

1. A team runs 20 instances in an Auto Scaling group behind an Application Load Balancer and has created launch template version 8 with a patched AMI. The rollout must never drop below current serving capacity, must pause after a quarter of the instances are replaced so testers can verify them, and must revert automatically if the ALB 5XX CloudWatch alarm enters the ALARM state. Which approach meets these requirements?

   • A. Point the Auto Scaling group at the $Latest template version and terminate five instances at a time from the console.
   • B. Start an instance refresh with a desired configuration of version 8, MinHealthyPercentage 100, MaxHealthyPercentage 110, CheckpointPercentages [25, 100], AutoRollback enabled, and the alarm in AlarmSpecification.
   • C. Start an instance refresh with MinHealthyPercentage 75, skip matching disabled, and a bake time of one hour.
   • D. Change the termination policy to OldestLaunchTemplate and let normal scale-in activity replace the instances.

2. A developer created an Auto Scaling group with the AWS CLI, attached it to an ALB target group, and set the health check type to ELB. New instances need about four minutes to finish their user data bootstrapping, and the group keeps terminating and replacing them before they ever serve traffic. Which TWO changes will stop the replacement loop? (Select TWO.)

   • A. Set the health check grace period to a value longer than the bootstrap time.
   • B. Add a launch lifecycle hook and call complete-lifecycle-action at the end of the user data script.
   • C. Increase the group's default cooldown to 600 seconds.
   • D. Replace the target tracking policy with a step scaling policy.
   • E. Increase the target group's deregistration delay to 3600 seconds.

11. CloudWatch for EC2: metrics, agent and status checks

Concept

• EC2 automatically publishes hypervisor-level metrics to the AWS/EC2 namespace: whatever can be measured from outside the guest OS (CPU, network, instance-store and EBS I/O, CPU credits, status checks).
• Anything only the OS knows (memory, disk-space usage, swap, processes, application logs) needs the unified CloudWatch agent or your own PutMetricData / PutLogEvents calls, with permissions coming from the instance role.
• Status checks tell you who owns the fix: system = AWS infrastructure, instance = your OS and configuration, attached EBS = volume reachability.
• Alarms turn a metric into an action: an SNS notification, an Auto Scaling policy, or an EC2 stop, terminate, reboot or recover.

Exam facts

• Basic monitoring (default, free): 5-minute data points; status-check metrics every 1 minute at no charge. Detailed monitoring: 1-minute data, billed per metric; enable at launch (--monitoring Enabled=true, launch template "Monitoring": {"Enabled": true}) or later (aws ec2 monitor-instances). With ASG scaling policies, use detailed monitoring so scaling reacts faster. Launch templates default to basic; launch configurations created with the CLI default to detailed. Changing the launch template affects only instances launched afterwards: running instances keep their monitoring type until an instance refresh replaces them or you run monitor-instances on them.
• Built-in instance metrics: CPUUtilization; NetworkIn / NetworkOut (bytes) and NetworkPacketsIn / NetworkPacketsOut (5-minute only); DiskReadOps, DiskWriteOps, DiskReadBytes, DiskWriteBytes (instance store only); EBSReadOps, EBSWriteOps, EBSReadBytes, EBSWriteBytes, EBSIOBalance%, EBSByteBalance% (Nitro instances); MetadataNoToken, MetadataNoTokenRejected (IMDSv1 usage, §2).
• Burstable T instances add CPUCreditUsage, CPUCreditBalance, CPUSurplusCreditBalance, CPUSurplusCreditsCharged, at 5-minute frequency only.
• Status-check metrics (0 = pass, 1 = fail, every minute): StatusCheckFailed, StatusCheckFailed_Instance, StatusCheckFailed_System, StatusCheckFailed_AttachedEBS.
• Not collected by default: memory, disk-space used/free, swap, per-process data, OS and application logs.
• Dimensions: InstanceId and AutoScalingGroupName (any monitoring level); ImageId and InstanceType (detailed monitoring only).
• Retention: data at < 60 s kept 3 hours; 1-minute 15 days; 5-minute 63 days; 1-hour 455 days (15 months).
• Unified CloudWatch agent: collects metrics and logs (and traces) from EC2 and on-premises servers; custom app metrics via StatsD (Linux and Windows, default :8125) and collectd (Linux); per-process metrics with the procstat plugin; default namespace CWAgent; agent metrics are billed as custom metrics; a metrics_collection_interval below 60 s makes them high-resolution. Configure with the wizard (amazon-cloudwatch-agent-config-wizard), a local JSON file, or SSM Parameter Store (one central config for a fleet); control it with amazon-cloudwatch-agent-ctl or the Run Command document AmazonCloudWatch-ManageAgent.
• Agent IAM: attach a role with CloudWatchAgentServerPolicy to the instance (cloudwatch:PutMetricData; logs:CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogStreams, DescribeLogGroups, PutRetentionPolicy; ec2:DescribeTags, ec2:DescribeVolumes; X-Ray writes; ssm:GetParameter only on parameter/AmazonCloudWatch-*). So a config stored in Parameter Store must be named AmazonCloudWatch-.... CloudWatchAgentAdminPolicy is for whoever writes that config. The agent also needs outbound HTTPS to the CloudWatch and CloudWatch Logs endpoints (NAT gateway or VPC endpoints).
• ⚠️ Changed: the old CloudWatch Logs agent (awslogs) is deprecated and does not work on instances that require IMDSv2; use the unified CloudWatch agent. Older material may present the Logs agent as a valid choice for shipping logs.
• No EC2 log reaches CloudWatch Logs unless an agent or SDK sends it; a log group keeps data indefinitely until you set a retention period (the agent can set retention_in_days if the role allows logs:PutRetentionPolicy).
• PutMetricData: the namespace must not start with AWS/. StorageResolution 60 (default, standard 1-minute) or 1 (high-resolution, stored per second for 3 hours, readable at 1, 5, 10 or 30 s or any multiple of 60 s). Up to 30 dimensions per metric, and every distinct dimension set is a separate metric. Timestamps accepted from 2 weeks in the past to 2 hours in the future (keep instance clocks synced). Up to 1,000 metrics and 1 MB per request (gzip allowed), 150 values per Values array, 500 requests per second by default.
• ⚠️ Changed: one PutMetricData call takes up to 1,000 metrics and 1 MB. Older material may say 20 metrics and 40 KB per call.
• Alarm evaluation: states OK, ALARM, INSUFFICIENT_DATA (a new alarm starts in INSUFFICIENT_DATA). Period = seconds per data point; EvaluationPeriods = the N most recent periods; DatapointsToAlarm = M breaching points out of N; TreatMissingData = missing (default), notBreaching, breaching or ignore. Period × EvaluationPeriods may not exceed 1 day for periods under 1 hour (7 days otherwise). Actions run only on a state change (Auto Scaling actions repeat every minute while the alarm stays in ALARM); up to 5 actions per state.
• ⚠️ Changed: an alarm period can be 10, 20 or 30 seconds (high-resolution alarm, extra charge, meaningful only for metrics stored with StorageResolution 1) or any multiple of 60 seconds. Older material may list only 10 and 30 seconds.
• aws cloudwatch set-alarm-state tests an alarm and really runs the actions for the new state. Composite alarms (AND/OR of other alarms, to cut noise) can notify but cannot run EC2 or Auto Scaling actions.
• EC2 alarm actions: arn:aws:automate:<region>:ec2:stop, :terminate, :reboot, :recover; usable on any per-instance metric, including custom metrics that carry an InstanceId dimension; CloudWatch performs them through the service-linked role AWSServiceRoleForCloudWatchEvents. Recover works only with StatusCheckFailed_System, never _Instance. AWS suggests 2 × 1-minute evaluation periods for a recover alarm and 3 × 1-minute for a reboot alarm, so the two don't race.
• A recovered instance keeps its instance ID, private, public and Elastic IP addresses, instance metadata, placement group, attached EBS volumes and AZ; RAM contents are lost, and with alarm-based recovery so is instance-store data. A manual stop/start usually moves the instance to new hardware too, but it gets a new public IPv4 unless it has an Elastic IP.
• ⚠️ Changed: simplified automatic recovery is enabled by default on supported instance types and acts on a failed system status check with no alarm needed. It does not cover bare-metal instances, instances with instance-store volumes, Dedicated Hosts, EFA instances or members of an Auto Scaling group (the ASG replaces those instead). Switch it with aws ec2 modify-instance-maintenance-options --auto-recovery disabled|default. Older material may say automatic recovery always requires a CloudWatch alarm with the recover action.
• Status checks run every minute, return pass/fail and cannot be disabled:

• ⚠️ Changed: EC2 now also offers opt-in application status checks (an HTTP/HTTPS probe of a path and port every 60 s, metric StatusCheckFailed_Application, integrated with Auto Scaling replacement; setup, limits and cost: see §10). Older material may list only the system and instance checks.

Developer angle

Detailed monitoring and a high-resolution custom metric:

aws ec2 monitor-instances --instance-ids i-0123456789abcdef0     # 1-minute metrics (undo: unmonitor-instances)

aws cloudwatch put-metric-data --namespace OrdersApp --metric-name QueueLagMs \
  --dimensions InstanceId=i-0123456789abcdef0,Env=prod \
  --unit Milliseconds --value 87 --storage-resolution 1

import boto3

cloudwatch = boto3.client("cloudwatch")  # credentials come from the instance role
cloudwatch.put_metric_data(
    Namespace="OrdersApp",
    MetricData=[{
        "MetricName": "QueueLagMs",
        "Dimensions": [{"Name": "Env", "Value": "prod"}],
        "Unit": "Milliseconds",
        "Value": 87.0,
        "StorageResolution": 1,
    }],
)

CloudWatch agent config for memory, root-disk usage and an application log (cwagent.json):

{
  "agent": { "metrics_collection_interval": 60 },
  "metrics": {
    "append_dimensions": { "InstanceId": "${aws:InstanceId}" },
    "metrics_collected": {
      "mem": { "measurement": ["mem_used_percent"] },
      "disk": { "measurement": ["used_percent"], "resources": ["/"] }
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/orders/app.log",
            "log_group_name": "/orders/app",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 30
          }
        ]
      }
    }
  }
}

# Store once (admin with CloudWatchAgentAdminPolicy or ssm:PutParameter); name must start with AmazonCloudWatch-
aws ssm put-parameter --name AmazonCloudWatch-orders-linux --type String --value file://cwagent.json
# On each instance (role: CloudWatchAgentServerPolicy)
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config -m ec2 -s -c ssm:AmazonCloudWatch-orders-linux

Alarms that recover the instance and watch agent memory, plus a notification test:

aws cloudwatch put-metric-alarm --alarm-name web-1-recover \
  --namespace AWS/EC2 --metric-name StatusCheckFailed_System \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --statistic Maximum --period 60 --evaluation-periods 2 --datapoints-to-alarm 2 \
  --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold \
  --alarm-actions arn:aws:automate:eu-west-1:ec2:recover arn:aws:sns:eu-west-1:111122223333:ops-alerts

# Agent metric: namespace CWAgent; dimensions must match what the agent publishes
aws cloudwatch put-metric-alarm --alarm-name web-1-mem-high \
  --namespace CWAgent --metric-name mem_used_percent \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --statistic Average --period 60 --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 90 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-alerts

# Fires the alarm's actions for real: use it on SNS-only alarms
aws cloudwatch set-alarm-state --alarm-name web-1-mem-high \
  --state-value ALARM --state-reason "notification test"

Traps

• Trap: "Alarm on memory with CPUUtilization, or turn on detailed monitoring to get memory." → Reality: EC2 never sees guest memory; install the CloudWatch agent (CWAgent mem_used_percent) or publish your own metric. Detailed monitoring only changes the period from 5 minutes to 1 minute.
• Trap: "DiskReadBytes shows the EBS root volume's I/O." → Reality: the Disk* metrics cover instance store only; use the EBS* instance metrics or the AWS/EBS namespace. Disk space always needs the agent.
• Trap: "The agent is installed, so metrics will appear." → Reality: without an instance role carrying CloudWatchAgentServerPolicy and an outbound path to the CloudWatch endpoints, nothing is published; a Parameter Store config not named AmazonCloudWatch-* cannot be read with that managed policy.
• Trap: "Reboot fixes a failed system status check." → Reality: a reboot stays on the same host; use recover or stop/start, which moves the instance to healthy hardware.
• Trap: "Add the recover action to a StatusCheckFailed_Instance alarm." → Reality: recover is only valid with StatusCheckFailed_System; instance-check failures need a reboot or an OS fix.
• Trap: "Recovery gives the instance a new ID and new IPs." → Reality: it keeps the instance ID, private, public and Elastic IPs and metadata; only RAM (and instance store with alarm-based recovery) is lost.
• Trap: "A 10-second alarm on a standard-resolution metric reacts faster." → Reality: sub-minute periods need metrics stored with StorageResolution 1 (agent: metrics_collection_interval below 60); on 1-minute data a 10-second alarm finds a data point in only one period of six, often lapses into INSUFFICIENT_DATA, and is still billed as a high-resolution alarm.
• Trap: "Back-fill last month's numbers with PutMetricData." → Reality: data points older than 2 weeks (or more than 2 hours ahead) are rejected.
• Trap: "set-alarm-state is a harmless dry run." → Reality: it executes the actions configured for the new state; don't use it on alarms that stop, terminate or recover instances.
• Trap: "A composite alarm can reboot or recover the instance." → Reality: composite alarms can notify, but cannot run EC2 or Auto Scaling actions.
• Trap: "Simplified automatic recovery protects every instance in my Auto Scaling group." → Reality: ASG members are excluded; the ASG health check replaces them.

Practice questions

1. An application on EC2 measures queue latency every 10 seconds. The team wants a CloudWatch alarm that can enter ALARM within 30 seconds of a latency spike. What should the developer do?

   • A. Enable detailed monitoring on the instance and create an alarm with a 60-second period.
   • B. Publish the metric with PutMetricData using StorageResolution 1 and create an alarm with a 10-second period.
   • C. Publish the metric with StorageResolution 60 and create an alarm with a 10-second period.
   • D. Create a composite alarm that evaluates the latency metric every 10 seconds.

2. A CloudWatch alarm reports StatusCheckFailed_System = 1 for an EBS-backed m7i instance that is not part of an Auto Scaling group. Which TWO actions can restore the instance on healthy hardware? (Select TWO.)

   • A. Stop the instance and then start it again.
   • B. Reboot the instance from the EC2 console.
   • C. Configure a CloudWatch alarm on StatusCheckFailed_System with the EC2 recover action.
   • D. Install the CloudWatch agent and restart the application.
   • E. Add an inbound rule for ICMP to the instance's security group.

Quick decision rules

IAM roles, IMDS and user data (§1–§3)

• If you see an application on EC2 that needs AWS access in the MOST secure way, think IAM role delivered through an instance profile (no access keys on the instance).
• If you see an instance role being ignored (calls run as an IAM user, broader access than the role, or AccessDenied naming the wrong principal), think credential-chain precedence: environment variables or ~/.aws/credentials win; confirm with aws sts get-caller-identity.
• If you see "change the role of a running instance without downtime", think replace-iam-instance-profile-association (associate-iam-instance-profile if none is attached).
• If you see UnauthorizedOperation or AccessDenied when launching an instance with a role, think missing iam:PassRole on the role ARN (or an empty instance profile).
• If you see an encoded authorization failure message, think aws sts decode-authorization-message (needs sts:DecodeAuthorizationMessage).
• If you see two roles needed on one instance, think one role per instance profile: merge the least-privilege permissions into one role, or use ECS task roles.
• If you see an SSRF or open reverse proxy risk to instance credentials, think IMDSv2 (HttpTokens=required).
• If you see HTTP 401 from 169.254.169.254, think missing or expired IMDSv2 token.
• If you see containers on EC2 failing to get credentials after IMDSv2 was required, think PUT response hop limit 1: set it to 2 (or use ECS task roles).
• If you see "prove nothing still uses IMDSv1 before enforcing", think MetadataNoToken = 0 (then watch MetadataNoTokenRejected).
• If you see "all new launches in the account must use IMDSv2", think modify-instance-metadata-defaults per Region (+ --http-tokens-enforced enabled) or an SCP on ec2:MetadataHttpTokens.
• If you see "make credentials obtained through IMDSv1 unusable", think a deny on ec2:RoleDelivery NumericLessThan 2.0.
• If you see a bootstrap script that did not rerun after a reboot or after editing user data, think user data runs once per instance by default.
• If you see "debug a failed user data script", think /var/log/cloud-init-output.log (or aws ec2 get-console-output without SSH).
• If you see a password or API key in user data, think Secrets Manager or a Parameter Store SecureString fetched at boot with the instance role.
• If you see slow Auto Scaling scale-out caused by installs at boot or long user-data scripts, think golden AMI built and distributed by EC2 Image Builder, plus minimal user data.
• If you see "CloudFormation must wait until the software is installed", think CreationPolicy + cfn-signal.
• If you see user data that must change on an existing instance, think stop → modify-instance-attribute with base64 → start, and remember it will not rerun automatically.
• If you see user data not running when supplied through the raw API or a launch template, think missing (or double) base64 encoding.

Connectivity and access (§4)

• If you see a connection that times out or a request that hangs, think security group inbound rule, NACL, route table or missing public IP: the packet never reached a listener.
• If you see an instant Connection refused, think the host was reached but nothing listens on that port (process down, wrong port, bound to 127.0.0.1).
• If you see AccessDenied or UnauthorizedOperation from an SDK or CLI call, think IAM (role policy, explicit deny, wrong identity), not security groups; use sts get-caller-identity and sts decode-authorization-message.
• If you see Unable to locate credentials on EC2, think no instance profile, IMDS disabled, or a container behind hop limit 1.
• If you see Permission denied (publickey), think wrong username, wrong key or key file permissions (chmod 400), not the network.
• If you see HTTP 403 from your site or ALB, think application authorization, AWS WAF or a resource policy: the network path works.
• If you see Connect timeout on endpoint URL from a private subnet, think NAT gateway or VPC endpoint (a fast Could not connect to the endpoint URL points at the Region, endpoint URL or DNS instead).
• If you see "allow the database only from the application tier", think DB security group inbound 3306/5432 with the app security group ID as the source.
• If you see shell access with no inbound ports, auditing and the least operational overhead, think Session Manager (SSM Agent + AmazonSSMManagedInstanceCore + outbound 443 or VPC endpoints).
• If you see SSH or RDP to a private-subnet instance without a bastion or public IP, think EC2 Instance Connect Endpoint (or Session Manager).
• If you see browser-based SSH controlled by IAM, think EC2 Instance Connect with TCP 22 allowed from the ec2-instance-connect prefix list.
• If you see a public IP that must survive stop/start, think Elastic IP, and remember every public IPv4 costs $0.005 per hour.

Purchasing options and Spot (§5)

• If you see the lowest cost for fault-tolerant, interruptible batch or CI jobs, think Spot through an ASG or EC2 Fleet with price-capacity-optimized and many instance types.
• If you see "save work before the Spot instance is reclaimed", think IMDS spot/instance-action or the EventBridge interruption warning, plus rebalance recommendations and checkpointing to S3 or DynamoDB.
• If you see steady usage that will change instance family or Region or move to Fargate or Lambda, think Compute Savings Plans.
• If you see steady usage locked to one instance family in one Region with the biggest discount, think EC2 Instance Savings Plan or Standard RI.
• If you see per-socket or per-core BYOL licensing or host-level visibility, think Dedicated Hosts.
• If you see guaranteed capacity in one AZ for an event with no long-term commitment, think On-Demand Capacity Reservation (add Savings Plans or regional RIs for a discount).
• If you see a persistent Spot request that keeps launching new instances, think cancel the request first, then terminate the instances.
• If you see an ASG that needs a fixed On-Demand baseline with Spot above it, think mixed instances policy with OnDemandBaseCapacity and OnDemandPercentageAboveBaseCapacity.
• If you see a mixed instances ASG that launches no Spot, think OnDemandPercentageAboveBaseCapacity is still at its default of 100.

EBS, AMIs and storage choice (§6–§7)

• If you see "encrypt an existing unencrypted EBS volume", think snapshot → create-volume --encrypted (or an encrypted snapshot copy) → swap; encryption is never added in place.
• If you see "share an encrypted snapshot or AMI with another account", think customer managed KMS key + key policy/IAM (+ grant) + createVolumePermission or launch permission; impossible with aws/ebs.
• If you see one volume that needs up to 80,000 IOPS or 2,000 MiB/s at the lowest cost, think gp3 with provisioned IOPS and throughput, not io1/io2.
• If you see more than 80,000 IOPS, sub-millisecond latency, 99.999% durability or Multi-Attach, think io2 Block Express on a Nitro instance.
• If you see the same AMI needed in another Region for deployment or DR, think copy-image run in the destination Region with a destination-Region KMS key, then the new AMI ID in launch templates.
• If you see root volume data that disappeared after termination, think DeleteOnTermination=true on the root by default; set it to false at launch or with modify-instance-attribute.
• If you see "move EBS data to another AZ or Region", think snapshot (copy-snapshot for another Region), then create-volume in the target AZ.
• If you see a volume that was enlarged while df still shows the old size, think growpart + xfs_growfs or resize2fs.
• If you see "recover an accidentally deleted snapshot, AMI or EBS volume", think a Recycle Bin retention rule that existed before the deletion.
• If you see the cheapest long-term retention for rarely used snapshots, think EBS Snapshots Archive (≥ 90 days, restore up to 72 h).
• If you see volumes restored from snapshots that are slow on first access, think Fast Snapshot Restore (per AZ) or a Provisioned Rate for Volume Initialization.
• If you see "automate snapshot/AMI creation, retention and cross-Region copy by tags", think Amazon Data Lifecycle Manager.
• If you see "always launch the latest Amazon Linux AMI from the CLI or CloudFormation", think the SSM public parameter /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 via resolve:ssm: or AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>.
• If you see instances going pending → terminated with Client.InternalError and encrypted volumes, think missing KMS key permissions for the caller or AWSServiceRoleForAutoScaling.
• If you see a deprecated AMI still being used by an Auto Scaling group, think deprecation only hides it: disable or deregister it to block launches.
• If you see a shared POSIX file system for instances in several AZs, think EFS Regional with a mount target per AZ and security groups allowing TCP 2049.
• If you see the highest IOPS for scratch or cache data that can be lost, think instance store (survives reboot, lost on stop, hibernate or terminate).
• If you see persistent storage for a single-instance database, think EBS gp3 (io2 Block Express at extreme IOPS).
• If you see "choose an EFS performance mode", think General Purpose; Max I/O is previous generation.
• If you see spiky or unpredictable EFS throughput, think Elastic throughput; Provisioned only for steady, known demand.
• If you see an EFS mount that hangs and ends in Connection timed out, think TCP 2049 blocked between the security groups (Failed to resolve server = no mount target in that AZ).
• If you see Lambda needing a shared persistent file system, think EFS access point + VPC configuration + /mnt/ mount path + ClientMount/ClientWrite.
• If you see "encrypt EFS data in transit", think mount -t efs -o tls plus a file system policy denying aws:SecureTransport = false.
• If you see an existing EFS file system that must become encrypted at rest, think a new encrypted file system and replicating or copying the data.
• If you see Windows instances that need a shared file system, think FSx for Windows File Server (SMB).
• If you see HPC/ML needing a parallel file system over S3 data, think FSx for Lustre.

Load balancing, TLS and Auto Scaling (§8–§10)

• If you see routing by URL path, host name, HTTP header, method or query string (or a Lambda function as a target), think ALB listener rules forwarding to target groups.
• If you see fixed or allow-listed IP addresses, Elastic IPs, TCP/UDP traffic or millions of requests per second with the client IP preserved, think Network Load Balancer.
• If you see third-party firewall, IDS/IPS or deep packet inspection appliances inserted transparently into traffic, think Gateway Load Balancer (GENEVE port 6081).
• If you see ALB HTTP 502, think the target closed or reset the connection or returned a malformed response (keep-alive shorter than the idle timeout, Lambda error or timeout).
• If you see ALB HTTP 503, think the target group has no registered (or usable) targets.
• If you see ALB HTTP 504, think the target did not answer before the idle timeout or could not be reached within 10 s (security group or NACL).
• If you see HTTP 460, think the client gave up before the load balancer answered.
• If you see HTTP 463, think more than 30 IP addresses in X-Forwarded-For.
• If you see HTTPCode_ELB_5XX_Count rising while HTTPCode_Target_5XX_Count stays flat, think a load balancer or connectivity problem, not application code.
• If you see the application logging the load balancer's IP instead of the user's, think X-Forwarded-For (and X-Forwarded-Proto for the original scheme).
• If you see in-flight requests dropped during deployments or scale-in, think deregistration delay (connection draining).
• If you see deployments slowed because targets stay draining for 5 minutes, think a lower deregistration_delay.timeout_seconds.
• If you see traffic shifted gradually to a new version at the load balancer, think weighted target groups on the ALB forward action.
• If you see users losing session data and the MOST scalable or resilient fix, think sessions externalized to ElastiCache or DynamoDB rather than sticky sessions.
• If you see one instance overloaded after enabling stickiness, think stickiness imbalance: shorten the duration or make the tier stateless.
• If you see several HTTPS domains on one load balancer, think an ALB or NLB listener certificate list with SNI (a CLB cannot).
• If you see an ACM certificate for a CloudFront distribution, think request or import it in us-east-1.
• If you see TLS that the load balancer must not decrypt, think NLB TCP listener passthrough.
• If you see client certificate authentication at the load balancer, think ALB mutual TLS in verify mode with a trust store.
• If you see an HTTPS listener created with the CLI or CloudFormation that must enforce TLS 1.2+, think an explicit security policy (the API default is ELBSecurityPolicy-2016-08).
• If you see keeping average CPU or requests per target at a value, think target tracking scaling.
• If you see a known calendar peak, think a scheduled action with cron and a time zone.
• If you see recurring daily or weekly load patterns with slow-booting instances, think predictive scaling paired with a dynamic policy.
• If you see queue workers scaling on SQS, think target tracking on backlog per instance via metric math.
• If you see a new AMI or launch template version rolled through an Auto Scaling group, think instance refresh (min/max healthy %, checkpoints, skip matching, auto rollback with alarms).
• If you see instances terminated and replaced in a loop right after launch, think health check grace period or a launch lifecycle hook (and check the health check path and security groups).
• If you see an Auto Scaling group not replacing instances that fail ALB health checks, think health check type ELB.
• If you see code that must run before an instance takes traffic or before it terminates, think lifecycle hooks with complete-lifecycle-action.
• If you see debugging an instance without Auto Scaling replacing it, think suspend ReplaceUnhealthy or enter-standby.
• If you see slow-booting apps that must scale out fast at low cost, think a warm pool of stopped instances.
• If you see Spot Instances in an Auto Scaling group at elevated interruption risk, think Capacity Rebalancing.

CloudWatch for EC2 (§11)

• If you see memory or disk-space utilisation metrics for EC2, think CloudWatch agent with CloudWatchAgentServerPolicy on the instance role (CWAgent namespace).
• If you see faster Auto Scaling reaction to EC2 metrics, think detailed (1-minute) monitoring.
• If you see sub-minute custom metrics or alarms, think PutMetricData with StorageResolution 1 and a 10, 20 or 30-second alarm period.
• If you see StatusCheckFailed_System, think AWS hardware: recover action or stop/start, not reboot.
• If you see StatusCheckFailed_Instance, think your OS or configuration: reboot or fix it; recover is not allowed on this metric.
• If you see a need to test an alarm notification, think aws cloudwatch set-alarm-state, which really runs the actions for the new state.

Changed vs older material

Learn the Current column; older practice material may still show the Older material may say value. The same rows, with verification dates, live in notes/current-facts.md.