Containers (ECS, ECR, Fargate) — DVA-C02 Mastery Guide
- Sources: course chapter 09 (Containers on AWS: Docker, Amazon ECS, AWS Fargate, Amazon ECR, Amazon EKS). The course serves as the syllabus only; all wording, examples and questions here are original.
- Written 2026-10-12; values checked against AWS docs on 2026-10-12 (Amazon ECS Developer Guide and API Reference, Amazon ECR User Guide, Amazon EKS User Guide, AWS General Reference quotas, Application Auto Scaling and EventBridge Scheduler API references, the DVA-C02 exam guide and its revision history, What's New posts and the AWS Containers blog). The marker
⚠️ Changed: <current>. Older material may say <old>.flags a value or behaviour that differs from the course or older practice tests: learn the current value.⚠️ unverifiedmarks something AWS documentation did not confirm (or contradicts itself on): don't build an exam answer on it. Every change is listed in Changed vs older material; rows you confirm during/ingestare added tonotes/current-facts.md.- Tier 3 (supporting), bank target 30 questions. Domains: Deployment (primary), Development with AWS Services.
- How to use it:
/explain <topic>teaches from these sections (concept → exam facts → developer angle → traps → 3 check questions);/quiz containersasks bank questions whosesource_notelinks point at the## N.headings below;/drill containersfires short recall prompts fromnotes/containers-cheat-sheet.md. Suggested loop: read a section and answer its practice questions before opening the answers →/explainwhat is still unclear →/quiz containers→/drill containers.- Owned by other modules (one-line pointers only here): instance profiles, the credential chain, IMDSv2 and the hop limit →
notes/ec2-mastery-guide.md§1–§2; EFS mounts and modes →notes/ec2-mastery-guide.md§7; ALB/NLB listeners, target groups, deregistration delay and502/503/504→notes/ec2-mastery-guide.md§8; EC2 Auto Scaling groups →notes/ec2-mastery-guide.md§10; CodeDeploy AppSpec and ECS deployment configurations, CodePipelineimagedefinitions.json→notes/cicd-guide.md§4 and §8; Parameter Store vs Secrets Manager →notes/security-encryption-guide.md§8–§9; Lambda container images →notes/lambda-guide.md§8; Beanstalk Docker platforms →notes/beanstalk-guide.md§12; CDK ECS patterns such asApplicationLoadBalancedFargateService→notes/cdk-guide.md§2;iam:PassRolein depth →notes/advanced-identity-guide.md§13; SQS worker semantics →notes/sqs-sns-kinesis-guide.md§2; EventBridge rules, Scheduler syntax, retries and DLQs →notes/monitoring-guide.md§6; X-Ray/ADOT sidecars on ECS →notes/monitoring-guide.md§8; subnets, NAT and VPC endpoint design →notes/vpc-guide.md§1 and §3.
How Containers (ECS, ECR, Fargate) appears on DVA-C02
DVA-C02 treats containers as one of the ways a developer packages and ships an application. It does not test cluster administration; it asks what a developer must declare, grant or choose so a containerised app builds, starts, gets its configuration, scales and deploys safely:
- Packaging and deployment (Deployment domain): building and pushing images to ECR, registering task definition revisions, rolling updates with
minimumHealthyPercent/maximumPercent, blue/green, linear and canary releases, circuit breaker rollbacks, and the deploy action of a pipeline. - Writing the app around the platform (Development domain): which IAM role the code uses, how environment variables and secrets arrive, port mappings behind a load balancer, scaling a queue worker, running scheduled or event-driven tasks.
- Troubleshooting:
CannotPullContainerError,ResourceInitializationError,OutOfMemoryError, tasks stuck inPENDING,AccessDeniedfrom inside a container, ELB health-check failures. - Choosing under a constraint: Fargate vs ECS Managed Instances vs EC2, ECS vs EKS, task role vs execution role, EFS vs EBS vs bind mounts,
binpackvsspread, which ECR feature (lifecycle policy, immutability, replication, pull through cache) meets "lowest cost", "least operational overhead" or "most secure".
Typical question shapes: a short scenario with one deciding constraint; a stopped-task reason or service event message to diagnose; "Which TWO actions…" with five options; the exact parameter, CLI command or IAM action that does the job.
Domain map: Deployment → §1, §6, §7, §9 · Development with AWS Services → §2, §4, §5, §10 · Security (roles and secrets show up inside development and deployment questions) → §3, §4, §9 · Troubleshooting → §2, §5, §8, §9.
Out of scope: operating-system administration of container hosts (kernel tuning, Docker daemon internals, iptables beyond the one-line IMDS block shown in §3) and network infrastructure design (subnet layout, routing, NAT and endpoint architecture). Subnets, NAT gateways and VPC endpoints appear here only as blockers you must recognise.
| # | Topic | Why it is on the exam |
|---|---|---|
| 1 | ECS building blocks and capacity options | Cluster, task definition, task and service vocabulary; Fargate vs ECS Managed Instances vs EC2; Fargate Spot; Copilot's end of support and Express Mode |
| 2 | Task definitions and container settings | CPU and memory limits, essential containers, health checks, logging, port mappings and storage choices decide whether a task starts and stays up |
| 3 | IAM roles for tasks and container instances | Task role vs task execution role vs instance role is the most tested container security distinction; ECS Exec permissions |
| 4 | Environment variables and secrets | environment, S3 environment files and secrets from Secrets Manager or Parameter Store, and why a rotated secret needs a new task |
| 5 | Services standalone tasks and load balancing | Replica vs daemon services, ALB/NLB integration, dynamic host ports vs awsvpc, EventBridge-launched tasks |
| 6 | Service auto scaling and cluster capacity | Application Auto Scaling policies and metrics, SQS backlog scaling, capacity providers for EC2 |
| 7 | Rolling and blue green deployments | Deployment arithmetic, circuit breaker and alarm rollbacks, ECS-native blue/green, linear and canary, CodeDeploy |
| 8 | Task placement strategies and constraints | binpack, spread, random, distinctInstance, memberOf and why tasks stay PENDING |
| 9 | Amazon ECR authentication images and lifecycle | Login tokens, push/pull permissions, repository policies, lifecycle and archive rules, immutability, scanning, replication |
| 10 | Amazon EKS basics | When to pick EKS, node options including Auto Mode, Pod Identity vs IRSA, access entries, storage drivers |
1. ECS building blocks and capacity options
Concept
- A container image bundles application code with its runtime, libraries and default configuration; a container is a running instance of that image. Containers on one host share the host kernel instead of each booting a guest operating system, so they start quickly and pack densely. The developer loop: describe the image in a
Dockerfile, build it, push it to a registry such as Amazon ECR (§9), and let an orchestrator pull and run it. - Amazon ECS is AWS's own orchestrator. Four objects carry most questions:
- Cluster: a logical boundary that holds capacity plus the tasks and services running on it.
- Task definition: a versioned JSON blueprint of the containers that run together (§2).
- Task: one running copy of a task definition; its containers start and stop as a unit.
- Service: a controller that keeps a desired number of tasks running, replaces failed ones, registers them with load balancers (§5), scales them (§6) and rolls out new versions (§7).
- Capacity is where tasks actually run:
- AWS Fargate: serverless capacity sized per task; nothing to patch, scale or right-size.
- Amazon ECS Managed Instances: EC2 instances in your account that AWS selects, provisions, patches and scales for you.
- Amazon EC2 (self-managed): instances you run, usually from the ECS-optimized AMI; the ECS container agent on each one registers it with the cluster as a container instance.
- External instances (ECS Anywhere): your own servers outside AWS.
- Amazon EKS (§10) is the managed Kubernetes alternative; Fargate can host both ECS tasks and EKS pods.
Exam facts
- ⚠️ Changed: ECS documents three capacity types for clusters: ECS Managed Instances (launched 2025-09-30), Fargate and self-managed EC2 (plus external instances). Older material may say ECS has only two launch types, EC2 and Fargate.
- Choosing the capacity:
- Fargate: least operational overhead, spiky or unpredictable load, strongest per-task isolation (a Fargate task shares no kernel, CPU, memory or ENI with other tasks). You pay for the task's vCPU and memory.
- ECS Managed Instances: EC2 capabilities (GPUs, specific instance families, large memory, privileged Linux capabilities) without running the fleet. AWS picks cost-optimised instances or honours your attribute-based requirements, packs several tasks per instance, patches it (Bottlerocket, no SSH, no custom AMIs) and drains each instance after 14 days, so tasks that must run longer than 14 days don't fit. You pay for the whole EC2 instance plus a management charge; Compute and EC2 Instance Savings Plans apply.
- Self-managed EC2: full control (custom AMIs, Capacity Reservations, host daemons), often cheapest at steady high utilisation, but you patch and scale the instances yourself (§6).
- Launch type or capacity provider strategy: a service or
RunTaskcall sets eitherlaunchType(EC2,FARGATE,EXTERNAL;MANAGED_INSTANCESis listed, but Managed Instances must be used through a capacity provider strategy) orcapacityProviderStrategy, never both. With neither, the cluster's default capacity provider strategy applies. - Capacity provider strategy items:
base= minimum tasks placed on that provider first (0–100,000; only one provider in the strategy may set it);weight= relative share of the remaining tasks (0–1,000). Defaultweightis 1 in the console but 0 through the API/CLI, and a strategy whose providers all have weight 0 fails. At most 20 capacity providers per strategy and per cluster. One strategy holds only one kind of provider: Fargate/Fargate Spot, Auto Scaling group providers, or Managed Instances. - Fargate Spot: interruption-tolerant tasks on spare capacity at a discount. AWS reclaims capacity with a 2-minute warning, sent as an
ECS Task State Changeevent (stopCodeSpotInterruption) and asSIGTERMto the containers; afterstopTimeout(default 30 s, max 120 s) the container getsSIGKILL. Fargate never swaps in On-Demand capacity by itself: abaseonFARGATEkeeps a guaranteed floor. - Fargate platform versions:
LATESTis Linux1.4.0and Windows1.0.0. New tasks always start on the newest patched revision; running tasks on a retired revision are stopped after a notice (account settingfargateTaskRetirementWaitPeriod: 0, 7 or 14 days, default 7; since 2025-12-18 EC2 event windows can pin retirement times). Service tasks are replaced automatically; standalone tasks are not. - Self-managed EC2 capacity: each instance needs the container agent and the container instance role (§3). The agent joins the cluster named in
/etc/ecs/ecs.config(ECS_CLUSTER=<name>, normally written by user data, for exampleecho "ECS_CLUSTER=orders" >> /etc/ecs/ecs.config); without it the instance registers in the cluster nameddefault. A container instance offers 1,024 CPU units per CPU core. - Quotas (per Region): 10,000 clusters per account (adjustable); 5,000 services per cluster; 5,000 tasks per service; 5,000 container instances per cluster; 10 tasks per
RunTaskcall. Fargate concurrency quotas are vCPU-based (separate On-Demand and Spot vCPU counts) and start low for new accounts. - Fast paths for new apps:
- ⚠️ Changed: the AWS Copilot CLI reached end of support on 2026-06-12 (announced 2026-03-06): no further updates, security patches or support; applications it deployed keep running as ordinary CloudFormation stacks. AWS recommends Amazon ECS Express Mode or the AWS CDK instead. Older material may say Copilot is the supported, recommended CLI for building and deploying ECS applications.
- Amazon ECS Express Mode (2025-11-21): supply a container image, a task execution role and an infrastructure role; ECS creates a Fargate service with an AWS-provided URL, an HTTPS Application Load Balancer (shared by Express Mode services with the same network settings), auto scaling, monitoring and networking. No extra charge; every resource stays visible and editable in your account.
- AWS CDK: L3 patterns such as
ApplicationLoadBalancedFargateServiceandQueueProcessingFargateService(seenotes/cdk-guide.md§2).
- ⚠️ Changed: exam guide version 2.1 (2024-12-12) lists Amazon ECR, Amazon ECS and Amazon EKS under Containers and removed AWS Copilot from the in-scope services (Fargate is tested as ECS capacity). Older material may say Copilot is an in-scope DVA-C02 service, as version 1.x of the exam guide listed it.
Developer angle
Create a Fargate cluster with a default strategy, register a task definition, run a service across Fargate and Fargate Spot, and launch a one-off task:
aws ecs create-cluster --cluster-name orders \
--capacity-providers FARGATE FARGATE_SPOT \
--default-capacity-provider-strategy capacityProvider=FARGATE,base=2,weight=1 capacityProvider=FARGATE_SPOT,weight=3
aws ecs register-task-definition --cli-input-json file://orders-api-taskdef.json
aws ecs create-service --cluster orders --service-name orders-api \
--task-definition orders-api:7 --desired-count 6 \
--capacity-provider-strategy capacityProvider=FARGATE,base=2,weight=1 capacityProvider=FARGATE_SPOT,weight=3 \
--network-configuration "awsvpcConfiguration={subnets=[subnet-0a1b2c3d4e5f60718,subnet-0f1e2d3c4b5a69788],securityGroups=[sg-0123456789abcdef0],assignPublicIp=DISABLED}"
# one-off job (database migration) as a standalone task
aws ecs run-task --cluster orders --task-definition orders-migrate:3 --count 1 \
--launch-type FARGATE \
--network-configuration "awsvpcConfiguration={subnets=[subnet-0a1b2c3d4e5f60718],securityGroups=[sg-0123456789abcdef0],assignPublicIp=DISABLED}"
With 6 tasks and that strategy, the first 2 go to FARGATE (base), and the remaining 4 split 1:3, so 1 more On-Demand task and 3 Spot tasks.
Spot-friendly (and deployment-friendly) code handles SIGTERM and finishes in-flight work before stopTimeout runs out:
import signal, sys
def shutdown(signum, frame):
# stop taking new work, finish or checkpoint in-flight items, flush logs, then exit
sys.exit(0)
signal.signal(signal.SIGTERM, shutdown)
Traps
- Trap: "Set
launchType: FARGATEand also addFARGATE_SPOTin the same request." → Reality:launchTypeandcapacityProviderStrategyare mutually exclusive; use a strategy with both providers. - Trap: "Fargate Spot falls back to On-Demand when Spot capacity runs out." → Reality: tasks wait for Spot capacity; only a
base/weightonFARGATEguarantees On-Demand tasks. - Trap: "A provider added to a strategy through the CLI without a weight still gets tasks." → Reality: the API default weight is 0, so it receives no tasks beyond a
base. - Trap: "ECS Managed Instances is billed like Fargate, per task." → Reality: you pay for the whole EC2 instance plus a management charge; several tasks share an instance.
- Trap: "A long-running singleton that must never restart belongs on Managed Instances." → Reality: instances are drained after 14 days (and Fargate retires tasks for patching too); only self-managed EC2 leaves the instance lifetime to you.
- Trap: "AWS restarts standalone Fargate tasks after a platform retirement." → Reality: only service tasks are replaced; standalone tasks just stop.
- Trap: "Use the Copilot CLI to bootstrap a new ECS service." → Reality: Copilot is past end of support (2026-06-12); use ECS Express Mode or the CDK.
- Trap: "Containers are a security boundary, so co-located EC2 tasks can't see each other." → Reality: only Fargate gives per-task isolation by default; on EC2, Managed Instances and Anywhere, tasks share the host (Managed Instances can be set to one task per instance).
Practice questions
-
A team runs a stateless order API as an Amazon ECS service on AWS Fargate with 8 tasks. To cut cost, most tasks should run on spare capacity, but at least 2 tasks must keep serving even when no spare capacity is available, and the team wants the LEAST operational overhead. What should a developer configure?
- A. A capacity provider strategy with
FARGATE(base=2,weight=1) andFARGATE_SPOT(weight=3). - B. The
FARGATElaunch type plus a second service that uses onlyFARGATE_SPOTwith a desired count of 6. - C. An Auto Scaling group of Spot Instances registered as a capacity provider with
binpackplacement. - D. A capacity provider strategy with only
FARGATE_SPOTand a desired count raised from 8 to 12.
- A. A capacity provider strategy with
-
A machine-learning team needs a specific GPU instance family for an inference service on Amazon ECS. They do not want to patch, scale or replace EC2 instances themselves, and each deployment restarts the tasks at least weekly. They are evaluating Amazon ECS Managed Instances. Which TWO statements are correct? (Select TWO.)
- A. AWS provisions, patches and scales the instances, which run in the team's own AWS account.
- B. Each task runs in a dedicated micro-VM and is billed per task vCPU and memory, as on Fargate.
- C. A custom capacity provider can require instance attributes such as an accelerator type.
- D. The team can install a custom AMI with its own GPU drivers and connect to instances over SSH.
- E. Instances keep running until the team drains them, because AWS never replaces healthy instances.
Answers
- A. The
baseguarantees 2 On-Demand tasks and the weights put about three of every four further tasks on Fargate Spot, all in one managed service. B splits one workload into two services that scale and deploy separately; C adds instance management the team wanted to avoid (and binpack is irrelevant to the floor); D has no guaranteed capacity at all when Spot is reclaimed. - A, C. Managed Instances are EC2 instances that AWS operates in your account, and custom capacity providers accept attribute-based requirements such as accelerators. Billing is per instance plus a management charge (B describes Fargate); custom AMIs and SSH aren't available (D); instances are drained after 14 days for patching (E).
2. Task definitions and container settings
Concept
- A task definition is the contract between your application and ECS: which images run together, how much CPU and memory they get, how they network, log, report health, start in order and mount storage.
- Definitions are registered under a family with an increasing revision (
orders-api:1,orders-api:2, …). A revision is immutable: any change, even one environment variable, means registering a new revision and pointing the service at it (§7). - Put containers that must live, scale and deploy together in one task (an app plus its log router or proxy sidecar). Put components that scale independently in separate task definitions and services.
Exam facts
familyandcontainerDefinitionsare required. Limits: 10 container definitions per task definition (not adjustable); 64 KiB per task definition; 1,000,000 revisions per family (deregistered or deleted revisions still count). Old revisions:DeregisterTaskDefinitionmarks themINACTIVE, thenDeleteTaskDefinitionsremoves them.requiresCompatibilities(for exampleFARGATE,EC2,MANAGED_INSTANCES) makes ECS validate the definition for that capacity at registration time; Managed Instances accepts Fargate definitions built for platform version 1.4.0.- Network mode (
networkMode):awsvpc: the task gets its own ENI, private IP and security groups; required on Fargate; recommended on EC2.bridge: Docker's virtual network on the instance; the Linux default on EC2 when unset; allows dynamic host ports (§5).host: container ports bind directly to the instance ENI; a host port can be used by only one task per instance.none: no external networking. ECS Managed Instances supportsawsvpcandhost.
- CPU and memory:
- Fargate needs task-level
cpuandmemoryfrom the valid pairs, for example 256 CPU units (0.25 vCPU) → 512 MiB, 1 GB or 2 GB; 1 vCPU → 2–8 GB; 4 vCPU → 8–30 GB; 16 vCPU → 32–120 GB. 1,024 CPU units = 1 vCPU. - ⚠️ Changed: Linux Fargate tasks can reach 32 vCPU with 60, 120 or 244 GB of memory (platform version 1.4.0). Older material may say 16 vCPU and 120 GB is the Fargate maximum.
- Container
memoryis a hard limit: a container that goes over it is killed (stopped reasonOutOfMemoryError: container killed due to memory usage).memoryReservationis a soft limit that ECS reserves for placement; the container may burst above it up tomemoryor whatever the instance has free. Without task-level memory (EC2), every container needsmemoryormemoryReservation; with both,memorymust be larger.
- Fargate needs task-level
- Container lifecycle:
essentialdefaults totrue: when an essential container stops, ECS stops the whole task. Every task needs at least one essential container; mark one-shot helpersessential: false.dependsOnconditions:START,COMPLETE(dependency exited),SUCCESS(exited with 0),HEALTHY(passed itshealthCheck).COMPLETEandSUCCESScan't point at an essential container.startTimeoutandstopTimeoutare capped at 120 s on Fargate;stopTimeoutdefaults to 30 s (the grace period betweenSIGTERMandSIGKILL).
healthCheck:command(CMDorCMD-SHELL; exit 0 = healthy),interval30 s (5–300),timeout5 s (2–60),retries3 (1–10),startPeriodoff by default (0–300 s).startPeriodis a bootstrap grace period: failed checks inside it don't count towardretries; a check that passes inside it marks the container healthy, and from then on every failure counts (for load balancer health checks, use the service'shealthCheckGracePeriodSeconds, §5). Container health feeds service health and the deployment circuit breaker (§7).portMappings:containerPort, optionalhostPort(inawsvpcandhostmode it must equalcontainerPortor be left out),protocoltcp(default) orudp;nameandappProtocolare used by Service Connect.- Logging (
logConfiguration):- Fargate supports
awslogs,awsfirelens(FireLens: a Fluent Bit or Fluentd log-router sidecar that ships logs to AWS services such as Data Firehose, Kinesis Data Streams, OpenSearch Service and S3, or to partner destinations) andsplunk; EC2 addsjson-file,syslog,journald,gelfandfluentd. awslogsneedsawslogs-groupandawslogs-region;awslogs-stream-prefixis required on Fargate and names streamsprefix/container-name/task-id.awslogs-create-group: "true"also needslogs:CreateLogGroupon the role that writes the logs.- ⚠️ Changed: since 2025-06-25 the default log delivery
modeisnon-blocking: logs wait in an in-memory buffer (max-buffer-size, default10m) and are dropped when it fills, so a logging outage no longer freezes the app. Setmodetoblockingper container or thedefaultLogDriverModeaccount setting to get the old behaviour. Older material may say ECS log delivery is blocking by default.
- Fargate supports
- Image tags vs digests: at deployment ECS resolves the
imagetag to a digest, and every task of that deployment runs the same digest (versionConsistency, defaultenabled). Re-pushing:latestchanges nothing until the next deployment (§7). - Fargate restrictions:
privileged,links,placementConstraints,dockerVolumeConfiguration,ipcModeand custom DNS settings are rejected; the only Linux capability you can add isCAP_SYS_PTRACE;nofileulimit defaults to 65535 (up to 1048576). - ⚠️ Changed: ECS attaches one new Amazon EBS volume per task, configured at launch, on Fargate (platform 1.4.0+), EC2 (Nitro instances) and Managed Instances. Older material may say ECS tasks can't use EBS volumes and that EFS is the only persistent storage on Fargate.
- Storage options:
| Option | Runs on | Lifetime | Typical use |
|---|---|---|---|
Bind mount (volumes + mountPoints, volumesFrom) |
Fargate, EC2, Managed Instances | Ephemeral: task lifetime on Fargate, instance lifetime on EC2 | Share scratch files between containers of one task, e.g. a log-shipping sidecar reading the app's log directory |
| Fargate ephemeral storage | Fargate | Ephemeral | 20 GiB default, raise to 200 GiB with ephemeralStorage.sizeInGiB (platform 1.4.0+); encrypted with AES-256 using an AWS owned key or your KMS key; the image itself uses part of it |
| Amazon EFS | Fargate (1.4.0+), EC2, Managed Instances | Persistent, shared across tasks and AZs | Shared uploads or content; access points; IAM authorization through the task role; NFS on port 2049 |
| Amazon EBS | Fargate (1.4.0+, Linux), EC2 (Nitro), Managed Instances | One new volume per task, configured at launch (configuredAtLaunch); always deleted when a service task ends, can be kept for a standalone task |
High-IOPS block storage such as a database or ETL scratch; needs the ECS infrastructure role |
| Amazon S3 Files | Fargate, Managed Instances; EC2 ⚠️ unverified (AWS pages disagree) | Persistent; synchronised with an S3 bucket | File-system access to data already in S3; s3filesVolumeConfiguration, task role mandatory, TLS always on |
| Docker volumes | EC2 only | Per instance or driver | Third-party volume drivers |
| FSx for Windows File Server | EC2 (Windows) | Persistent | SMB shares for Windows tasks |
Developer angle
A Fargate task definition with a health check, ordered start-up, CloudWatch logging and extra ephemeral storage:
{
"family": "orders-api",
"requiresCompatibilities": ["FARGATE"],
"networkMode": "awsvpc",
"cpu": "1024", "memory": "2048",
"runtimePlatform": { "operatingSystemFamily": "LINUX", "cpuArchitecture": "ARM64" },
"executionRoleArn": "arn:aws:iam::111122223333:role/ordersTaskExecutionRole",
"taskRoleArn": "arn:aws:iam::111122223333:role/ordersApiTaskRole",
"ephemeralStorage": { "sizeInGiB": 40 },
"containerDefinitions": [
{ "name": "migrate", "essential": false,
"image": "111122223333.dkr.ecr.eu-west-1.amazonaws.com/orders-migrate:1.8.2" },
{
"name": "api",
"image": "111122223333.dkr.ecr.eu-west-1.amazonaws.com/orders-api:1.8.2",
"essential": true,
"portMappings": [{ "containerPort": 8080, "protocol": "tcp" }],
"dependsOn": [{ "containerName": "migrate", "condition": "SUCCESS" }],
"healthCheck": { "command": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"],
"interval": 30, "timeout": 5, "retries": 3, "startPeriod": 60 },
"stopTimeout": 60,
"logConfiguration": { "logDriver": "awslogs", "options": {
"awslogs-group": "/ecs/orders-api", "awslogs-region": "eu-west-1", "awslogs-stream-prefix": "orders" } }
}
]
}
Revisions and stopped-task diagnosis:
aws ecs register-task-definition --cli-input-json file://orders-api-taskdef.json # returns orders-api:<n+1>
aws ecs deregister-task-definition --task-definition orders-api:5
aws ecs delete-task-definitions --task-definitions orders-api:5
# why did a task stop? stoppedReason, stopCode and per-container exit codes
aws ecs describe-tasks --cluster orders --tasks 0f1e2d3c4b5a49788796a5b4c3d2e1f0 \
--query 'tasks[0].{reason:stoppedReason,code:stopCode,containers:containers[].{name:name,exit:exitCode,why:reason}}'
Traps
- Trap: "Edit revision 7 so the running service picks up the new variable." → Reality: revisions are immutable; register revision 8 and update the service.
- Trap: "
memoryReservationstops a runaway container." → Reality: only the hardmemorylimit kills a container; the soft reservation is for placement. - Trap: "A sidecar that crashes always takes the app down." → Reality: only if the sidecar is
essential; setessential: falsefor helpers. - Trap: "Map container port 8080 to host port 80 on Fargate." → Reality: in
awsvpcmodehostPortmust equalcontainerPortor be omitted; the load balancer listener provides port 80. - Trap: "Put twelve microservices into one task definition." → Reality: the limit is 10 containers, and they would scale and deploy as one unit anyway.
- Trap: "One EBS volume can be shared by all tasks of a service." → Reality: ECS attaches one new volume per task; use EFS (or S3 Files) for shared data.
- Trap: "Fargate ephemeral storage survives a task restart." → Reality: it lives and dies with the task; use EFS for anything durable (a service task's EBS volume is deleted when the task ends too; only a standalone task can keep its volume, with
deleteOnTermination: false). - Trap: "Some log lines are missing under load, so CloudWatch dropped them." → Reality: with the default
non-blockingmode a full buffer drops lines; raisemax-buffer-sizeor chooseblockingif you can't lose logs.
Practice questions
-
An ECS task on Amazon EC2 runs a reporting container with
memoryReservationset to 512 MiB andmemoryset to 1024 MiB. During month-end processing the task stops with the reasonOutOfMemoryError: container killed due to memory usage. Other containers on the instance have plenty of headroom. What should the developer change?- A. Raise
memoryReservationto 1024 MiB so ECS reserves more memory at placement time. - B. Add a
healthCheckwith moreretriesso ECS waits longer before stopping the container. - C. Mark the reporting container as
essential: falseso the task keeps running when it stops. - D. Raise the container's
memoryhard limit (and task memory, if set) to cover the month-end peak.
- A. Raise
-
A developer is choosing storage for two containers that run in the same ECS service on AWS Fargate. The web container writes log files that a log-shipping sidecar in the same task must read, and all tasks must share a common set of uploaded images that survive task replacement. Which TWO configurations meet these requirements? (Select TWO.)
- A. An Amazon EBS volume attached to each task through
configuredAtLaunch. - B. A bind mount volume shared by both containers of the task through
mountPoints. - C. A Docker volume with the
localdriver declared indockerVolumeConfiguration. - D. An Amazon EFS volume mounted by every task of the service.
- E. Fargate ephemeral storage increased to 200 GiB with
ephemeralStorage.
- A. An Amazon EBS volume attached to each task through
Answers
- D. The container is killed because it exceeds its hard limit; raising
memory(and task memory) lets it use the instance headroom. A only changes placement reservation and leaves the 1024 MiB ceiling; B has nothing to do with memory kills; C would leave the task running without the work it exists to do. - B, D. A bind mount shares files between containers of one task, and EFS gives every task the same persistent files. A creates a separate volume per task (and deletes it when the service task ends); C isn't supported on Fargate; E is per task and disappears with it.
3. IAM roles for tasks and container instances
Concept
Three roles answer three different questions, and exam distractors swap them:
- Task role (
taskRoleArn): what may my application code call? (S3, DynamoDB, SQS…) - Task execution role (
executionRoleArn): what may ECS itself do to start my task? (pull the image, write logs, fetch secrets and environment files) - Container instance role (instance profile on self-managed EC2): what may the ECS agent on the host do to join the cluster and report state?
Around them sit the infrastructure role (trusts ecs.amazonaws.com, passed with iam:PassedToService = ecs.amazonaws.com; ECS manages EBS volumes, Service Connect TLS, VPC Lattice targets, ECS-native blue/green listener changes, Managed Instances and Express Mode resources for you), the service-linked role AWSServiceRoleForECS (ENIs, target registration, scaling; created automatically) and roles that other services use to start tasks, such as the EventBridge role (§5).
Exam facts
- Task role:
- Trust principal
ecs-tasks.amazonaws.com; recommended confused-deputy conditionsaws:SourceAccountandaws:SourceArn(arn:aws:ecs:<region>:<account>:*; a specific cluster ARN isn't supported there, so use the wildcard). - One role per task definition gives each service least privilege; tasks of different services on the same host still get different credentials.
- Delivery: the agent sets
AWS_CONTAINER_CREDENTIALS_RELATIVE_URIin each container; SDKs and the CLI call169.254.170.2plus that path (the container credential provider, which comes before IMDS in every default chain, seenotes/ec2-mastery-guide.md§1). Credentials are valid for 6 hours by default and rotated automatically. - Code in the task uses the task role exclusively; it does not inherit the instance role's permissions through the SDK. CloudTrail shows
assumed-role/<role-name>/<task-id>. - Don't call
sts:AssumeRoleon the role the task already has; if a role must assume itself, its trust policy has to allow that explicitly.
- Trust principal
- Task execution role:
- Trust
ecs-tasks.amazonaws.com; AWS managed policyAmazonECSTaskExecutionRolePolicycovers ECR pulls andawslogs; the credentials go to the ECS container agent or Fargate agent and are never visible to your containers. - Required on Fargate, Managed Instances and external instances to pull private ECR images (also cross-account) and to send
awslogslogs; required on Fargate and EC2 for task-definition secrets from Secrets Manager or Parameter Store, private registry credentials (repositoryCredentials), environment files in S3 and Runtime Monitoring. - Add per feature:
secretsmanager:GetSecretValue,ssm:GetParameters,kms:Decrypt(customer managed keys),s3:GetObjectpluss3:GetBucketLocation(environment files),logs:CreateLogGroup(ifawslogs-create-groupis used). - ⚠️ Changed: secrets referenced in a task definition are fetched with the task execution role on Fargate and on EC2, and on Fargate that same role pulls ECR images and writes
awslogslogs. Older material may say the EC2 instance profile fetches Secrets Manager and Parameter Store values, pulls the image and sends the logs.
- Trust
- Container instance role (
ecsInstanceRole, managed policyAmazonEC2ContainerServiceforEC2Role, trustec2.amazonaws.com, attached through an instance profile): lets the agent register the instance and talk to the control plane (ecs:RegisterContainerInstance,ecs:DiscoverPollEndpoint,ecs:Poll,ecs:Submit*); its optional ECR andlogs:CreateLogStream/logs:PutLogEventspermissions are used by EC2 tasks that have no execution role. Keep it minimal: containers on the host can reach it. - Isolation on shared hosts: on EC2, Managed Instances and external instances there is no task-level isolation; containers can read the instance role through IMDS unless you block it:
awsvpctasks:ECS_AWSVPC_BLOCK_IMDS=truein/etc/ecs/ecs.config;bridgetasks: an iptables rule that drops traffic fromdocker+to169.254.169.254;hosttasks:ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=false. On Fargate, instance profiles aren't available to containers at all.
- Passing roles: whoever hands a task role or execution role to ECS (a deployer that registers task definitions and updates services, or the role of an EventBridge rule or schedule that runs tasks) needs
iam:PassRoleon those role ARNs; narrow it withiam:PassedToService=ecs-tasks.amazonaws.com. The AWS managed EventBridge policyAmazonEC2ContainerServiceEventsRolepairsecs:RunTaskwith exactly that condition (details on PassRole:notes/advanced-identity-guide.md§13). - ECS Exec (interactive shell or one-off command inside a running container, no SSH or open ports):
- Built on Systems Manager Session Manager; the task role needs
ssmmessages:CreateControlChannel,ssmmessages:CreateDataChannel,ssmmessages:OpenControlChannelandssmmessages:OpenDataChannel(on EC2 without a task role, the instance role is used). - Turn it on with
--enable-execute-commandoncreate-service,update-service,run-taskorstart-task; it applies to new tasks only, so a service needs a new deployment. - Client: AWS CLI 2.3.6+ (or 1.22.3+) plus the Session Manager plugin; the caller needs
ecs:ExecuteCommand, which you can scope withecs:cluster,ecs:task,ecs:container-nameandecs:ResourceTag/<key>;ecs:enable-execute-commandlets a policy forbid turning the feature on. - Commands run as
root; the session times out after 20 idle minutes (fixed);readonlyRootFilesystemisn't supported; private subnets need anssmmessagesVPC endpoint; requires Fargate platform 1.4.0+ or EC2 agent 1.50.2+. - Audit:
ExecuteCommandevents in CloudTrail; command output can be logged to CloudWatch Logs or S3 through the cluster'sexecuteCommandConfiguration(the task role then also needs those write permissions).
- Built on Systems Manager Session Manager; the task role needs
Developer angle
Trust policy shared by task roles and execution roles (with confused-deputy protection):
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow", "Principal": { "Service": "ecs-tasks.amazonaws.com" }, "Action": "sts:AssumeRole",
"Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:ecs:eu-west-1:111122223333:*" },
"StringEquals": { "aws:SourceAccount": "111122223333" } }
}]
}
Policy for the CI role that deploys the service (it passes both roles, and only to ECS tasks):
{
"Version": "2012-10-17",
"Statement": [
{ "Effect": "Allow", "Resource": "*",
"Action": ["ecs:RegisterTaskDefinition", "ecs:DescribeTaskDefinition", "ecs:UpdateService", "ecs:DescribeServices"] },
{ "Effect": "Allow", "Action": "iam:PassRole",
"Resource": ["arn:aws:iam::111122223333:role/ordersApiTaskRole", "arn:aws:iam::111122223333:role/ordersTaskExecutionRole"],
"Condition": { "StringEquals": { "iam:PassedToService": "ecs-tasks.amazonaws.com" } } }
]
}
Check identity from inside a container and open a shell with ECS Exec:
aws sts get-caller-identity # expect assumed-role/ordersApiTaskRole/<task-id>
echo "$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"
aws ecs update-service --cluster orders --service orders-api \
--enable-execute-command --force-new-deployment
aws ecs describe-tasks --cluster orders --tasks 0f1e2d3c4b5a49788796a5b4c3d2e1f0 \
--query 'tasks[0].{exec:enableExecuteCommand,agents:containers[].managedAgents}'
aws ecs execute-command --cluster orders --task 0f1e2d3c4b5a49788796a5b4c3d2e1f0 \
--container api --interactive --command "/bin/sh"
Traps
- Trap: "Code in a Fargate task gets
AccessDeniedfrom S3, so adds3:GetObjectto the execution role." → Reality: application calls use the task role; the execution role is only for ECS's own start-up work. - Trap: "Add
ecr:BatchGetImageto the task role so Fargate can pull the image." → Reality: image pulls on Fargate use the execution role. - Trap: "On EC2 the instance role already covers task-definition secrets." → Reality:
secretsneed a task execution role withGetSecretValue/GetParameters. - Trap: "Trust
ecs.amazonaws.comin the task role." → Reality: task and execution roles trustecs-tasks.amazonaws.com;ecs.amazonaws.comis the principal of the infrastructure role. - Trap: "Restrict the trust policy's
aws:SourceArnto one cluster ARN." → Reality: not supported; use the account-widearn:aws:ecs:<region>:<account>:*pattern. - Trap: "A broad instance role is harmless because tasks use task roles." → Reality: containers on EC2 can still reach IMDS unless you block it; keep the instance role minimal.
- Trap: "Enable ECS Exec on the service and connect to the tasks already running." → Reality: it applies only to tasks started after the change; force a new deployment.
- Trap: "ECS Exec failed, so open port 22 in the task's security group." → Reality: ECS Exec needs no inbound port; check the task role's
ssmmessagespermissions, the Session Manager plugin and outbound access or thessmmessagesendpoint.
Practice questions
-
A Python service runs as an Amazon ECS task on AWS Fargate. It must read objects from one S3 bucket. The developer attached a policy allowing
s3:GetObjecton that bucket to the role named in the task definition'sexecutionRoleArn, but the application'sGetObjectcalls still fail withAccessDenied. The image pulls and logs work. What is the MOST secure fix?- A. Attach
AmazonS3ReadOnlyAccessto the execution role so that it covers every bucket. - B. Create a task role trusted by
ecs-tasks.amazonaws.comwiths3:GetObjecton the bucket and set it astaskRoleArn. - C. Store an IAM user's access keys in Secrets Manager and inject them through the
secretsparameter. - D. Add a bucket policy that allows
s3:GetObjectfor theecs-tasks.amazonaws.comservice principal.
- A. Attach
-
A developer turned on ECS Exec for an existing Amazon ECS service on AWS Fargate with
update-service --enable-execute-command. Runningaws ecs execute-commandagainst a task that started last week fails, and the task'senableExecuteCommandattribute isfalse. Which TWO actions are required to open a shell in the service's containers? (Select TWO.)- A. Add an inbound rule for port 22 to the task's security group and attach an EC2 key pair.
- B. Add the
ssmmessagespermissions to the task execution role instead of the task role. - C. Start a new deployment of the service so that tasks are launched with ECS Exec enabled.
- D. Set
readonlyRootFilesystemtotrueso that the SSM agent cannot be tampered with. - E. Grant the task role the
ssmmessageschannel permissions that the managed SSM agent needs.
Answers
- B. Application code signs requests with the task role delivered through
169.254.170.2; a resource-scoped task role is least privilege. A grants broader access to a role the code never uses; C replaces temporary credentials with long-term keys; D names a service principal, not the task's role session, and grants too broadly. - C, E. ECS Exec applies only to tasks started after it is enabled, and the agent inside the container uses the task role's
ssmmessagespermissions. A is unnecessary (no inbound ports or SSH); B is the wrong role; D breaks ECS Exec, which needs a writable root file system.
4. Environment variables and secrets
Concept
- Each container definition can receive configuration in three ways:
environment: plain name/value pairs written into the task definition itself.environmentFiles:.envobjects stored in S3 and loaded in bulk.secrets: references (valueFrom) to AWS Secrets Manager secrets or Systems Manager Parameter Store parameters; ECS fetches the values and exposes them as environment variables (log-driver secrets usesecretOptionsinlogConfiguration).
- ECS resolves all three when the container starts. The container keeps those values for its whole life, so an edited or rotated value reaches only tasks started afterwards.
- When a value must change without restarting tasks (frequent rotation, feature flags), let the code read it at runtime with the SDK through the task role and cache it.
- Which store to pick (rotation, cost, size, hierarchy) is covered in
notes/security-encryption-guide.md§8–§9; this section covers how ECS consumes them.
Exam facts
environment: visible to anyone who can callDescribeTaskDefinitionor read the console, so keep secrets out of it. A variable inenvironmentoverrides the same name from an environment file.environmentFiles: entries{ "value": "<S3 object ARN>", "type": "s3" }.- Files need the
.envextension and UTF-8; oneVARIABLE=VALUEper line, lines starting with#are comments, and quotes or spaces become part of the value. - At most 10 files per task definition. When several files define the same variable, the first one processed wins; an environment file given as a container override replaces the files from the definition.
- Needs Fargate platform 1.4.0+ (Linux) or EC2 agent 1.39.0+; not available for Windows containers.
- The task execution role needs
s3:GetObjecton the object ands3:GetBucketLocationon the bucket (add anaws:ResourceAccountcondition to guard against a deleted and re-created bucket of the same name). ECS sets no size limit, but every task downloads its own copy.
- Files need the
secretsentries are{ "name": "<ENV_VAR>", "valueFrom": "<reference>" }:- Secrets Manager: the secret ARN, optionally followed by
:json-key:version-stage:version-id; empty fields keep their colons (…:secret:prod/orders/db-AbCdEf:password::). Without a stage or ID, ECS readsAWSCURRENT. Whole secrets need Fargate 1.3.0+ or EC2 agent 1.22.0+; a JSON key or version needs Fargate 1.4.0+ (Linux; 1.0.0 Windows) or EC2 agent 1.37.0+. - Parameter Store: the parameter name works in the same Region; a parameter in another Region needs its full ARN (
arn:aws:ssm:<region>:<account>:parameter/<name>).SecureStringvalues arrive decrypted. - Permissions belong on the task execution role:
secretsmanager:GetSecretValue,ssm:GetParameters, pluskms:Decryptfor a customer managed key (§3).
- Secrets Manager: the secret ARN, optionally followed by
- Rotation: ECS never refreshes injected values. After rotating or editing a value, start new tasks:
aws ecs update-service --force-new-deploymentfor a service, or launch a new standalone task. - Exposure: injected values are readable by everything in the container, including the app's logs if it prints its environment, debugging tools and ECS Exec users. Prefer runtime retrieval for the most sensitive values.
- Private registry credentials:
repositoryCredentials.credentialsParameterholds the ARN of a Secrets Manager secret with the registry username and password (the only way on Fargate); the execution role needssecretsmanager:GetSecretValue(andkms:Decryptfor a customer managed key). - Private subnets without NAT: ECS reaches Secrets Manager or Systems Manager through interface VPC endpoints, and S3 environment files through an S3 endpoint (endpoint design:
notes/vpc-guide.md§3). ResourceInitializationErrormessages (task stops before your code runs):pull command failed: unable to pull secrets or registry auth→ AWS documents it as a network misconfiguration: no path from the task to ECR, Secrets Manager or Systems Manager (NAT, public IP or interface endpoints).The task can't pull the secret from Secrets Manager … Check whether the secret exists in the specified Region→ wrong ARN or Region, missingGetSecretValue, or a network timeout (strings such asi/o timeoutorcontext deadline exceededmean connectivity).The task can't download the environment variable files from Amazon S3→ missing route or endpoint to S3, or missing S3 permissions.
Developer angle
Container definition fragment that mixes all three mechanisms:
{
"name": "api",
"image": "111122223333.dkr.ecr.eu-west-1.amazonaws.com/orders-api:1.8.2",
"environment": [
{ "name": "LOG_LEVEL", "value": "info" },
{ "name": "PAYMENTS_URL", "value": "https://payments.internal.example.com" }
],
"environmentFiles": [
{ "value": "arn:aws:s3:::orders-config-111122223333/prod/orders-api.env", "type": "s3" }
],
"secrets": [
{ "name": "DB_PASSWORD",
"valueFrom": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:prod/orders/db-AbCdEf:password::" },
{ "name": "PARTNER_API_KEY",
"valueFrom": "arn:aws:ssm:eu-west-1:111122223333:parameter/orders/prod/partner-api-key" }
]
}
Inline policy for the task execution role (scoped to exactly what the definition references):
{
"Version": "2012-10-17",
"Statement": [
{ "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:prod/orders/db-AbCdEf" },
{ "Effect": "Allow", "Action": "ssm:GetParameters", "Resource": "arn:aws:ssm:eu-west-1:111122223333:parameter/orders/prod/partner-api-key" },
{ "Effect": "Allow", "Action": "kms:Decrypt", "Resource": "arn:aws:kms:eu-west-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab" },
{ "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::orders-config-111122223333/prod/orders-api.env" },
{ "Effect": "Allow", "Action": "s3:GetBucketLocation", "Resource": "arn:aws:s3:::orders-config-111122223333" }
]
}
Pick up a rotated secret, or read it at runtime instead (task role needs secretsmanager:GetSecretValue):
aws ecs update-service --cluster orders --service orders-api --force-new-deployment
import json, boto3
secrets = boto3.client("secretsmanager")
def db_credentials():
# called at start-up and again after an authentication failure, so a rotation needs no redeploy
response = secrets.get_secret_value(SecretId="prod/orders/db")
return json.loads(response["SecretString"])
Traps
- Trap: "Secrets Manager rotated the password, so running tasks now use the new one." → Reality: values are injected once at container start; force a new deployment or read the secret at runtime.
- Trap: "Grant
secretsmanager:GetSecretValueto the task role sovalueFromworks." → Reality: ECS fetchessecretswith the task execution role; the task role matters only for runtime SDK calls. - Trap: "
valueFrom…:secret:db-AbCdEf:passwordselects the JSON key." → Reality: the version-stage and version-ID fields must still be present as empty colons:…:password::. - Trap: "Reference a parameter from another Region by its name." → Reality: a parameter in a different Region needs the full ARN.
- Trap: "Put the database password in
environmentand restrict the console." → Reality: it is plaintext in every describe call and revision; usesecrets. - Trap: "A variable in an S3 environment file overrides the same name in
environment." → Reality:environmentwins over files, and among files the first value wins. - Trap: "Environment files work everywhere." → Reality: not on Windows containers; Fargate needs platform 1.4.0 or later.
- Trap: "
unable to pull secrets or registry authalways means a missing IAM permission." → Reality: AWS lists it as a connectivity error: no NAT, public IP or Secrets Manager/SSM/ECR endpoint from the task's subnets.
Practice questions
-
An Amazon ECS service on AWS Fargate injects a database password from AWS Secrets Manager through the
secretsparameter. After the security team rotated the secret, the running tasks keep failing to authenticate, while tasks that start later work. The team wants the new password in use within minutes with the LEAST effort and no code changes. What should a developer do?- A. Add
secretsmanager:GetSecretValuefor the secret to the task role of the service. - B. Shorten the secret's rotation schedule so that ECS refreshes the value more often.
- C. Run
aws ecs update-servicewith--force-new-deploymentfor the service. - D. Change
valueFromto reference the secret with theAWSPREVIOUSstaging label.
- A. Add
-
Fargate tasks in private subnets with no NAT gateway stop with
ResourceInitializationError: unable to pull secrets or registry auth. The VPC has interface endpoints for Amazon ECR and an S3 gateway endpoint, and images pulled fine before a Secrets Manager reference was added to the task definition. The task execution role has only theAmazonECSTaskExecutionRolePolicymanaged policy. Which TWO actions will resolve the error? (Select TWO.)- A. Add
secretsmanager:GetSecretValueon the secret to the task role. - B. Create an interface VPC endpoint for Secrets Manager with private DNS enabled in the tasks' VPC.
- C. Set
assignPublicIptoENABLEDin the service's network configuration. - D. Raise the IMDS hop limit to 2 so that containers can obtain credentials.
- E. Allow
secretsmanager:GetSecretValueon the secret (andkms:Decryptfor its customer managed key) in the task execution role.
- A. Add
Answers
- C. A new deployment starts tasks that fetch the current secret value at container start, with no code change. A changes nothing for injected values (and the task role isn't used for
valueFrom); B doesn't make ECS re-read values; D would inject the old password. - B, E. The tasks need a private path to Secrets Manager and the execution role needs permission to read (and decrypt) the secret. A puts the permission on a role ECS doesn't use for
secrets; C doesn't help in a private subnet without an internet route; D is an EC2 metadata setting that is irrelevant on Fargate.
5. Services standalone tasks and load balancing
Concept
- A service keeps long-running tasks alive at a desired count, replaces unhealthy ones, registers them with Elastic Load Balancing, and is what auto scaling (§6) and deployments (§7) act on.
- A standalone task runs until its containers exit: migrations, batch jobs, scheduled reports, event-driven processing. Something has to start it:
RunTask, an EventBridge rule, EventBridge Scheduler, Step Functions or AWS Batch. ECS does not restart it. - How a load balancer reaches a task depends on the network mode (§2): with
awsvpceach task has its own IP address and becomes aniptarget; withbridgeon EC2 the load balancer targets the instance on a dynamically assigned host port. - ECS publishes task, service and deployment state changes to EventBridge, so you can react to stopped tasks or failed deployments without polling.
Exam facts
- Scheduling strategies (the service's
schedulingStrategy, set at creation withcreate-service --scheduling-strategy):REPLICA(default whenschedulingStrategyis omitted): keeps the desired count, spread across Availability Zones by default; placement strategies apply on EC2 (§8).DAEMON: exactly one task on every active container instance that meets the placement constraints; no desired count, placement strategy or service auto scaling; not supported on Fargate (use a sidecar instead). Defaults:minimumHealthyPercent0 % via API/CLI (50 % in the console),maximumPercent100 %.
- Health: a task of a service without a load balancer counts as healthy once
RUNNING(and its essential containers pass their health checks); with a load balancer the target group must also report it healthy. healthCheckGracePeriodSeconds: how long the scheduler ignores ELB, VPC Lattice and container health checks after a task starts; default 0, up to 2,147,483,647 seconds. Use it for slow-starting apps instead of weakening the health check itself.- Load balancer choice:
- ALB (recommended): HTTP/HTTPS, path- and host-based rules (several services can share one listener), multiple target groups per service, dynamic host port mapping.
- NLB: TCP/UDP at layer 4, for example non-HTTP protocols or exposure through AWS PrivateLink.
- GWLB: inline virtual appliances such as firewalls.
- Fargate and Managed Instances services support ALB, NLB and GWLB with target type
ip. ⚠️ Changed: Classic Load Balancers are legacy (at most 1 per service, not for Fargate); current docs describe ALB, NLB and GWLB. Older material may say ECS integrates with ALB, NLB and CLB. - Up to 5 target groups per service. An existing service's load balancer settings can be changed only through the API, CLI or CloudFormation (not the console) and only with the
ECSdeployment controller; the change starts a new deployment.
bridgemode on EC2 (dynamic host ports): declare onlycontainerPort(omithostPortor set 0). Each task gets a free host port from the instance's ephemeral range (read from/proc/sys/net/ipv4/ip_local_port_range, falling back to 49153–65535 on Linux), and the ALB registers instance ID plus port, so several copies run on one instance. The instances' security group must accept that whole port range from the load balancer's security group. Reserved host ports: 22, 2375, 2376, 51678–51680; at most 100 reserved host ports per instance (dynamic ports don't count).hostmode or fixedhostPort: one task per port per instance; extra tasks fail placement with port conflicts.awsvpcmode: each task has its own ENI, private IP and security groups; the task security group allows the container port from the ALB security group, and the ALB security group allows 80/443 from clients. Up to 16 subnets and 5 security groups perawsvpcConfiguration. On EC2, ENIs per instance cap the task count (theawsvpcTrunkingaccount setting raises it on supported instance types). A Fargate task in a public subnet needsassignPublicIp: ENABLED(and an internet gateway route) to pull images; in a private subnet it needs NAT or VPC endpoints (§9).- Roles: with
awsvpc, ECS uses its service-linked role for target registration and the service'sroleparameter must be omitted. - Service-to-service calls: Service Connect (a Cloud Map namespace, short names such as
http://payments:8080, a managed proxy with metrics) or plain Cloud Map service discovery (DNS records; limits a service to 1,000 tasks). - Standalone tasks:
RunTaskstarts at most 10 tasks per call (--count);StartTasktargets up to 10 specific container instances.- EventBridge rules start tasks from events (for example an S3
Object Createdevent → a Fargate task that transcodes the new file with its task role's permissions). The rule's role needsecs:RunTaskandiam:PassRolefor the task's roles. - EventBridge Scheduler is what the ECS docs point to for scheduled tasks (more scale and features than scheduled rules): rate, cron or one-time schedules in any time zone, a retry policy and an SQS DLQ (syntax and limits:
notes/monitoring-guide.md§6). Its execution role trustsscheduler.amazonaws.comand needsecs:RunTaskplusiam:PassRole. - Standalone tasks are never restarted after a failure, Spot interruption or platform retirement: monitor them.
- Events: source
aws.ecs;ECS Task State Changeevents carrylastStatus,desiredStatus,stoppedReasonandstopCode. ECS also emits container instance, service action and deployment state change events (for exampleSERVICE_DEPLOYMENT_FAILED). Events can arrive more than once: compare theversioninsidedetailto drop duplicates. - Service event messages to recognise:
service … has reached a steady state(healthy at desired count);(task …) is unhealthy in (elb …)(target group health checks failing);service … is unable to consistently start tasks successfully(repeated start failures; ECS backs off between retries); placement messages are in §8.
Developer angle
Service behind an ALB target group (ip targets) with a start-up grace period:
aws ecs create-service --cluster orders --service-name orders-web \
--task-definition orders-web:12 --desired-count 4 --launch-type FARGATE \
--load-balancers targetGroupArn=arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/orders-web/0123456789abcdef,containerName=web,containerPort=8080 \
--health-check-grace-period-seconds 240 \
--network-configuration "awsvpcConfiguration={subnets=[subnet-0a1b2c3d4e5f60718,subnet-0f1e2d3c4b5a69788],securityGroups=[sg-0123456789abcdef0],assignPublicIp=DISABLED}"
Email the team whenever a task in the cluster stops (this target has no RoleArn, so the SNS topic's access policy must allow events.amazonaws.com to publish):
aws events put-rule --name orders-task-stopped --event-pattern file://task-stopped.json
aws events put-targets --rule orders-task-stopped \
--targets Id=ops-email,Arn=arn:aws:sns:eu-west-1:111122223333:ops-alerts
task-stopped.json:
{ "source": ["aws.ecs"], "detail-type": ["ECS Task State Change"],
"detail": { "clusterArn": ["arn:aws:ecs:eu-west-1:111122223333:cluster/orders"], "lastStatus": ["STOPPED"] } }
⚠️ Changed: since 2025-03-11 an SNS target can use an IAM execution role instead (RoleArn=<role-arn> in --targets, the role allowing sns:Publish); EventBridge falls back to the topic's access policy only when no role is set (notes/monitoring-guide.md §6). Older material may say an SNS target always needs the topic's access policy.
A Fargate task as the target of a schedule (aws scheduler create-schedule … --target file://invoice-export-target.json; expression syntax in notes/monitoring-guide.md §6):
{
"Arn": "arn:aws:ecs:eu-west-1:111122223333:cluster/reports",
"RoleArn": "arn:aws:iam::111122223333:role/scheduler-run-invoice-export",
"EcsParameters": {
"TaskDefinitionArn": "arn:aws:ecs:eu-west-1:111122223333:task-definition/invoice-export:4",
"LaunchType": "FARGATE", "TaskCount": 1,
"NetworkConfiguration": { "awsvpcConfiguration": {
"Subnets": ["subnet-0a1b2c3d4e5f60718"], "SecurityGroups": ["sg-0123456789abcdef0"], "AssignPublicIp": "DISABLED" } }
}
}
Traps
- Trap: "Register Fargate tasks in a target group of type
instance." → Reality:awsvpctasks need target typeip. - Trap: "Open only the container port on the EC2 instances behind the ALB." → Reality: with dynamic host ports the instance security group must allow the ephemeral range from the ALB security group.
- Trap: "Set
hostPort: 8080in bridge mode and run three copies per instance." → Reality: a fixed host port allows one task per instance; omithostPortfor dynamic mapping. - Trap: "Run a log agent as a
DAEMONservice on Fargate." → Reality:DAEMONisn't supported on Fargate; add the agent as a sidecar container. - Trap: "Raise the target group's unhealthy threshold so slow-starting tasks survive." → Reality: that weakens detection for the task's whole life; set
healthCheckGracePeriodSecondsinstead. - Trap: "
RunTask --count 50starts 50 tasks." → Reality: 10 per call; loop, or use a service. - Trap: "A scheduled task that failed at 02:00 is retried by ECS." → Reality: ECS never restarts standalone tasks, and the Scheduler retry policy only re-runs a failed invocation (the
RunTaskcall); alert on the stopped-task event. - Trap: "Switch the service to a different load balancer in the console." → Reality: use
update-service(CLI/SDK/CloudFormation) with theECSdeployment controller; it triggers a deployment.
Practice questions
-
A web application runs as an Amazon ECS service on AWS Fargate behind an Application Load Balancer. After each deployment the new tasks need about three minutes to warm their caches before
/healthreturns HTTP 200, and ECS keeps stopping them as unhealthy, so the deployment never finishes. Failures that happen later must still be detected quickly. What should a developer do?- A. Raise the target group's unhealthy threshold count and health check interval to their maximum values.
- B. Add a container
healthCheckwith 10 retries to the task definition's application container. - C. Replace the Application Load Balancer with a Network Load Balancer that uses TCP health checks.
- D. Set
healthCheckGracePeriodSecondson the service to a value longer than the warm-up time.
-
An Amazon ECS service on self-managed EC2 instances uses the
bridgenetwork mode behind an Application Load Balancer. The developer wants several copies of the task on each instance, with every copy receiving traffic from the load balancer. Which TWO configurations are required? (Select TWO.)- A. Omit
hostPort(or set it to 0) in the container's port mapping so each task gets a dynamic host port. - B. Change the target group to target type
ipso that the load balancer reaches each container directly. - C. Set
hostPortto the same value ascontainerPortin every task definition revision. - D. Allow the instances' ephemeral port range from the load balancer's security group in the instances' security group.
- E. Replace the Application Load Balancer with a Classic Load Balancer that listens on each host port.
- A. Omit
Answers
- D. The grace period makes the scheduler ignore failing health checks only during start-up, so later failures are still caught at the normal threshold. A slows detection for the whole life of every task; B adds a container check but the ALB check still fails; C changes protocols without fixing the warm-up problem.
- A, D. Dynamic host ports let several tasks share an instance, and the instances must accept the whole ephemeral range from the ALB. B is for
awsvpctasks (bridge-mode targets are instances plus ports); C pins one task per instance; E is legacy and doesn't support dynamic port mapping.
6. Service auto scaling and cluster capacity
Concept
- Scaling happens at two layers, and the exam checks that you keep them apart:
- Tasks: ECS Service Auto Scaling moves the service's desired count up and down; underneath, the service is an Application Auto Scaling scalable target (dimension
ecs:service:DesiredCount). - Capacity: something must provide room for those tasks. On Fargate nothing to do (AWS supplies capacity per task); on ECS Managed Instances ECS adds and removes instances; on self-managed EC2 an Auto Scaling group capacity provider with managed scaling grows and shrinks the group (EC2 Auto Scaling basics:
notes/ec2-mastery-guide.md§10).
- Tasks: ECS Service Auto Scaling moves the service's desired count up and down; underneath, the service is an Application Auto Scaling scalable target (dimension
- Without capacity scaling, a task-level scale-out on EC2 simply leaves new tasks waiting for room.
Exam facts
- Policy types for services:
- Target tracking (simplest): keep a metric at a target value. Predefined metrics:
ECSServiceAverageCPUUtilization,ECSServiceAverageMemoryUtilization,ALBRequestCountPerTarget(needs aResourceLabelofapp/<lb-name>/<lb-id>/targetgroup/<tg-name>/<tg-id>), plusECSServiceAverageCPUUtilizationHighResolutionandECSServiceAverageMemoryUtilizationHighResolution, which evaluate 20-second metrics once high-resolution metrics are enabled on the service. Custom metrics work too. - Step scaling: add or remove tasks in steps that depend on how far a CloudWatch alarm is breached; reacts fastest to sudden jumps.
- Scheduled scaling: change minimum/maximum capacity at set times for known peaks.
- Predictive scaling: learns daily and weekly patterns and scales out ahead of them; needs at least 24 hours of history (two weeks is ideal), analyses up to 14 days, forecasts 48 hours ahead in hourly blocks, refreshes every 6 hours, starts in forecast-only mode, never scales in by itself (pair it with target tracking or step scaling), and requires the service-linked role rather than a custom role on the scalable target.
- ⚠️ Changed: ECS service auto scaling offers target tracking, step, scheduled and predictive scaling, and target tracking can use 20-second high-resolution CPU and memory metrics. Older material may say ECS services scale only with target tracking, step and scheduled policies on 1-minute metrics.
- Target tracking (simplest): keep a metric at a target value. Predefined metrics:
- Queue workers: scale on backlog per task (visible messages divided by running tasks) with target tracking on that custom metric; the target is how many messages one task may have waiting given its processing time and your latency goal. Raw queue length alone doesn't account for how many tasks share the work.
- Behaviour to know:
- ECS publishes service CPU and memory metrics every minute (20-second resolution once the service's
monitoringconfiguration setsresolutionSeconds20); alarms can't exist for metrics that haven't been published yet. - CloudWatch Container Insights (cluster setting or the
containerInsightsaccount setting) adds task- and service-level metrics such asRunningTaskCount; its metrics are billed as custom metrics. - Scale-out reacts quickly; scale-in is conservative (cooldowns block it, and a scale-out alarm cancels a running scale-in cooldown).
- Scale-in is turned off while a deployment is in progress (scale-out continues), except for services using the
EXTERNALcontroller; suspend both explicitly withregister-scalable-target(DynamicScalingInSuspended/DynamicScalingOutSuspended) if needed. - Several target tracking policies on one service: scale out if any wants to, scale in only if all agree.
- Don't edit or delete the CloudWatch alarms that target tracking creates.
- A minimum capacity of 0 lets a worker service scale to zero;
ALBRequestCountPerTargettarget tracking isn't supported for the blue/green deployment type. - Application Auto Scaling starts from the actual running task count, which prevents runaway scaling when capacity is missing.
- ECS publishes service CPU and memory metrics every minute (20-second resolution once the service's
- ASG capacity providers (self-managed EC2):
- Turn on managed scaling and set
targetCapacity(percent of the group's capacity you want in use). ECS publishesCapacityProviderReservation= instances needed ÷ instances running × 100 in theAWS/ECS/ManagedScalingnamespace and attaches its own target tracking policy to the group. - With
targetCapacity100 (the default) there is no spare instance: new tasks wait until the added instances register. A lower value (for example 90) keeps headroom so tasks start sooner, at extra cost. - Scaling out from zero instances launches 2. Up to 500 tasks per cluster can wait in
PROVISIONINGfor an Auto Scaling group capacity provider. - Managed termination protection lets ECS manage instance scale-in protection so that instances still running tasks aren't terminated; AWS calls
binpack(§8) the most capacity-efficient placement strategy. - Don't attach your own scaling policies to that group, don't remove its
AmazonECSManagedtag, and give the creatorautoscaling:CreateOrUpdateTags. With managed scaling on, a capacity provider belongs to one cluster.
- Turn on managed scaling and set
- Fargate: capacity is AWS's job, but account vCPU quotas cap how many tasks can run;
You've reached the limit on the number of vCPUs you can run concurrentlymeans request a quota increase.
Developer angle
Target tracking on average CPU for a service:
aws application-autoscaling register-scalable-target --service-namespace ecs \
--resource-id service/orders/orders-api --scalable-dimension ecs:service:DesiredCount \
--min-capacity 2 --max-capacity 20
aws application-autoscaling put-scaling-policy --service-namespace ecs \
--resource-id service/orders/orders-api --scalable-dimension ecs:service:DesiredCount \
--policy-name cpu-60 --policy-type TargetTrackingScaling \
--target-tracking-scaling-policy-configuration \
'{"TargetValue":60.0,"PredefinedMetricSpecification":{"PredefinedMetricType":"ECSServiceAverageCPUUtilization"},"ScaleOutCooldown":60,"ScaleInCooldown":180}'
EC2 capacity that follows the tasks (managed scaling with 10 % headroom):
aws ecs create-capacity-provider --name orders-asg-cp \
--auto-scaling-group-provider "autoScalingGroupArn=arn:aws:autoscaling:eu-west-1:111122223333:autoScalingGroup:7d3e9f4a-1b2c-4d5e-8f90-a1b2c3d4e5f6:autoScalingGroupName/orders-ecs-asg,managedScaling={status=ENABLED,targetCapacity=90}"
# replaces the cluster's whole provider list: include every provider the cluster should keep
aws ecs put-cluster-capacity-providers --cluster orders \
--capacity-providers orders-asg-cp \
--default-capacity-provider-strategy capacityProvider=orders-asg-cp,weight=1
Traps
- Trap: "ECS Service Auto Scaling adds EC2 instances when tasks don't fit." → Reality: it only changes the desired task count; an ASG capacity provider with managed scaling (or Fargate) supplies capacity.
- Trap: "Add a CPU target tracking policy to the Auto Scaling group behind the capacity provider." → Reality: ECS manages that group's scaling policy; extra policies fight it.
- Trap: "Scale an SQS worker on
ApproximateNumberOfMessagesVisiblealone." → Reality: use backlog per task, so the target means the same thing at 2 or 200 tasks. - Trap: "Predictive scaling also scales the service back in at night." → Reality: it only scales out; keep a dynamic policy for scale-in.
- Trap: "Tune the alarms that target tracking created." → Reality: don't edit them; change the policy instead.
- Trap: "The service scales in halfway through a rolling deployment." → Reality: Application Auto Scaling pauses scale-in during ECS deployments.
- Trap: "
targetCapacity100 keeps a spare instance ready." → Reality: 100 means no headroom; lower it to keep spare capacity. - Trap: "Fargate services need a capacity provider ASG to scale." → Reality: Fargate supplies capacity per task; only vCPU quotas limit it.
Practice questions
-
An Amazon ECS service on AWS Fargate processes jobs from an Amazon SQS queue. Each task handles about 10 messages per minute, and jobs must start within 5 minutes of arriving. Queue depth swings from almost zero to tens of thousands within an hour. Which scaling approach meets the latency goal MOST reliably?
- A. Target tracking on
ECSServiceAverageCPUUtilizationwith a target value of 50 percent. - B. Target tracking on a custom backlog-per-task metric with a target of about 50 messages per task.
- C. Scheduled scaling actions that raise the maximum capacity every morning and lower it every evening.
- D. Predictive scaling on
ECSServiceAverageMemoryUtilizationin forecast-only mode.
- A. Target tracking on
-
A team runs an Amazon ECS service on self-managed EC2 instances in an Auto Scaling group. When service auto scaling raises the desired count, new tasks fail placement with
no container instance met all of its requirements, and the team has been adding instances by hand. The team wants instances added and removed automatically, and new tasks should usually start without waiting for instances to boot. Which TWO actions meet these requirements? (Select TWO.)- A. Attach a target tracking policy on average CPU utilization directly to the Auto Scaling group.
- B. Create an Auto Scaling group capacity provider with managed scaling turned on and use it in the service's capacity provider strategy.
- C. Set the capacity provider's
targetCapacitybelow 100 percent so that spare capacity stays available. - D. Change the service's scheduling strategy from
REPLICAtoDAEMON. - E. Raise the maximum capacity of the service's scalable target in Application Auto Scaling.
Answers
- B. Backlog per task ties the task count to the work waiting: with 10 messages per minute per task, 50 waiting messages each is about 5 minutes of work. A tracks CPU, which may not rise with queue depth (I/O-bound work); C can't follow unpredictable swings; D in forecast-only mode never scales, and memory doesn't reflect the backlog.
- B, C. A capacity provider with managed scaling grows and shrinks the group from the tasks' needs, and a
targetCapacitybelow 100 keeps headroom so tasks rarely wait. A conflicts with the policy ECS manages and ignores pending tasks; D runs exactly one task per instance and drops the desired count; E only lets the task count rise further without adding instances.
7. Rolling and blue green deployments
Concept
- A deployment replaces the tasks of one service revision with tasks of another. It starts when you update the service with a new task definition revision, platform version, network or load balancer settings, or ask for
--force-new-deployment; changing only the desired count does not deploy. - The deployment controller decides how:
ECS(default) supports four strategies indeploymentConfiguration.strategy:ROLLING,BLUE_GREEN,LINEARandCANARY.CODE_DEPLOY: blue/green through AWS CodeDeploy task sets and AppSpec hooks (details innotes/cicd-guide.md§8).EXTERNAL: a third-party tool drives task sets.
- Rolling updates trade spare capacity for speed through two percentages; blue/green-style strategies run both revisions side by side, test the new one, shift traffic and keep the old one ready for instant rollback.
Exam facts
- Rolling update arithmetic (
deploymentConfiguration):minimumHealthyPercent: lower bound of tasks that must stayRUNNINGand healthy, as a percent ofdesiredCount, rounded up. Default 100 % for replica services (daemon: 0 % via API, 50 % in the console).maximumPercent: upper bound of tasks inRUNNING,PENDINGorSTOPPING, rounded down. Default 200 % (daemon: 100 %).- Desired 4: 50/100 → stop 2 old, start 2 new, repeat (no spare capacity needed); 100/150 → start 2 new, then stop 2 old, repeat; 100/200 → start 4 new, then stop 4 old (needs room for 8 tasks).
- The settings must allow at least one task to stop or start; 100/100 stalls with the event
unable to stop or start tasks during a deployment because of the service deployment configuration. - The same percentages govern instance draining and Fargate task retirement. The course slides draw ranges of 0–100 % and 100–200 %; ⚠️ unverified: the API documentation states the defaults but no fixed upper bound for
maximumPercent, so learn the defaults and the rounding.
- Image digests: a deployment resolves image tags to digests once and runs that exact image everywhere; pushing a new image under the same tag needs
--force-new-deploymentto roll out. - Detecting a bad deployment (either or both; first one to trip wins; both work for rolling and ECS-native blue/green deployments on the
ECScontroller, not forCODE_DEPLOYorEXTERNAL):- Deployment circuit breaker (
deploymentCircuitBreakerwithenableandrollback): stage 1 counts tasks that never reachRUNNING; stage 2 counts failed ELB, Cloud Map and container health checks. Default threshold (BOUNDED_PERCENT, value 50) = 50 % of desired count, at least 3 and at most 200. ⚠️ Changed: the threshold is configurable (thresholdConfigurationtypeCOUNT,BOUNDED_PERCENTorUNBOUNDED_PERCENT, andresetOnHealthyTask, defaulttrue, which counts only consecutive failures). Older material may say the threshold is always half the desired count, from 3 to 200. - CloudWatch alarms (
alarmswithalarmNames,enable,rollback): fail the deployment on application metrics such as 5xx rate or latency. - On failure the rollout state becomes
FAILED(watchrolloutStateindescribe-services), ECS emitsSERVICE_DEPLOYMENT_FAILED, and with rollback on it redeploys the lastCOMPLETEDdeployment; with no completed deployment to return to, the deployment stalls.
- Deployment circuit breaker (
- ⚠️ Changed: the
ECScontroller runs blue/green (since 2025-07-17) and linear and canary deployments (since 2025-10-30) itself, with lifecycle hooks, bake time and alarm-based rollback; CodeDeploy is no longer needed for these patterns. Older material may say ECS natively supports only rolling updates and that blue/green requires the CodeDeploy controller. - ECS-native blue/green:
- ECS starts the green revision at full size next to blue, optionally sends test traffic through a test listener rule, then shifts all production traffic at once by updating the production listener rule, and keeps blue running for
bakeTimeInMinutes(required forBLUE_GREEN), during which a rollback is instant and has no downtime. - Needs, per load balancer entry,
advancedConfigurationwithalternateTargetGroupArn,productionListenerRule, optionaltestListenerRuleandroleArn(an infrastructure role that lets ECS edit the load balancer). Works with ALB, NLB (the traffic-shift stages take an extra 10 minutes), Service Connect and VPC Lattice. - Capacity roughly doubles during the deployment.
- ECS starts the green revision at full size next to blue, optionally sends test traffic through a test listener rule, then shifts all production traffic at once by updating the production listener rule, and keeps blue running for
- Linear and canary use the same traffic-shifting resources as native blue/green (launched with ALB and Service Connect support):
LINEARshifts equal percentages (stepPercent3–100) with a bake between steps (0–1,440 minutes);CANARYshifts a small percentage first, waits, then moves the rest. - Lifecycle hooks (
deploymentConfiguration.lifecycleHooks):- Stages:
RECONCILE_SERVICE,PRE_SCALE_UP,POST_SCALE_UP,TEST_TRAFFIC_SHIFT,POST_TEST_TRAFFIC_SHIFT,PRE_PRODUCTION_TRAFFIC_SHIFT,PRODUCTION_TRAFFIC_SHIFT,POST_PRODUCTION_TRAFFIC_SHIFT(the two production-shift stages repeat at every linear or canary step). - Lambda hooks (
targetTypeAWS_LAMBDA, the default) must returnhookStatusSUCCEEDED,FAILEDorIN_PROGRESSwithin 15 minutes;IN_PROGRESSmakes ECS call again (callBackDelaySecondssets the wait). Pause hooks (PAUSE) wait for aContinueServiceDeploymentcall to continue or roll back (not allowed atTEST_TRAFFIC_SHIFTorPRODUCTION_TRAFFIC_SHIFT). - Each stage may last up to 24 hours (a pause hook's timeout can be set up to 14 days; a whole deployment times out after 30 days); the hook's
roleArnlets ECS invoke the function.
- Stages:
- CodeDeploy blue/green (
CODE_DEPLOYcontroller): AppSpec withTaskDefinition,ContainerNameandContainerPort, predefinedECSCanary…,ECSLinear…andECSAllAtOnceconfigurations; with this controller (andEXTERNAL) you can't set customminimumHealthyPercent/maximumPercentfor EC2 tasks, and the circuit breaker doesn't apply. A CodePipeline ECS deploy action takes animagedefinitions.jsonartifact (notes/cicd-guide.md§4). ALBRequestCountPerTargettarget tracking isn't supported for the blue/green deployment type (§6).
Developer angle
Rolling deployment with both failure detectors and a watch on the rollout:
aws ecs update-service --cluster orders --service orders-api --task-definition orders-api:8 \
--deployment-configuration '{"minimumHealthyPercent":100,"maximumPercent":150,"deploymentCircuitBreaker":{"enable":true,"rollback":true},"alarms":{"alarmNames":["orders-api-5xx-rate"],"enable":true,"rollback":true}}'
aws ecs describe-services --cluster orders --services orders-api \
--query 'services[0].deployments[].{status:status,rollout:rolloutState,reason:rolloutStateReason,td:taskDefinition}'
ECS-native blue/green settings in a create-service/update-service input file:
{
"loadBalancers": [{
"targetGroupArn": "arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/orders-blue/0123456789abcdef",
"containerName": "api", "containerPort": 8080,
"advancedConfiguration": {
"alternateTargetGroupArn": "arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/orders-green/fedcba9876543210",
"productionListenerRule": "arn:aws:elasticloadbalancing:eu-west-1:111122223333:listener-rule/app/orders-alb/50dc6c495c0c9188/f2f7dc8efc522ab2/9683b2d02a6cabee",
"testListenerRule": "arn:aws:elasticloadbalancing:eu-west-1:111122223333:listener-rule/app/orders-alb/50dc6c495c0c9188/0467ef3c8400ae65/b2b1a5c7d1e04f3a",
"roleArn": "arn:aws:iam::111122223333:role/ecsInfrastructureRoleForLoadBalancers"
}
}],
"deploymentConfiguration": {
"strategy": "BLUE_GREEN", "bakeTimeInMinutes": 15,
"lifecycleHooks": [{
"hookTargetArn": "arn:aws:lambda:eu-west-1:111122223333:function:orders-green-smoke-test",
"roleArn": "arn:aws:iam::111122223333:role/ecsLifecycleHookInvokeRole",
"lifecycleStages": ["POST_TEST_TRAFFIC_SHIFT"]
}]
}
}
A smoke-test hook that checks the green revision through the test listener:
import os, urllib.request
def handler(event, context):
url = os.environ["GREEN_TEST_URL"] # e.g. the ALB test listener plus /health
try:
with urllib.request.urlopen(url, timeout=5) as response:
healthy = response.status == 200
except Exception:
healthy = False
return {"hookStatus": "SUCCEEDED" if healthy else "FAILED"}
Traps
- Trap: "Set
minimumHealthyPercent100 andmaximumPercent100 to avoid any capacity change." → Reality: ECS can neither stop nor start a task, and the deployment stalls. - Trap: "Pushing a fixed image under the same tag updates the service." → Reality: tasks keep the digest resolved at deployment; force a new deployment.
- Trap: "Turn on the circuit breaker to roll back a CodeDeploy blue/green deployment." → Reality: the circuit breaker works only with the
ECScontroller (rolling and native blue/green); CodeDeploy deployments roll back through CodeDeploy's own settings. - Trap: "ECS-native blue/green requires the
CODE_DEPLOYcontroller and an AppSpec file." → Reality: it uses theECScontroller withstrategyBLUE_GREENandadvancedConfigurationon the load balancer. - Trap: "Native blue/green works with a single target group." → Reality: it needs a primary and an alternate target group plus the production listener rule.
- Trap: "Bake time is optional for
BLUE_GREEN." → Reality:bakeTimeInMinutesmust be provided for that strategy. - Trap: "A Lambda lifecycle hook may run its tests for an hour in one invocation." → Reality: it must answer within 15 minutes; return
IN_PROGRESSto be invoked again. - Trap: "Raising the desired count deploys the newest task definition." → Reality: a desired-count change is not a deployment.
Practice questions
-
An Amazon ECS service on EC2 has a desired count of 4. The cluster has no free capacity for even one extra task, and at least 2 tasks must keep serving traffic during every deployment. Which deployment configuration should the developer use?
- A.
minimumHealthyPercent50 andmaximumPercent100. - B.
minimumHealthyPercent100 andmaximumPercent200. - C.
minimumHealthyPercent0 andmaximumPercent100. - D.
minimumHealthyPercent100 andmaximumPercent125.
- A.
-
A developer wants Amazon ECS to run blue/green deployments for an existing service behind an Application Load Balancer, test the new revision through a test listener, and keep the old revision available for 15 minutes after the switch, without using AWS CodeDeploy. Which TWO configuration changes are required? (Select TWO.)
- A. Change the service's deployment controller to
CODE_DEPLOYand create a CodeDeploy deployment group. - B. Add an
appspec.yamlfile that names the task definition, container name and container port. - C. Turn on the deployment circuit breaker with rollback so that ECS can return to the blue revision.
- D. Add an alternate target group, the production and test listener rules, and an infrastructure role in the load balancer's
advancedConfiguration. - E. Set
deploymentConfiguration.strategytoBLUE_GREENwithbakeTimeInMinutesset to 15.
- A. Change the service's deployment controller to
Answers
- A. 50 % keeps 2 of 4 tasks (rounded up) running, and 100 % allows no extra task, so ECS replaces 2 at a time inside existing capacity. B needs room for 8 tasks; C allows every task to stop at once; D rounds down to 5 tasks, one more than the cluster can hold.
- D, E. ECS-native blue/green uses the
ECScontroller with theBLUE_GREENstrategy and bake time, and needs the alternate target group, listener rules and an infrastructure role to move traffic. A and B describe the CodeDeploy approach the team wants to avoid; C is optional failure detection (it also works with native blue/green) and delivers neither the test listener nor the 15-minute bake.
8. Task placement strategies and constraints
Concept
- Every task launch on EC2 capacity makes ECS pick a host, and every scale-in makes it pick a task to stop. You steer those choices with placement strategies (preferences, best effort) and placement constraints (rules, binding).
- On Fargate AWS places tasks itself: strategies and constraints don't apply, and
placementConstraintsis rejected in Fargate task definitions. ECS Managed Instances accepts constraints but no strategies (ECS spreads tasks across AZs on a best-effort basis). - Placement explains a whole family of troubleshooting questions: a task that "never starts" is usually
PENDINGbecause no instance passes the resource checks or a constraint.
Exam facts
- Placement process for each task: (1) keep instances with enough CPU, GPU, memory and free ports for the task definition → (2) keep those that satisfy the constraints → (3) rank them by the strategies → (4) place the task.
- Strategies (
placementStrategy, each withtypeandfield):binpackoncpuormemory: fill the fullest instance that can still take the task, so the fleet stays small and cheap. On scale-in it stops the task whose instance would be left with the most free resources.spreadoninstanceId(orhost) or any attribute such asattribute:ecs.availability-zone: distribute evenly → availability. Service tasks spread relative to the service's tasks, standalone tasks relative to their task group; scale-in keeps the AZs balanced.random: no field.- Several strategies apply in order:
spreadon Availability Zone first, thenbinpackon memory inside each AZ, is the classic "available and cheap" combination. - Services default to
spreadonattribute:ecs.availability-zone; standalone tasks have no default constraint. - Strategy rules per service: the
CreateServiceAPI reference allows 5 (one developer-guide page says four, ⚠️ unverified which is current).
- Constraints (
placementConstraints):distinctInstance: no two tasks of the group share an instance. Allowed withRunTaskandCreateService, not inside task definitions, and it takes no expression. ECS judges by desired status, so a stopping task can briefly share an instance with its replacement.memberOf: candidates are filtered by a cluster query language expression; allowed in task definitions,RunTask,CreateServiceandUpdateService.- At most 10 constraints per task (task definition and run-time constraints combined). If no instance satisfies a constraint, the task stays
PENDING.
- Cluster query language:
subject operator [argument].- Subjects:
attribute:<name>,ec2InstanceId,agentVersion,agentConnected,registeredAt,runningTasksCount,task:group. - Operators:
==,!=,>,>=,<,<=,exists,!exists,in,!in,=~(Java regular expression),!~; combine withand/&&,or/||,not/!and parentheses. - Examples:
attribute:ecs.instance-type == t2.small;attribute:ecs.availability-zone in [us-east-1a, us-east-1b];attribute:ecs.instance-type =~ g2.* and attribute:ecs.availability-zone != us-east-1d;task:group == service:production(affinity);not(task:group == database)(anti-affinity).
- Subjects:
- Attributes: built-in
ecs.ami-id,ecs.availability-zone,ecs.instance-type,ecs.os-type,ecs.os-family,ecs.cpu-architecture,ecs.vpc-id,ecs.subnet-id; custom attributes such asstack=prodare added withput-attributes(names and values 1–128 characters). - ⚠️ Changed: placement constraints also work on ECS Managed Instances, which support only the attributes
ecs.subnet-id,ecs.availability-zone,ecs.instance-typeandecs.cpu-architecture; placement strategies remain EC2-only (Managed Instances doesn't support them) and Fargate supports neither. Older material may say placement strategies and constraints exist only for the EC2 launch type. DAEMONservices place exactly one task per matching instance and don't use placement strategies (§5).- Placement failure messages (service events start with
service … was unable to place a task because no container instance met all of its requirements):… has insufficient CPU units availableor not enough memory → CPU and memory reservations don't fit (1,024 CPU units per core); reduce the task size or add instances.The closest matching container-instance is already using a port required by your taskor not enough ports → fixed host ports; use dynamic ports or more instances.- Too many ports registered → over 100 reserved host ports on the instance.
- Not enough ENI attachment points →
awsvpctasks on EC2 exhausted the instance's ENIs (awsvpcTrunkinghelps on supported types). - Container instance missing required attribute → for example a log driver not listed in
ECS_AVAILABLE_LOGGING_DRIVERS, or no instance in the subnets of theawsvpcConfiguration. - No container instances were found in your cluster → nothing registered (agent,
ECS_CLUSTER, instance role).
Developer angle
A service that survives an AZ failure but packs tasks tightly inside each AZ, restricted to Graviton general-purpose instances:
aws ecs create-service --cluster orders --service-name orders-batch \
--task-definition orders-batch:4 --desired-count 6 --launch-type EC2 \
--placement-strategy '[{"type":"spread","field":"attribute:ecs.availability-zone"},{"type":"binpack","field":"memory"}]' \
--placement-constraints '[{"type":"memberOf","expression":"attribute:ecs.instance-type =~ m7g.*"}]'
Custom attributes and query-language filters:
aws ecs put-attributes --cluster orders \
--attributes name=stack,value=prod,targetId=arn:aws:ecs:eu-west-1:111122223333:container-instance/orders/0a1b2c3d4e5f60718293a4b5c6d7e8f9
# which instances would a memberOf expression select?
aws ecs list-container-instances --cluster orders --filter "attribute:stack == prod"
# one copy per instance for a standalone job
aws ecs run-task --cluster orders --task-definition cache-warmer:2 --count 3 \
--launch-type EC2 --placement-constraints type=distinctInstance
A constraint inside an EC2 task definition (allowed for memberOf only):
{
"family": "gpu-inference",
"requiresCompatibilities": ["EC2"],
"placementConstraints": [{ "type": "memberOf", "expression": "attribute:ecs.instance-type =~ g5.*" }],
"containerDefinitions": [{ "name": "model", "image": "111122223333.dkr.ecr.eu-west-1.amazonaws.com/model:3.1", "memory": 8192 }]
}
Traps
- Trap: "Use
binpackfor the highest availability." → Reality:binpackconcentrates tasks; putspreadon Availability Zone first. - Trap: "
spreadoninstanceIdalso spreads across Availability Zones." → Reality: it balances instances, not AZs; spread onattribute:ecs.availability-zone. - Trap: "Add a placement strategy to a Fargate or Managed Instances service to control AZ spread." → Reality: Fargate supports neither strategies nor constraints, Managed Instances only constraints; both spread across AZs on a best-effort basis.
- Trap: "Put
distinctInstancein the task definition." → Reality: onlymemberOfgoes into a task definition;distinctInstanceis set atRunTaskorCreateService. - Trap: "A strategy that can't be met blocks the task." → Reality: strategies are best effort; only constraints (and missing resources) keep a task
PENDING. - Trap: "
randomspreads tasks evenly." → Reality: random gives no balance guarantee; usespread. - Trap: "Fix
insufficient memoryplacement errors with amemberOfconstraint." → Reality: constraints only narrow the candidates; reduce the task's reservation or add capacity. - Trap: "Set a placement strategy on a
DAEMONservice." → Reality: daemon services run one task per instance and ignore strategies.
Practice questions
-
An Amazon ECS service runs on EC2 instances spread over three Availability Zones. It must keep serving if one Availability Zone fails, and the team wants to use as few instances as possible inside each Availability Zone to cut cost. Which placement configuration should the developer use?
- A. A single
binpackstrategy onmemory. - B. A
spreadstrategy oninstanceIdfollowed by arandomstrategy. - C. A
spreadstrategy onattribute:ecs.availability-zonefollowed by abinpackstrategy onmemory. - D. A
distinctInstanceconstraint combined with arandomstrategy.
- A. A single
-
After a developer added the constraint
memberOfwith the expressionattribute:ecs.instance-type =~ c7i.*to an EC2 task definition, new tasks stay in thePENDINGstate, while tasks from the previous revision run normally. Which TWO conditions could explain this? (Select TWO.)- A. No container instance registered in the cluster has an instance type that matches the expression.
- B. The service's
binpackstrategy on memory is binding and rejects instances that would keep free memory. - C. A
memberOfconstraint only takes effect when adistinctInstanceconstraint is also present. - D. The cluster query language can't apply the
=~operator to built-in attributes. - E. The instances that match the expression lack enough unreserved CPU or memory for the new tasks.
Answers
- C. Spreading across Availability Zones first protects against an AZ failure, and bin packing on memory within each AZ minimises the instances in use. A may place every task in one AZ; B balances instances but ignores both AZs and cost; D forces one task per instance, which maximises the instance count.
- A, E. Constraints are binding: with no matching instance, or matching instances that fail the resource check, the task can't be placed and stays
PENDING. B is wrong because strategies are best effort; C invents a dependency between constraints; D is wrong because=~is a supported pattern-match operator.
9. Amazon ECR authentication images and lifecycle
Concept
- Amazon ECR is the managed registry for container images and OCI artifacts (Helm charts, signatures, SOCI indexes). Each account has one private registry per Region,
<account-id>.dkr.ecr.<region>.amazonaws.com, holding repositories; an image is referenced as<registry>/<repo>:<tag>or, immutably,<registry>/<repo>@sha256:<digest>. Image layers are stored in Amazon S3. - Amazon ECR Public hosts public repositories shown in the ECR Public Gallery (
gallery.ecr.aws) underpublic.ecr.aws. - Access is IAM-based: identity policies for your principals, repository policies (resource-based, for other accounts and services) and a registry policy (replication and pull through cache permissions).
- The Docker CLI can't sign AWS requests, so you trade IAM credentials for a temporary registry password first. The same images feed ECS, EKS, Lambda container images (
notes/lambda-guide.md§8) and CodeBuild build environments.
Exam facts
- Authentication:
GetAuthorizationTokenreturns a base64 token holding userAWSand a password. It is valid for 12 hours, carries the permissions of the IAM principal that requested it, and works for every registry that principal can access; log in to each registry separately.aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account-id>.dkr.ecr.<region>.amazonaws.com. ⚠️ Changed: AWS CLI v2 removedaws ecr get-login; useget-login-password, which pipes the password todocker login(also available in CLI v1.17.10 and later). Older material may say to log in with$(aws ecr get-login --no-include-email).- The Amazon ECR Docker Credential Helper fetches and refreshes tokens automatically (it doesn't support MFA).
- ECR Public:
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws; always authenticate againstus-east-1.
- Permissions:
- Every client needs
ecr:GetAuthorizationTokenon"*"in an identity policy; a repository policy can't grant it. - Pull:
ecr:BatchGetImage(manifest, once per pull) andecr:GetDownloadUrlForLayer(each layer not already cached), usually withecr:BatchCheckLayerAvailability. - Push:
ecr:InitiateLayerUpload,ecr:UploadLayerPart,ecr:CompleteLayerUpload,ecr:BatchCheckLayerAvailabilityandecr:PutImageon the repository ARN (AWS's example addsecr:BatchGetImage). - AWS managed policies:
AmazonEC2ContainerRegistryPullOnly(pull only, added 2024-10-10),AmazonEC2ContainerRegistryReadOnly(list, describe, pull),AmazonEC2ContainerRegistryPowerUser(read and write, but no repository deletion or policy changes),AmazonEC2ContainerRegistryFullAccess. - For ECS, the task execution role pulls on Fargate and Managed Instances; on EC2 the container instance role can (§3).
- Every client needs
- Repository policies grant other accounts or services access to one repository (for example a production account pulling images built in a tools account, or
codebuild.amazonaws.comscoped withaws:SourceArnandaws:SourceAccount); the other account's principals still needecr:GetAuthorizationTokenin their own identity policies. - Tag mutability:
MUTABLEby default. WithIMMUTABLE, pushing a tag that already exists fails withImageTagAlreadyExistsException, so every build needs a unique tag (version or commit SHA). ⚠️ Changed: wildcard exclusion filters (IMMUTABLE_WITH_EXCLUSION,MUTABLE_WITH_EXCLUSION) let a few tags such aslatestbehave differently from the rest. Older material may say immutability always applies to every tag in the repository. Keep pull through cache repositories mutable so cached tags can refresh. - Encryption at rest:
AES256(SSE-S3) by default orKMSwith the AWS managed or your own key; chosen at creation and can't be changed afterwards. - Image scanning:
- Basic: ⚠️ Changed: AWS native scanning technology with the CVE database checks operating-system packages, on push or manually (one scan per image per 24 hours; 100,000 basic scans per 24 hours per Region); read results with
DescribeImageScanFindings. Older material may say basic scanning uses the open-source Clair scanner. - Enhanced: Amazon Inspector scans OS and programming-language packages, on push and continuously as new CVEs appear, and sends findings to EventBridge. Configure scanning at the registry level (
put-registry-scanning-configurationwith repository filters). Switching between basic and enhanced hides the earlier type's scans until you switch back.
- Basic: ⚠️ Changed: AWS native scanning technology with the CVE database checks operating-system packages, on push or manually (one scan per image per 24 hours; 100,000 basic scans per 24 hours per Region); read results with
- Lifecycle policies (JSON rules per repository):
- Each rule has a unique
rulePriority(lower number = applied first), aselectionand anaction. An image is handled by at most one rule; an image matched by a higher-priority rule can't be acted on by a lower one. tagStatus:tagged(requirestagPatternList, up to four*per pattern, ortagPrefixList, never both),untagged, orany(must have the highestrulePrioritynumber).countType:imageCountMoreThan(keep the newest N by push time),sinceImagePushed,sinceImagePulled(archives images not pulled for N days),sinceImageTransitioned(expires images archived for N days);countUnitdaysfor the time-based types.action.type:expire, ortransitionwithtargetStorageClassarchive.- ⚠️ Changed: lifecycle rules can move images to the low-cost archive storage class (archived images can't be pulled or scanned;
update-image-storage-class --target-storage-class STANDARDrestores one, typically within 20 minutes, and re-pushing it restores it at once) and can use last-pull and time-since-archive criteria. Older material may say lifecycle policies only expire images by count or by age since push. - Actions happen within 24 hours of an image meeting the criteria and are recorded in CloudTrail; run a lifecycle policy preview first. Limits: 50 rules and 30,720 characters per policy. An image referenced by a manifest list can't be expired before the list.
- Each rule has a unique
- Replication (registry setting): cross-Region and cross-account; only images pushed or restored after you configure it are copied; repository names stay the same; the destination registry policy must allow
ecr:ReplicateImage(andecr:CreateRepositoryto create missing repositories). Repository policies, lifecycle policies and settings aren't replicated (use repository creation templates). Up to 25 rules and 25 destinations; most images arrive within 30 minutes; replication doesn't chain and never deletes. - Pull through cache: a rule maps a repository prefix in your registry to an upstream: ECR Public, the Kubernetes registry and Quay (no credentials); Docker Hub, Azure Container Registry, GitHub Container Registry, GitLab (SaaS) and Chainguard (credentials in a Secrets Manager secret whose name starts with
ecr-pullthroughcache/, in the same account and Region); another ECR registry (IAM role). The first pull of<registry>/<prefix>/<image>:<tag>creates the repository and caches the image; ECR checks upstream for a newer version at most once per 24 hours and serves the cached copy if upstream is down. Up to 50 rules per registry; Lambda can't pull through these rules. Good answer to Docker Hub rate limits. - Repository creation templates: settings (tag mutability, encryption, repository policy, lifecycle policy, tags) that ECR applies to repositories it creates for you, matched by prefix (
ROOT= everything) andappliedForPULL_THROUGH_CACHE,CREATE_ON_PUSHorREPLICATION. ⚠️ Changed: a push to a repository that doesn't exist creates it when aCREATE_ON_PUSHtemplate matches (without a matching template ECR still creates nothing). Older material may say a repository must always be created before the first push. - Private subnets: pulls need the interface endpoints
com.amazonaws.<region>.ecr.dkr(Docker registry API) andcom.amazonaws.<region>.ecr.api(ECR API), plus an S3 gateway endpoint, because layers download from S3 (prod-<region>-starport-layer-bucket);awslogsalso needscom.amazonaws.<region>.logs. Fargate platform 1.4.0+ needs both ECR endpoints and the S3 endpoint. - Quotas: 100,000 images per repository and 100,000 repositories per Region (both adjustable); 1,000 tags per image; layers up to 52,000 MiB; API layer parts of 5–10 MiB;
GetAuthorizationToken500 requests per second,PutImage10 per second (adjustable). CannotPullContainerErrorpatterns:pull image manifest has been retried 5 time(s): failed to resolve ref→ the tag or digest doesn't exist (often a moved or deletedlatest).The task can't pull the image. Check that the role has the permissions…orpull access denied→ missing pull permissions on the execution role (Fargate) or instance role (EC2), or a wrong repository or account.request canceled while waiting for connection,context canceled,i/o timeout→ no network path: public subnet without a public IP, or private subnet without NAT or the ECR and S3 endpoints.toomanyrequests→ Docker Hub pull limits; move the image to ECR or use pull through cache.
Developer angle
Create a repository, log in, build, tag, push and read back the digest:
REGISTRY=111122223333.dkr.ecr.eu-west-1.amazonaws.com
aws ecr create-repository --repository-name orders-api \
--image-tag-mutability IMMUTABLE --image-scanning-configuration scanOnPush=true
aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin "$REGISTRY"
docker build -t orders-api:1.8.2 .
docker tag orders-api:1.8.2 "$REGISTRY/orders-api:1.8.2"
docker push "$REGISTRY/orders-api:1.8.2"
aws ecr describe-images --repository-name orders-api --image-ids imageTag=1.8.2 \
--query 'imageDetails[0].imageDigest'
Lifecycle policy lifecycle.json, previewed before it is applied:
{
"rules": [
{ "rulePriority": 1, "description": "untagged: expire 7 days after push", "action": { "type": "expire" },
"selection": { "tagStatus": "untagged", "countType": "sinceImagePushed", "countUnit": "days", "countNumber": 7 } },
{ "rulePriority": 2, "description": "release-*: keep the newest 30", "action": { "type": "expire" },
"selection": { "tagStatus": "tagged", "tagPatternList": ["release-*"], "countType": "imageCountMoreThan", "countNumber": 30 } },
{ "rulePriority": 3, "description": "feature-*: archive after 90 days without a pull", "action": { "type": "transition", "targetStorageClass": "archive" },
"selection": { "tagStatus": "tagged", "tagPatternList": ["feature-*"], "countType": "sinceImagePulled", "countUnit": "days", "countNumber": 90 } }
]
}
aws ecr start-lifecycle-policy-preview --repository-name orders-api --lifecycle-policy-text file://lifecycle.json
aws ecr get-lifecycle-policy-preview --repository-name orders-api
aws ecr put-lifecycle-policy --repository-name orders-api --lifecycle-policy-text file://lifecycle.json
Let the production account pull from the tools account's repository (aws ecr set-repository-policy --repository-name orders-api --policy-text file://repo-policy.json):
{
"Version": "2012-10-17",
"Statement": [{ "Sid": "AllowProdAccountPull", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::444455556666:root" },
"Action": ["ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer", "ecr:BatchCheckLayerAvailability"], "Resource": "*" }]
}
Traps
- Trap: "Grant
ecr:GetAuthorizationTokento the other account in the repository policy." → Reality: that action must be in each principal's identity policy; the repository policy grants the image actions. - Trap: "Log in once a day in CI with
aws ecr get-login." → Reality: tokens last 12 hours and AWS CLI v2 has noget-login; pipeget-login-passwordintodocker loginat the start of each job (or use the credential helper). - Trap: "Turn on immutable tags and keep pushing
latest." → Reality: the second push fails withImageTagAlreadyExistsException; tag builds uniquely or excludelatest. - Trap: "A new lifecycle policy deletes images immediately." → Reality: actions run within 24 hours; preview the policy first.
- Trap: "Turning on replication copies the existing images and their repository's lifecycle policy." → Reality: only images pushed or restored afterwards replicate, and policies don't travel; use a repository creation template for
REPLICATION. - Trap: "A private subnet only needs the
ecr.apiendpoint." → Reality: pulls also needecr.dkrand the S3 gateway endpoint for layers. - Trap: "Switch the repository to KMS encryption later." → Reality: encryption is fixed at creation; create a new repository and push again.
Practice questions
-
A CI pipeline pushes every build of an image as
latestto an Amazon ECR repository. After the security team turned on tag immutability, pushes fail withImageTagAlreadyExistsException. Tags must stay immutable, and each deployment must be traceable to the exact build. What should the developer change?- A. Set the repository's tag mutability back to
MUTABLEso thatlatestcan be overwritten. - B. Tag each build with a unique value such as the commit SHA and reference that tag in the task definition.
- C. Call
batch-delete-imagefor thelatesttag before every push. - D. Add
ecr:PutImagefor the pipeline's role to the repository policy.
- A. Set the repository's tag mutability back to
-
Amazon ECS tasks on AWS Fargate run in private subnets that have no NAT gateway and must not get any path to the internet. The tasks fail to start with
CannotPullContainerErrorand acontext canceledmessage when pulling from Amazon ECR in the same Region. The task execution role already hasAmazonECSTaskExecutionRolePolicy. Which TWO actions will fix the image pulls? (Select TWO.)- A. Set
assignPublicIptoENABLEDin the service'sawsvpcConfiguration. - B. Grant the task role
ecr:BatchGetImageandecr:GetDownloadUrlForLayer. - C. Create the
com.amazonaws.<region>.ecr.dkrandcom.amazonaws.<region>.ecr.apiinterface endpoints with private DNS enabled. - D. Create an Amazon S3 gateway endpoint and associate it with the private subnets' route tables.
- E. Create a pull through cache rule that points at the repository's own registry.
- A. Set
Answers
- B. Unique, immutable tags keep every image traceable and never collide. A breaks the immutability requirement; C removes the rollback image and keeps an ambiguous tag; D addresses permissions, but the push is rejected because the tag exists.
- C, D. Fargate needs both ECR interface endpoints for the registry and API calls and an S3 gateway endpoint to download the layers. A gives no route in a private subnet and violates the no-internet rule; B is the wrong role and not a network fix; E doesn't create a network path (and pull through cache needs internet access for the first pull).
10. Amazon EKS basics
Concept
- Amazon EKS runs the Kubernetes control plane for you, across multiple Availability Zones; you deploy with standard Kubernetes tooling (
kubectl, manifests, Helm). Where ECS uses AWS's own API, EKS exposes upstream Kubernetes, so manifests and tools carry over from other environments. - Pick EKS when the organisation has standardised on Kubernetes elsewhere (its own data centre or a different cloud provider), wants workloads to stay portable, or depends on Kubernetes ecosystem tools; pick ECS for AWS-native simplicity with fewer moving parts. Clusters are Regional, so a multi-Region design needs a cluster in each Region.
- Worker capacity comes from EKS Auto Mode, managed node groups, self-managed nodes, AWS Fargate or EKS Hybrid Nodes (on-premises machines). Pods get AWS permissions through EKS Pod Identity or IAM roles for service accounts (IRSA); people and pipelines get Kubernetes access through EKS access entries.
Exam facts
- Compute options:
- EKS Auto Mode (launched 2024-12-01): AWS selects, provisions, patches, scales and upgrades the EC2 nodes in your account and runs the core storage, networking and load-balancing capabilities; you configure nodes through
NodePoolandNodeClassresources; no SSH and no custom AMIs; nodes are replaced after at most 21 days; you pay an Auto Mode fee per instance on top of EC2. - Managed node groups: you describe the group (instance types, On-Demand or Spot, size limits) and EKS runs the Auto Scaling group and EKS-optimized AMIs behind it; you trigger AMI and Kubernetes version updates (one click for EKS AMIs).
- Self-managed nodes: you build and operate the Auto Scaling groups, pick the AMI (EKS-optimized or custom) and join the nodes to the cluster; most control, most maintenance.
- AWS Fargate: a Fargate profile selects pods by namespace and labels; each pod runs in its own VM-isolated environment. Pods that match no profile stay
Pending. Fargate pods run only in private subnets; no DaemonSets (run agents as sidecars), privileged containers,HostPort/HostNetwork, GPUs or EBS volumes (EFS works with static provisioning); load balancers must useiptargets; IMDS isn't available (use IRSA); Fargate Spot isn't supported on EKS. - ⚠️ Changed: EKS compute options include EKS Auto Mode (launched 2024-12-01) and EKS Hybrid Nodes in addition to managed node groups, self-managed nodes and Fargate. Older material may say EKS offers only managed node groups, self-managed nodes and Fargate.
- EKS Auto Mode (launched 2024-12-01): AWS selects, provisions, patches, scales and upgrades the EC2 nodes in your account and runs the core storage, networking and load-balancing capabilities; you configure nodes through
- AWS permissions for pods:
- EKS Pod Identity (AWS recommends it whenever possible): the role trusts
pods.eks.amazonaws.comforsts:AssumeRoleandsts:TagSession; you create an association between the role and a namespace/service account through the EKS API (no service-account annotation, no OIDC provider). The Pod Identity Agent DaemonSet (built into Auto Mode) hands out credentials; SDKs pick them up through the container credential provider (AWS_CONTAINER_CREDENTIALS_FULL_URI,AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE). Session tags (cluster, namespace, service account) allow one role with ABAC across workloads, and the same role works in new clusters without trust-policy edits. Up to 5,000 associations per cluster; the creator needsiam:PassRole. Linux EC2 nodes only: not on Fargate and not on Windows nodes. - IRSA: an IAM OIDC provider per cluster (IAM allows 100 per account by default), a role trusting that
Federatedprovider forsts:AssumeRoleWithWebIdentitywith:sub=system:serviceaccount:<namespace>:<service-account>and:aud=sts.amazonaws.com, and the annotationeks.amazonaws.com/role-arnon the service account. SDKs use the web identity provider (AWS_WEB_IDENTITY_TOKEN_FILE,AWS_ROLE_ARN). Works on Fargate and outside EKS (EKS Anywhere, self-managed Kubernetes on EC2, Red Hat OpenShift Service on AWS). - The node IAM role is for the kubelet and node agents. Pods can reach it through IMDS unless access is restricted, so never grant application permissions to it.
- ⚠️ Changed: AWS recommends EKS Pod Identity for granting pods AWS permissions, with IRSA kept for Fargate pods and non-EKS Kubernetes. Older material may say IRSA (or the node role) is the way to give pods AWS permissions.
- EKS Pod Identity (AWS recommends it whenever possible): the role trusts
- Kubernetes API access for people and pipelines:
aws eks update-kubeconfig --region <region> --name <cluster>writes the kubeconfig (needseks:DescribeCluster);kubectlthen authenticates throughaws eks get-tokenwith whatever identity the CLI resolves (aws sts get-caller-identityshows it).- Authorization comes from EKS access entries, which map an IAM principal to AWS-managed access policies or Kubernetes groups through the EKS API, CloudFormation or Terraform. ⚠️ Changed: AWS calls access entries the best way to grant IAM principals access to the Kubernetes API and treats the
aws-authConfigMap as legacy. Older material may say you grant access by editing theaws-authConfigMap.
- Storage (a
StorageClassandPersistentVolumeClaimbacked by a CSI driver):- EBS CSI driver: block storage; an EBS volume lives in one AZ and attaches to one node at a time; not for Fargate pods.
- EFS CSI driver: shared file storage across AZs and pods, the option that works with Fargate; it also mounts S3 Files file systems.
- Also FSx for Lustre, FSx for NetApp ONTAP, FSx for OpenZFS, Amazon File Cache, the Mountpoint for Amazon S3 CSI driver and EC2 instance store.
- ⚠️ Changed: EKS lists CSI-based storage for EBS, EFS, S3 Files (through the EFS CSI driver), FSx for Lustre, NetApp ONTAP and OpenZFS, File Cache, Mountpoint for Amazon S3 and instance store. Older material may say EKS storage is limited to EBS, EFS, FSx for Lustre and FSx for NetApp ONTAP.
- Load balancing: the AWS Load Balancer Controller provisions Application Load Balancers for Kubernetes Ingress and Network Load Balancers for
LoadBalancerservices. - Observability: CloudWatch Container Insights collects cluster, node and pod metrics and logs (the ECS side is in §6).
Developer angle
Connect kubectl and confirm which identity it uses:
aws eks update-kubeconfig --region eu-west-1 --name orders-prod
aws sts get-caller-identity
kubectl get pods -n orders
EKS Pod Identity: role trust policy and the association (no change inside the cluster):
{ "Version": "2012-10-17",
"Statement": [{ "Effect": "Allow", "Principal": { "Service": "pods.eks.amazonaws.com" }, "Action": ["sts:AssumeRole", "sts:TagSession"] }] }
aws eks create-pod-identity-association --cluster-name orders-prod \
--namespace orders --service-account orders-api \
--role-arn arn:aws:iam::111122223333:role/orders-api-pod-role
IRSA instead (needed on Fargate): annotate the service account, then reference it from the workload:
apiVersion: v1
kind: ServiceAccount
metadata:
name: orders-api
namespace: orders
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/orders-api-irsa
---
apiVersion: v1
kind: Pod
metadata:
name: orders-api
namespace: orders
spec:
serviceAccountName: orders-api
containers:
- name: api
image: 111122223333.dkr.ecr.eu-west-1.amazonaws.com/orders-api:1.8.2
Traps
- Trap: "Give pods S3 access by adding it to the node IAM role." → Reality: every pod on the node would inherit it; use Pod Identity (or IRSA) per service account.
- Trap: "Use EKS Pod Identity for pods on Fargate." → Reality: Pod Identity runs only on Linux EC2 nodes; Fargate pods use IRSA.
- Trap: "Run the log agent as a DaemonSet on Fargate." → Reality: DaemonSets don't run on Fargate; add a sidecar.
- Trap: "Mount an EBS volume in a Fargate pod." → Reality: not supported; use EFS.
- Trap: "Turn on Fargate Spot for an EKS Fargate profile." → Reality: EKS doesn't support Fargate Spot.
- Trap: "
aws eks update-kubeconfiggrants the user cluster permissions." → Reality: it only writes the connection settings; authorization comes from an access entry (or the legacyaws-authmap). - Trap: "IRSA roles can be reused in a new cluster without changes." → Reality: each cluster has its own OIDC provider, so the trust policy needs a new entry; Pod Identity avoids that.
- Trap: "Fargate pods can run in public subnets for direct internet access." → Reality: EKS Fargate pods run only in private subnets.
Practice questions
-
A team runs Amazon EKS managed node groups (Linux) in three clusters. The pods of one application must read a single DynamoDB table with least privilege. The platform team wants to reuse the same IAM role in all three clusters without editing its trust policy per cluster and without creating IAM OIDC providers. What should the developer do?
- A. Add the DynamoDB permissions to the node IAM role used by the managed node groups.
- B. Annotate the application's service account with
eks.amazonaws.com/role-arnand trust each cluster's OIDC provider. - C. Store an IAM user's access keys in a Kubernetes Secret mounted into the application's pods.
- D. Create an EKS Pod Identity association for the application's service account in each cluster.
-
A team plans to move several workloads from Amazon EKS managed node groups to Fargate profiles. Which TWO statements about EKS pods on Fargate are correct? (Select TWO.)
- A. Workloads that run as DaemonSets must be redesigned, for example as sidecar containers.
- B. Pods that need AWS permissions must use IAM roles for service accounts, because EKS Pod Identity doesn't support Fargate.
- C. Pods can mount Amazon EBS volumes through the EBS CSI driver for block storage.
- D. Fargate Spot can be enabled on a Fargate profile to lower the cost of interruptible pods.
- E. Pods can run in public subnets and receive public IP addresses for outbound access.
Answers
- D. Pod Identity associations need no OIDC provider, and a role trusting
pods.eks.amazonaws.comworks in any cluster unchanged while scoping permissions to one service account. A gives every pod on those nodes the permissions; B is IRSA, which requires an OIDC provider and a trust-policy entry per cluster; C uses long-term credentials. - A, B. DaemonSets don't run on Fargate, and Pod Identity is limited to Linux EC2 nodes, so Fargate pods use IRSA. C is wrong (EBS can't be mounted on Fargate pods); D is wrong (EKS doesn't support Fargate Spot); E is wrong (Fargate pods run only in private subnets).
Quick decision rules
Capacity and task definitions (§1–§2)
- If you see "run containers with the LEAST operational overhead" and no special hardware, think AWS Fargate.
- If you see GPUs or specific instance types but "don't want to patch or scale instances", think ECS Managed Instances.
- If you see custom AMIs, host-level daemons or Capacity Reservations, think self-managed EC2 capacity.
- If you see "mostly Spot to save money, but keep a guaranteed minimum", think a capacity provider strategy with a
FARGATEbase plusFARGATE_SPOTweight. - If you see AWS Copilot in a new design, think ECS Express Mode or the AWS CDK (Copilot support ended 2026-06-12).
- If you see
OutOfMemoryError: container killed due to memory usage, think raise the container's hardmemorylimit, notmemoryReservation. - If you see files shared by containers of one task, think bind mount; durable files shared by many tasks → EFS; high-IOPS block storage per task → EBS.
- If you see a helper container whose exit stops the whole task, think
essential: false(plusdependsOnfor start order).
- If you see
AccessDeniedfrom application code in a task, think task role (taskRoleArn). - If you see image pull,
awslogsorsecretsfailures, think task execution role (executionRoleArn). - If you see containers on EC2 using the instance role's permissions, think block IMDS for tasks and give each task definition its own task role.
- If you see "shell into a running Fargate container without SSH", think ECS Exec (task role
ssmmessages, Session Manager plugin, new tasks). - If you see a rotated secret that running tasks don't pick up, think
update-service --force-new-deployment(or read the secret at runtime). - If you see
unable to pull secrets or registry authin private subnets, think Secrets Manager or SSM VPC endpoints plus execution-role permissions.
Services, scaling and deployments (§5–§7)
- If you see several tasks per EC2 instance behind an ALB, think
bridgemode with dynamic host ports and the ephemeral range opened from the ALB security group. - If you see Fargate or
awsvpctasks behind a load balancer, think target typeip. - If you see healthy-but-slow-starting tasks killed by health checks, think
healthCheckGracePeriodSeconds. - If you see "run a container on a schedule or when an event arrives", think EventBridge Scheduler or an EventBridge rule with
ecs:RunTaskandiam:PassRole. - If you see a queue worker that must keep its backlog under control, think target tracking on backlog per task.
- If you see EC2 tasks waiting because instances are full, think an Auto Scaling group capacity provider with managed scaling (
targetCapacitybelow 100 for headroom). - If you see "deploy without any extra capacity", think
maximumPercent100 with aminimumHealthyPercentbelow 100. - If you see "roll back automatically when new tasks can't start or stay healthy", think the deployment circuit breaker with
rollback; for metric regressions, CloudWatch alarms in the deployment configuration. - If you see blue/green with test traffic and a bake period but no CodeDeploy, think
strategyBLUE_GREENwith an alternate target group and listener rules. - If you see gradual percentage-based traffic shifting on ECS without CodeDeploy, think the
LINEARorCANARYstrategy on theECScontroller.
Placement, ECR and EKS (§8–§10)
- If you see "use the fewest instances", think
binpack; "survive an AZ failure", thinkspreadonattribute:ecs.availability-zonefirst. - If you see "one task per instance", think
distinctInstance; "only on a given instance type", thinkmemberOf. - If you see
ImageTagAlreadyExistsException, think immutable tags: push a unique tag per build. - If you see "keep only the newest N images" or "clean up untagged images", think an ECR lifecycle policy (preview it first).
- If you see Fargate in private subnets failing to pull from ECR, think
ecr.dkrandecr.apiinterface endpoints plus an S3 gateway endpoint. - If you see Docker Hub rate limits, think ECR pull through cache.
- If you see EKS pods on EC2 nodes needing AWS permissions, think EKS Pod Identity (IRSA for Fargate pods).
Changed vs older material
Learn the Current column; older practice material may still show the Older material may say value. Rows you confirm during /ingest are added to notes/current-facts.md.
| Topic | Older material may say | Current (verified 2026-10-12) | Source |
|---|---|---|---|
| §1 · ECS capacity types | ECS has two launch types, EC2 and Fargate | Three capacity types: ECS Managed Instances (since 2025-09-30), Fargate and self-managed EC2, plus external instances | AWS docs |
| §1 · AWS Copilot CLI | Supported, recommended CLI for building and deploying ECS apps | End of support 2026-06-12 (announced 2026-03-06): no updates, patches or support; deployed apps keep running; use ECS Express Mode (2025-11-21) or the AWS CDK | AWS docs |
| §1 · AWS Copilot in the DVA-C02 scope | Listed under Containers (exam guide version 1.x) | Exam guide version 2.1 (2024-12-12) removed it; Containers = Amazon ECR, Amazon ECS, Amazon EKS | AWS docs |
| §2 · Fargate maximum task size | 16 vCPU and 120 GB | 32 vCPU with 60, 120 or 244 GB (Linux, platform version 1.4.0) | AWS docs |
| §2 · Default log delivery mode | Blocking | non-blocking since 2025-06-25 (max-buffer-size default 10m); set mode: blocking or the defaultLogDriverMode account setting to keep blocking |
AWS docs |
| §2 · EBS volumes for ECS tasks | Tasks can't use EBS; EFS is the only persistent option on Fargate | One new EBS volume per task, configured at launch (Fargate 1.4.0+, EC2 on Nitro, Managed Instances); deleted when a service task ends | AWS docs |
| §2 · S3 Files volumes on the EC2 launch type | Not supported (course) | ⚠️ unverified: the S3 User Guide and the ECS storage table say Fargate and Managed Instances only; the ECS "Configuring S3 Files" page lists EC2 as fully supported | AWS docs |
| §3 · Role that fetches task-definition secrets, pulls images and sends logs | The EC2 instance profile | Task execution role: secrets on Fargate and EC2; ECR pulls and awslogs on Fargate, Managed Instances and external instances (the instance role can still pull and log for EC2 tasks) |
AWS docs |
| §5 · Load balancers for ECS services | ALB, NLB and Classic Load Balancer | ALB (recommended), NLB and GWLB; Classic is legacy (EC2 only, 1 per service) | AWS docs |
| §5 · EventBridge permission to publish to an SNS target | The topic's access policy must allow EventBridge | The topic's access policy (principal events.amazonaws.com) or, since 2025-03-11, an IAM execution role on the target (RoleArn allowing sns:Publish); with no role set, EventBridge uses the topic policy |
AWS docs |
| §6 · Service auto scaling policy types | Target tracking, step and scheduled scaling | Also predictive scaling; 20-second high-resolution CPU and memory metrics for target tracking | AWS docs |
| §7 · ECS-native deployment strategies | Rolling updates only; blue/green requires CodeDeploy | ROLLING, BLUE_GREEN (2025-07-17), LINEAR and CANARY (2025-10-30) on the ECS controller, with lifecycle hooks and bake time; CodeDeploy still supported |
AWS docs |
| §7 · Circuit breaker failure threshold | Always 50 % of the desired count, at least 3 and at most 200 | Same default, but configurable: thresholdConfiguration (COUNT, BOUNDED_PERCENT, UNBOUNDED_PERCENT) and resetOnHealthyTask; works for rolling and ECS-native blue/green deployments |
AWS docs |
| §7 · Rolling update percentage ranges | Minimum 0–100 %, maximum 100–200 % (course slides) | ⚠️ unverified: the docs give the defaults (100 % and 200 %) and rounding rules but no fixed upper bound for maximumPercent |
AWS docs |
| §8 · Where placement strategies and constraints apply | Only the EC2 launch type | Strategies: EC2 only. Constraints: EC2 and ECS Managed Instances (attributes ecs.subnet-id, ecs.availability-zone, ecs.instance-type, ecs.cpu-architecture). Fargate: neither |
AWS docs |
| §9 · ECR login command | aws ecr get-login |
aws ecr get-login-password piped into docker login --password-stdin; get-login is gone from AWS CLI v2 |
AWS docs |
| §9 · ECR lifecycle policy actions | Expire only, by image count or age since push | Also transition to the archive storage class, sinceImagePulled and sinceImageTransitioned; a restore typically completes within 20 minutes |
AWS docs |
| §9 · Tag immutability | Applies to every tag or none | Wildcard exclusion filters (IMMUTABLE_WITH_EXCLUSION, MUTABLE_WITH_EXCLUSION) |
AWS docs |
| §9 · Basic image scanning engine | Open-source Clair | AWS native scanning technology with the CVE database; read results with DescribeImageScanFindings |
AWS docs |
| §9 · Pushing to a repository that doesn't exist | Always create the repository first | A matching CREATE_ON_PUSH repository creation template creates it on push |
AWS docs |
| §10 · EKS compute options | Managed node groups, self-managed nodes and Fargate | Also EKS Auto Mode (launched 2024-12-01) and EKS Hybrid Nodes | AWS docs |
| §10 · AWS permissions for pods | IRSA (or the node role) | EKS Pod Identity recommended; IRSA for Fargate pods and non-EKS Kubernetes | AWS docs |
| §10 · IAM principals' access to the Kubernetes API | Edit the aws-auth ConfigMap |
EKS access entries (recommended); aws-auth is legacy |
AWS docs |
| §10 · EKS storage drivers | EBS, EFS, FSx for Lustre and FSx for NetApp ONTAP | Also FSx for OpenZFS, File Cache, Mountpoint for Amazon S3, S3 Files (through the EFS CSI driver) and instance store | AWS docs |