SQS, SNS and Kinesis — DVA-C02 Mastery Guide

• Sources: course chapter 12 (SQS, SNS and Kinesis, notes/source/12.SQS_SNS_Kinesis.pdf), used as the syllabus only; all wording here is original. Lambda event source mappings, EventBridge, S3 event notifications, MSK and Step Functions belong to other modules and appear only as cross-references.
• Written 2026-10-12; values checked against AWS docs on 2026-10-12. The marker ⚠️ Changed: <current>. Older material may say <old>. flags a value or behaviour that differs from the course or from older practice tests: learn the current value. ⚠️ unverified marks a detail that could not be confirmed in AWS documentation: don't build an answer on it. Every change is listed in Changed vs older material.
• Tier 2 (deployment and messaging), bank target 40 questions. Domains: Development with AWS Services (primary), Troubleshooting and Optimization.
• How to use it: /explain <topic> teaches from these sections (concept → exam facts → developer angle → traps → 3 check questions); /quiz sqs-sns-kinesis asks bank questions whose source_note links point at the ## N. headings below; /drill sqs-sns-kinesis fires short recall prompts from notes/sqs-sns-kinesis-cheat-sheet.md. Suggested loop: read a section, answer its two practice questions before opening the answers, then /quiz and /drill.

How SQS, SNS and Kinesis appears on DVA-C02

The exam treats these services as the glue between application components. A question describes a producer, one or more consumers and a requirement, then asks which service, setting or API call meets it:

• Choosing the service: queue (pull; one logical consumer per queue), topic (push; many subscribers), stream (ordered, replayable, many independent readers) or Firehose stream (load into storage with no code). The deciding words are usually ordering, exactly once, fan out, replay, real time, no code and least operational overhead.
• Getting the settings right: visibility timeout vs processing time, long polling, delay seconds, dead-letter queues and maxReceiveCount, FIFO message group and deduplication IDs, filter policies, shard counts and partition keys, Firehose buffering.
• Troubleshooting: duplicates, messages that reappear, empty receives, a subscribed queue that never receives anything (queue policy or KMS key policy), ProvisionedThroughputExceededException, rising iterator age, failing Lambda transformations.
• Security in passing: queue and topic resource policies, cross-account access, encryption with SQS-owned or KMS keys and the key policies that AWS service producers need.

Typical question shapes: a scenario with one deciding constraint; a symptom (metric, error code, duplicate or missing message) to diagnose; "Which TWO actions…" with five options; the exact attribute, parameter or API action that does the job.

Domain map: Development with AWS Services → §1, §4–§11, §13, §14 · Troubleshooting and Optimization → §2, §3, §12 and the metric and failure parts of §1, §8 and §13. Policy and encryption questions (§6, §7, §8) also touch the Security domain; in this module's bank they count under development.

Covered elsewhere (one-line cross-references only): Lambda event source mapping tuning (batch size, partial batch responses, ParallelizationFactor, bisecting): see notes/lambda-guide.md · EventBridge rules and Scheduler: see notes/monitoring-guide.md · S3 event notification setup: see notes/s3-guide.md · Auto Scaling on queue backlog: see notes/ec2-mastery-guide.md §10 · MSK vs Kinesis, Step Functions, SES: see notes/other-services-guide.md · KMS key types and key policy mechanics: see notes/security-encryption-guide.md.

1. SQS standard queues and the message lifecycle

Concept

• A direct call between two services ties them together: if the receiver is slow, down or swamped by a burst, the sender fails too. Putting Amazon SQS in between turns the call into a message that waits until a consumer is ready, so each side scales and fails on its own.
• SQS is pull-based: producers call SendMessage; consumers call ReceiveMessage, do the work, then call DeleteMessage. Receiving removes nothing: a message stays in the queue until it is deleted or its retention period runs out.
• A message passes through three states: stored (sent, not yet received; no limit on how many), in flight (received, not yet deleted; limited) and deleted.
• A standard queue trades guarantees for scale: nearly unlimited API calls per second per action, no cap on the backlog, at-least-once delivery (an occasional duplicate) and best-effort ordering. Consumers must therefore be idempotent.

Exam facts

• Message size: 1 byte to 1 MiB (1,048,576 bytes). ⚠️ Changed: 1 MiB since 2025-08-04 (Lambda's SQS event source mapping accepts it too). Older material may say 256 KB. The queue attribute MaximumMessageSize can only lower the cap (1,024 bytes to 1 MiB, default 1 MiB); anything bigger goes through S3 (§4).
• Body and attributes: the body is XML, JSON or plain text from a restricted Unicode set (encode binary data, for example as base64, or carry it in a Binary attribute); up to 10 message attributes (DataType String, Number or Binary) count toward the size limit. System attributes (ApproximateReceiveCount, SentTimestamp, AWSTraceHeader, MessageGroupId…) are requested with MessageSystemAttributeNames; the older AttributeNames parameter is deprecated.
• Retention: MessageRetentionPeriod default 345,600 s (4 days), minimum 60 s, maximum 1,209,600 s (14 days). Most attribute changes propagate within 60 s; a retention change can take up to 15 minutes, and lowering it below the age of queued messages deletes them.
• Receive: MaxNumberOfMessages 1–10 (default 1). Every receive returns a new receipt handle; DeleteMessage and ChangeMessageVisibility need the handle of the most recent receive (an old handle can return success without deleting anything). The MessageId is never used to delete.
• Batch actions: SendMessageBatch, DeleteMessageBatch and ChangeMessageVisibilityBatch take up to 10 entries with a combined payload of up to 1 MiB; each entry has an Id (up to 80 characters), and the response lists Successful and Failed entries, so one call can partly fail.
• Billing: each 64 KB chunk of payload counts as one request (a 1 MiB message = 16 requests); a batch of small messages is a single request; the first 1 million requests per month are free; FIFO actions are billed at FIFO rates.
• Duplicates after a delete: on rare occasions a standard queue returns a message that was already deleted (a copy sat on a server that was unavailable during the delete). Idempotency is the only defence.
• In-flight limit: about 120,000 per standard queue and 120,000 per FIFO queue. At the limit, short polling on a standard queue returns OverLimit; long polling returns no messages and no error. ⚠️ Changed: the FIFO in-flight limit is 120,000 (since 2024-11-21). Older material may say 20,000.
• PurgeQueue deletes every message, in-flight ones included; deletion takes up to 60 s, a second purge within 60 s fails with PurgeQueueInProgress, and purged messages can't be recovered.
• Queue names: up to 80 characters (letters, digits, -, _), case-sensitive. CreateQueue with an existing name and identical attributes returns the existing URL; with different attributes it fails with QueueNameExists. After DeleteQueue, wait 60 s before reusing the name (QueueDeletedRecently).
• Fair queues (standard queues): put a tenant ID in MessageGroupId and SQS stops one noisy tenant's backlog from raising the dwell time of the others; no consumer change, no ordering, no throughput penalty. ⚠️ Changed: standard queues accept MessageGroupId for fair queues (since 2025-07-21). Older material may say MessageGroupId exists only on FIFO queues.
• Metrics (AWS/SQS, dimension QueueName, 1-minute): ApproximateNumberOfMessagesVisible (backlog), ApproximateNumberOfMessagesNotVisible (in flight), ApproximateNumberOfMessagesDelayed, ApproximateAgeOfOldestMessage, NumberOfEmptyReceives, NumberOfMessagesSent, NumberOfMessagesDeleted.
• Scaling consumers: add pollers horizontally. ⚠️ Changed: the CloudWatch backlog metric is ApproximateNumberOfMessagesVisible; ApproximateNumberOfMessages is the GetQueueAttributes attribute. Older material may say to scale on a CloudWatch metric named ApproximateNumberOfMessages. EC2 workers scale on backlog per instance: see notes/ec2-mastery-guide.md §10; Lambda consumers: see notes/lambda-guide.md.

Developer angle

aws sqs create-queue --queue-name orders \
  --attributes MessageRetentionPeriod=1209600,VisibilityTimeout=60,ReceiveMessageWaitTimeSeconds=20
QURL=$(aws sqs get-queue-url --queue-name orders --query QueueUrl --output text)
aws sqs send-message --queue-url "$QURL" --message-body '{"orderId":"o-1001","total":42.5}' \
  --message-attributes '{"channel":{"DataType":"String","StringValue":"web"}}'
# backlog and in-flight counts (queue attribute names, not CloudWatch metric names)
aws sqs get-queue-attributes --queue-url "$QURL" \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

A consumer loop that long-polls, processes and deletes in batches:

import boto3
sqs = boto3.client("sqs")
queue_url = sqs.get_queue_url(QueueName="orders")["QueueUrl"]

while True:
    resp = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=10, WaitTimeSeconds=20)
    done = []
    for msg in resp.get("Messages", []):
        handle_order(msg["Body"])  # your idempotent logic: a message can arrive twice
        done.append({"Id": msg["MessageId"], "ReceiptHandle": msg["ReceiptHandle"]})
    if done:
        result = sqs.delete_message_batch(QueueUrl=queue_url, Entries=done)
        for failure in result.get("Failed", []):
            print("delete failed:", failure["Id"], failure["Code"])

• IAM: producers need sqs:SendMessage (it also authorises SendMessageBatch); consumers need sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility, sqs:GetQueueAttributes; sqs:GetQueueUrl resolves a name. Scope Resource to arn:aws:sqs:<region>:<account>:<queue-name>.
• Errors to recognise: QueueDoesNotExist (wrong URL, Region or letter case), OverLimit, InvalidAttributeValue, ReceiptHandleIsInvalid, RequestThrottled, InvalidSecurity (request not over HTTPS with SigV4).

Traps

• Trap: "Delete the message by its MessageId." → Reality: deletion uses the receipt handle from the latest receive.
• Trap: "A received message has left the queue." → Reality: it's only hidden; without DeleteMessage it returns when the visibility timeout ends.
• Trap: "Standard queues keep the send order." → Reality: ordering is best effort; use FIFO (§5) when order matters.
• Trap: "A successful DeleteMessageBatch call deleted every message." → Reality: read the Failed list in the response.
• Trap: "Call PurgeQueue in a loop to empty a busy queue faster." → Reality: a purge takes up to 60 s and a repeat within 60 s fails with PurgeQueueInProgress.
• Trap: "Alarm on the CloudWatch metric ApproximateNumberOfMessages." → Reality: the metric is ApproximateNumberOfMessagesVisible; the shorter name is a queue attribute.
• Trap: "Fair queues keep each tenant's messages in order." → Reality: on a standard queue MessageGroupId only identifies the tenant; ordering needs FIFO.

Practice questions

1. A developer runs a payment-capture worker that long-polls an SQS standard queue and deletes each message right after processing it. Logs show that a few payments are captured twice each week. The team must keep the existing queue and producers unchanged. What should the developer do?

   • A. Increase the queue's MessageRetentionPeriod to 14 days.
   • B. Set MaxNumberOfMessages to 1 on every ReceiveMessage call and give each message its own worker thread.
   • C. Record each payment ID with a conditional write and skip IDs already recorded.
   • D. Call PurgeQueue after each batch of messages has been processed.

2. A developer's consumer receives messages from an SQS standard queue and deletes them later from a cleanup thread, using the MessageId values it saved. Processed messages keep coming back, and request costs are high because each ReceiveMessage call returns a single message. Which TWO changes fix these problems? (Select TWO.)

   • A. Send every message with a MessageDeduplicationId.
   • B. Delete each message with the ReceiptHandle returned by its most recent receive.
   • C. Raise the queue's MaximumMessageSize to its maximum value.
   • D. Set MaxNumberOfMessages to 10 on each ReceiveMessage call.
   • E. Turn on server-side encryption with an SSE-KMS key.

2. SQS visibility timeout and long polling

Concept

• When a consumer receives a message, SQS hides it from other consumers for the visibility timeout instead of removing it. Deleted in time, it's gone; if the timer runs out first (slow job, crash, lost connection), the message becomes visible again and the next receive picks it up. That is how SQS retries.
• The value is a trade-off: too short and the message reappears while the first worker is still busy, so it's processed twice; too long and, after a crash, nobody can retry until the timer expires.
• The polling style decides cost and latency. Short polling (the default) asks a sample of SQS servers and answers at once, even with nothing, and can miss messages held on other servers. Long polling asks all servers and holds the request open until a message arrives or the wait time ends, so there are far fewer empty, billable responses.

Exam facts

• VisibilityTimeout: default 30 s, range 0–43,200 s (12 hours). Three levels: the queue attribute, the VisibilityTimeout parameter of one ReceiveMessage call, or ChangeMessageVisibility for one received message.
• ChangeMessageVisibility sets a new timeout counted from the moment of the call; it doesn't add to the time left. The 12-hour ceiling counts from the first receive and extending doesn't reset it (a value beyond the remaining maximum is rejected). VisibilityTimeout=0 releases the message to other consumers immediately. The changed value applies to that receive only: if the message is received again, it gets the queue's timeout.
• Heartbeat pattern for jobs of unpredictable length: start with a moderate timeout and extend it periodically while the work continues. Work that can run longer than 12 hours belongs in smaller steps or a Step Functions workflow (see notes/other-services-guide.md).
• A receive that ends without a delete counts as a failed attempt toward maxReceiveCount (§3). The visibility timeout reduces, but on standard queues doesn't eliminate, duplicate deliveries.
• Errors: MessageNotInflight (the message isn't currently in flight), ReceiptHandleIsInvalid (stale or malformed handle). In-flight limits: §1.
• Lambda as consumer: set the queue's visibility timeout to at least 6 × the function timeout (plus MaximumBatchingWindowInSeconds); Lambda refuses an event source mapping whose function timeout exceeds the queue's visibility timeout. The rest of the mapping: see notes/lambda-guide.md.
• Long polling is on whenever the wait time is above 0: queue attribute ReceiveMessageWaitTimeSeconds (default 0) or the WaitTimeSeconds parameter of ReceiveMessage, which overrides the queue attribute when present. Maximum 20 s. A long poll returns as soon as at least one message is available (up to MaxNumberOfMessages); an empty response comes back only when the wait expires.
• Short polling on a queue with fewer than about 1,000 messages can return fewer messages than requested, or none: call again (or long-poll). NumberOfEmptyReceives exposes the wasted polls.
• The SDK's HTTP read timeout must be longer than WaitTimeSeconds, otherwise long polls fail with client-side timeouts.

Developer angle

# queue-wide defaults
aws sqs set-queue-attributes --queue-url "$QURL" \
  --attributes VisibilityTimeout=120,ReceiveMessageWaitTimeSeconds=20
# per call: long poll, batch of 10, a longer timeout for this batch only
aws sqs receive-message --queue-url "$QURL" --wait-time-seconds 20 \
  --max-number-of-messages 10 --visibility-timeout 300 \
  --message-system-attribute-names ApproximateReceiveCount
# give one message 10 more minutes, counted from now
aws sqs change-message-visibility --queue-url "$QURL" \
  --receipt-handle "$RECEIPT_HANDLE" --visibility-timeout 600

Heartbeat for long jobs:

import threading
import boto3

sqs = boto3.client("sqs")

def process_with_heartbeat(queue_url, msg, work, extend_to=120, every=60):
    """Keep the message hidden while work() runs, then delete it."""
    stop = threading.Event()
    def heartbeat():
        while not stop.wait(every):   # every 60 s: hidden for another 120 s, counted from now
            sqs.change_message_visibility(QueueUrl=queue_url, ReceiptHandle=msg["ReceiptHandle"],
                                          VisibilityTimeout=extend_to)
    threading.Thread(target=heartbeat, daemon=True).start()
    try:
        work(msg["Body"])
        sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=msg["ReceiptHandle"])
    finally:
        stop.set()

• IAM: the heartbeat needs sqs:ChangeMessageVisibility (which also covers the batch variant). boto3's default read timeout (60 s) already exceeds a 20 s long poll; other SDK clients may need their HTTP client timeout raised.
• Choosing values: visibility timeout ≥ the normal processing time of a whole batch plus a margin; 20 s long polls for background workers.

Traps

• Trap: "Raise the visibility timeout so new messages stay hidden for 5 minutes after they're sent." → Reality: visibility starts at receive; hiding new messages is a delay (§4).
• Trap: "ChangeMessageVisibility adds its value to the time left." → Reality: it restarts the clock with the new value.
• Trap: "Keep extending the timeout to process one message for a day." → Reality: 12 hours from the first receive is the ceiling.
• Trap: "Set the visibility timeout to 12 hours so nothing is processed twice." → Reality: after a crash the retry then waits hours, and standard queues can still duplicate.
• Trap: "Long polling costs more because requests stay open." → Reality: it reduces the number of billed empty receives.
• Trap: "An empty short-poll response proves the queue is empty." → Reality: short polling samples servers; long-poll or check the queue metrics.
• Trap: "WaitTimeSeconds=60." → Reality: the maximum is 20 s.

Practice questions

1. A video service sends transcoding jobs to an SQS standard queue. Jobs take between 2 and 40 minutes. With the queue's visibility timeout at 5 minutes, long jobs are often transcoded twice. If a worker crashes, its job must be retried within a few minutes. What should the developer do?

   • A. Set the queue's visibility timeout to 12 hours, the maximum that SQS allows for a message.
   • B. Set DelaySeconds to 900 on the queue.
   • C. Enable long polling on the queue with a 20-second wait time.
   • D. Extend the timeout with periodic ChangeMessageVisibility calls while a job runs.

2. A developer sees that the CloudWatch NumberOfEmptyReceives metric of an SQS queue is very high and drives most of the queue's cost. The queue usually holds messages, yet many ReceiveMessage calls return nothing. Which TWO changes would each reduce the empty responses? (Select TWO.)

   • A. Set the queue attribute ReceiveMessageWaitTimeSeconds to 20.
   • B. Set the queue's VisibilityTimeout to 0.
   • C. Pass WaitTimeSeconds=20 on each ReceiveMessage call.
   • D. Set the queue's DelaySeconds to 20.
   • E. Re-create the queue as a FIFO queue.

3. SQS dead letter queues and redrive

Concept

• Some messages can never succeed (malformed payload, a missing record, a bug for one edge case). Without a limit they cycle forever (receive, fail, reappear), wasting consumer capacity and hiding the real backlog.
• A dead-letter queue (DLQ) is an ordinary queue that a source queue names in its redrive policy. Once a message has been received more times than maxReceiveCount, SQS moves it to the DLQ, where it can be inspected without blocking healthy traffic.
• After the bug is fixed, DLQ redrive moves the messages back to the source queue (or another queue) without custom code.

Exam facts

• RedrivePolicy (source queue attribute, a JSON string): deadLetterTargetArn + maxReceiveCount (1–1,000). A value of 1 sends a message to the DLQ after a single transient error; with Lambda consumers AWS recommends at least 5.
• The DLQ must exist first (SQS doesn't create it), be the same type (FIFO source → FIFO DLQ, standard → standard) and live in the same account and Region.
• RedriveAllowPolicy on the DLQ controls which queues may use it: allowAll (default: any queue in the account and Region), denyAll, or byQueue with up to 10 sourceQueueArns.
• Retention trap (standard): expiry is based on the original enqueue time, which doesn't change when the message moves; a message that spent 3 days in a 4-day source queue lives only one more day in a 4-day DLQ. Give the DLQ a longer retention than its source (14 days is common). FIFO: the enqueue time resets when the message moves.
• Monitoring: alarm on the DLQ's ApproximateNumberOfMessagesVisible (≥ 1). NumberOfMessagesSent ignores messages that SQS moves automatically, so it stays at 0.
• ApproximateReceiveCount shows how often a message was received; DLQ messages can also return DeadLetterQueueSourceArn.
• Standard queues may move a message that has been received 3 or more times without being deleted to the back of the queue; ApproximateAgeOfOldestMessage then skips such poison messages.
• FIFO: a failing message blocks its message group until it's deleted or moved. A DLQ unblocks the group but breaks strict order, so skip it where an out-of-order operation would be worse than a stall.
• Redrive: console Start DLQ redrive or StartMessageMoveTask with SourceArn (the DLQ), optional DestinationArn (default: each message's original source queue; a custom destination must be of the same queue type) and optional MaxNumberOfMessagesPerSecond (up to 500; omitted = system-optimised rate). ListMessageMoveTasks shows the latest 10 tasks; CancelMessageMoveTask stops a RUNNING task (messages already moved stay moved).
• Redrive limits: one active task per queue, 36 hours per task, 100 active tasks per account; messages can't be filtered or edited on the way; redriven messages get a new message ID and enqueue time, so retention restarts.
• Only DLQs whose sources are SQS queues can be redriven this way; the DLQs of SNS subscriptions or Lambda asynchronous invocations hold messages that you consume yourself.
• ⚠️ Changed: redrive works through the API (StartMessageMoveTask, since 2023-06-07) and for FIFO queues, back to the original queue or to another queue of the same type. Older material may say redrive is console-only and limited to standard queues.
• Permissions to start a redrive: on the DLQ sqs:StartMessageMoveTask, sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes (plus sqs:ListDeadLetterSourceQueues, which answers "which queues use this DLQ?"); on the destination sqs:SendMessage; with SSE-KMS, kms:Decrypt on the DLQ's key and kms:GenerateDataKey + kms:Decrypt on the destination's key.

Developer angle

attrs.json for the source queue (the policy value is a JSON string):

{"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:111122223333:orders-dlq\",\"maxReceiveCount\":\"5\"}"}

The DLQ's RedriveAllowPolicy uses the same string format, for example {"redrivePermission":"byQueue","sourceQueueArns":["arn:aws:sqs:eu-west-1:111122223333:orders"]}.

aws sqs create-queue --queue-name orders-dlq --attributes MessageRetentionPeriod=1209600
aws sqs set-queue-attributes --queue-url "$QURL" --attributes file://attrs.json
# after the fix: move the messages back at a controlled rate
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:eu-west-1:111122223333:orders-dlq \
  --max-number-of-messages-per-second 50
aws sqs list-message-move-tasks --source-arn arn:aws:sqs:eu-west-1:111122223333:orders-dlq
# page someone as soon as anything lands in the DLQ
aws cloudwatch put-metric-alarm --alarm-name orders-dlq-not-empty \
  --namespace AWS/SQS --metric-name ApproximateNumberOfMessagesVisible \
  --dimensions Name=QueueName,Value=orders-dlq --statistic Maximum \
  --period 60 --evaluation-periods 1 --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-alerts

• Inspect before redriving: receive from the DLQ with MessageSystemAttributeNames ApproximateReceiveCount and DeadLetterQueueSourceArn, fix the consumer, test on a sample, then start the move task.

Traps

• Trap: "Setting maxReceiveCount makes SQS create a DLQ." → Reality: create the DLQ first and reference its ARN.
• Trap: "A FIFO queue can use a cheaper standard DLQ." → Reality: the types must match.
• Trap: "The DLQ's 4-day retention starts when the message arrives there." → Reality: standard queues keep the original enqueue time.
• Trap: "maxReceiveCount = 1 catches failures fastest." → Reality: one throttling error or timeout then sends a good message to the DLQ.
• Trap: "Alarm on the DLQ's NumberOfMessagesSent." → Reality: automatic moves aren't counted; use ApproximateNumberOfMessagesVisible.
• Trap: "The redrive can repair payloads on the way back." → Reality: no filtering or modification; fix the consumer, then redrive.
• Trap: "Redrive an SNS subscription's DLQ with StartMessageMoveTask." → Reality: only DLQs of SQS source queues are supported.

Practice questions

1. A developer configured a standard order queue and its standard DLQ with 4-day retention and a maxReceiveCount of 5. Messages that sat in the source queue for about 3 days before failing vanish from the DLQ the next day, before anyone can investigate them. What should the developer change?

   • A. Raise maxReceiveCount on the source queue to 50.
   • B. Set the DLQ's message retention period to 14 days.
   • C. Enable long polling on the DLQ.
   • D. Recreate the DLQ as a FIFO queue.

2. A developer has deployed a bug fix, and 50,000 messages in a DLQ must go back to their source queue. The downstream database can absorb only about 100 extra writes per second, and the team wants no custom code. Which TWO actions should the developer take? (Select TWO.)

   • A. Call PurgeQueue on the DLQ before starting the redrive.
   • B. Set the DLQ's DelaySeconds attribute to 900.
   • C. Call StartMessageMoveTask on the DLQ with MaxNumberOfMessagesPerSecond set to 100.
   • D. Write a Lambda function that polls the DLQ and re-sends each message to the source queue.
   • E. Give the operator role sqs:StartMessageMoveTask, sqs:ReceiveMessage, sqs:DeleteMessage and sqs:GetQueueAttributes on the DLQ and sqs:SendMessage on the source queue.

4. SQS delay queues and large messages

Concept

• Sometimes a message must not be processed right away: a dependent system needs time to catch up, or a short follow-up should happen a few minutes later. A delay queue hides every new message for a fixed time; a message timer sets the delay on a single message.
• Delay and visibility timeout both make a message invisible; the difference is when. A delay applies before the first receive, a visibility timeout after each receive.
• SQS caps a message at 1 MiB. For bigger payloads, store the payload in Amazon S3 and send a small message that points at it; the Extended Client Library automates both ends.

Exam facts

• DelaySeconds (queue attribute): 0–900 s (15 minutes), default 0. Any non-zero value makes the queue a delay queue.
• Message timer: the DelaySeconds parameter of SendMessage (or of an entry in SendMessageBatch), 0–900 s, overrides the queue's delay for that message. FIFO queues don't support per-message timers, only the queue-level delay.
• Changing the queue's delay on a standard queue affects only messages sent afterwards; on a FIFO queue it also applies to messages already waiting.
• Delayed messages are counted in ApproximateNumberOfMessagesDelayed, not in the visible backlog.
• Schedules beyond 15 minutes, or recurring ones: EventBridge Scheduler (see notes/monitoring-guide.md), not SQS.
• Extended Client Library: available for Java and Python; stores the payload in S3 (up to 2 GB), sends a reference message, resolves it on receive and can delete the S3 object when the message is deleted; it can route every message through S3 or only those above a size threshold; it works with synchronous clients only. ⚠️ Changed: there is an Extended Client Library for Python as well. Older material may say the library exists only for Java. SNS has its own extended client libraries (§7).
• Producers and consumers both need the library (or code that understands the pointer) plus S3 permissions on the payload bucket: s3:PutObject for the producer, s3:GetObject for the consumer, s3:DeleteObject for whoever cleans up.
• Add an S3 lifecycle rule that expires orphaned payloads (for example after 14 days, the longest possible queue retention).

Developer angle

# every new message waits 2 minutes
aws sqs set-queue-attributes --queue-url "$QURL" --attributes DelaySeconds=120
# one message waits 15 minutes (standard queues only); overrides the queue delay
aws sqs send-message --queue-url "$QURL" \
  --message-body '{"type":"reminder","userId":"u-77"}' --delay-seconds 900

Java producer with the Extended Client Library (AWS SDK for Java 2.x):

S3Client s3 = S3Client.create();
ExtendedClientConfiguration config = new ExtendedClientConfiguration()
        .withPayloadSupportEnabled(s3, "orders-large-payloads");
SqsClient sqsExtended = new AmazonSQSExtendedClient(SqsClient.builder().build(), config);

sqsExtended.sendMessage(SendMessageRequest.builder()
        .queueUrl(queueUrl)
        .messageBody(largeJsonDocument)   // stored in S3; SQS receives a pointer
        .build());

• Without the library, any SDK can do the same by hand: PutObject the payload, then send a small JSON body such as {"s3Bucket": "...", "s3Key": "..."} that the consumer resolves with GetObject.

Traps

• Trap: "Use the visibility timeout to keep a brand-new message hidden." → Reality: visibility only starts at receive; use DelaySeconds.
• Trap: "Delay each FIFO message individually." → Reality: FIFO queues support only the queue-level delay.
• Trap: "Schedule a message 2 hours ahead with DelaySeconds=7200." → Reality: the maximum is 900 s; use EventBridge Scheduler.
• Trap: "Lowering a standard queue's delay releases waiting messages early." → Reality: only messages sent after the change get the new value.
• Trap: "Set MaximumMessageSize to 2 GB." → Reality: 1 MiB is the maximum; S3 holds anything larger.
• Trap: "The Extended Client works from the AWS CLI or the console." → Reality: it's a client library (Java, Python) for synchronous SDK clients.

Practice questions

1. A sign-up service publishes several event types to one SQS standard queue. Only the welcome-reminder events must become available to consumers 10 minutes after they're sent; every other event must be processed immediately. What should the developer do?

   • A. Send the reminder events with the DelaySeconds parameter set to 600.
   • B. Set the queue's DelaySeconds attribute to 600.
   • C. Set the queue's VisibilityTimeout to 600.
   • D. Move the reminders to a FIFO queue and set DelaySeconds on each reminder message.

2. A developer must pass 5 MB JSON documents between two Java components through SQS with the least custom code. Which TWO actions should the developer take? (Select TWO.)

   • A. Set the queue's MaximumMessageSize attribute to 5,242,880 bytes.
   • B. Use the Amazon SQS Extended Client Library for Java with an S3 payload bucket in both the producer and the consumer.
   • C. Split each document into 1 MiB parts and send the parts with SendMessageBatch.
   • D. Enable content-based deduplication on the queue.
   • E. Grant the producer s3:PutObject and the consumer s3:GetObject and s3:DeleteObject on the payload bucket.

5. SQS FIFO queues

Concept

• A FIFO queue adds two guarantees: messages that share a message group are delivered in the order they were sent, and the queue doesn't introduce duplicates (exactly-once processing within the deduplication interval).
• The message group is the unit of both ordering and parallelism. One group behaves like a single ordered lane; many groups (one per account, order or device) are consumed in parallel while each lane stays ordered.
• The price is lower throughput per API action and stricter rules: a required group ID, no per-message delays, and a FIFO dead-letter queue.

Exam facts

• The name must end in .fifo (counted in the 80 characters), and FifoQueue=true can be set only at creation. A standard queue can't be converted: create a new FIFO queue and switch producers and consumers to it.
• MessageGroupId (up to 128 characters) is required on every send; without it the call fails. There's no limit on the number of groups.
• While a message of a group is in flight, SQS returns no later message of that group until it's deleted or its visibility timeout expires; other groups keep flowing. A single group therefore serialises all processing.
• Deduplication: every message needs a MessageDeduplicationId, sent explicitly or generated by ContentBasedDeduplication=true (SHA-256 of the body only, attributes excluded). An explicit ID overrides the generated one; with neither, the send fails. A duplicate sent within the 5-minute interval is accepted but not delivered, and NumberOfDeduplicatedSentMessages counts it.
• Retrying a send with the same deduplication ID within 5 minutes is safe. A retry that rebuilds the body (new timestamp) and relies on content-based deduplication creates a second message.
• Normal throughput: 300 transactions per second per API action (SendMessage, ReceiveMessage, DeleteMessage) per partition; with batches of 10, up to 3,000 messages per second.
• High throughput mode: set DeduplicationScope=messageGroup and FifoThroughputLimit=perMessageGroupId. Quotas depend on the Region: up to 70,000 TPS per API action in US East (N. Virginia), US West (Oregon) and Europe (Ireland) (700,000 messages/s with batching), 19,000, 9,000 or 4,500 TPS in some other listed Regions, and a default of 2,400 TPS (24,000 messages/s) everywhere else. Spread traffic over many distinct group IDs. ⚠️ Changed: high throughput mode lifts FIFO limits far above 300/3,000 per second. Older material may say FIFO queues are capped at 300 messages/s, or 3,000 with batching.
• In-flight limit: 120,000 (§1). Delay: queue-level only (§4). DLQ: must be FIFO (§3).
• ReceiveRequestAttemptId: when a ReceiveMessage call fails on the network, retrying with the same ID (valid for 5 minutes) returns the same messages and receipt handles.
• Lambda reads FIFO queues in batches of up to 10 (see notes/lambda-guide.md); SNS FIFO topics fan out to FIFO queues (§9).
• FIFO requests cost more than standard ones: choose FIFO only when order or deduplication is a requirement.

Developer angle

aws sqs create-queue --queue-name payments.fifo \
  --attributes FifoQueue=true,DeduplicationScope=messageGroup,FifoThroughputLimit=perMessageGroupId
aws sqs send-message --queue-url "$FIFO_URL" \
  --message-body '{"account":"acct-42","op":"debit","amount":25}' \
  --message-group-id acct-42 --message-deduplication-id pay-000981

• In SendMessageBatch, every entry carries its own MessageGroupId and MessageDeduplicationId; batching messages of the same group makes best use of a partition's capacity.
• Group ID = the entity whose events must stay ordered (customer, order, device), never a constant. Deduplication ID = the business operation (order ID plus version), not a random UUID generated per attempt.
• IAM actions are the same as for standard queues (sqs:SendMessage, sqs:ReceiveMessage, …).

Traps

• Trap: "Rename the queue with .fifo to convert it." → Reality: the type is fixed at creation; create a new queue.
• Trap: "Use one MessageGroupId for everything to get global order at full speed." → Reality: one group is one lane and serialises the consumers.
• Trap: "Content-based deduplication compares body and attributes." → Reality: body only.
• Trap: "Deduplication lasts 24 hours." → Reality: 5 minutes.
• Trap: "FIFO means a slow consumer never sees a message twice." → Reality: if processing outlives the visibility timeout, the message is received again.
• Trap: "Give each retry a fresh random deduplication ID." → Reality: every retry then counts as a new message.

Practice questions

1. A ledger service must apply debit and credit events in the order they were produced for each customer account. There are thousands of active accounts, and events for different accounts may be processed in parallel. Which design meets these requirements?

   • A. An SQS FIFO queue with the same MessageGroupId value on every message.
   • B. An SQS standard queue whose consumers sort each batch by SentTimestamp.
   • C. An SQS standard queue with fair queues, using the account ID as MessageGroupId.
   • D. An SQS FIFO queue with MessageGroupId set to the account ID.

2. A developer is moving an order-intake producer from a standard queue to a new FIFO queue. The producer retries SendMessage after network timeouts, and each retry rebuilds the message body with a fresh requestedAt timestamp. Duplicate orders must not reach consumers. Which TWO actions should the developer take? (Select TWO.)

   • A. Create the queue as orders.fifo with the FifoQueue attribute set to true.
   • B. Enable ContentBasedDeduplication and stop sending a deduplication ID.
   • C. Set the DelaySeconds parameter on each retried message.
   • D. Send every attempt with the same MessageDeduplicationId, derived from the order ID.
   • E. Add the .fifo suffix to the existing standard queue's name.

6. SQS security and access policies

Concept

• Three layers protect queue data: TLS in transit (HTTPS endpoints), server-side encryption at rest (SQS-owned keys or KMS keys) and access control (IAM identity policies plus the queue's own resource policy).
• IAM policies cover principals in the queue's own account. The queue policy is needed when the caller is another account or an AWS service (SNS, S3, EventBridge) that sends messages on its own behalf.

Exam facts

• SSE-SQS (SQS-owned keys, AES-256, no extra cost) is applied by default to new queues created without encryption attributes. ⚠️ Changed: new queues are encrypted with SSE-SQS by default (since 2022). Older material may say queues are unencrypted unless SSE is enabled with a KMS key. A queue created through the plain-HTTP endpoint doesn't get this default.
• SSE-KMS: KmsMasterKeyId = alias/aws/sqs (AWS managed key) or a customer managed key; KmsDataKeyReusePeriodSeconds 60–86,400 s (default 300): a longer period means fewer KMS calls and lower cost, a shorter one tighter security. A queue uses one SSE option at a time (SqsManagedSseEnabled or KmsMasterKeyId).
• SSE encrypts the message body only, not queue metadata, message metadata (ID, timestamp, attributes) or metrics. Only messages sent after it's enabled are encrypted, and they stay encrypted if SSE is switched off later.
• On an encrypted queue every request must use HTTPS with SigV4 and anonymous requests are rejected (InvalidSecurity).
• KMS permissions: consumers need kms:Decrypt; producers need kms:GenerateDataKey. Whether producers also need kms:Decrypt is ⚠️ unverified: the SQS documentation history (2024-07-24) says SendMessage no longer requires it, while the key-management page still lists both, so grant both until the docs agree. Failures show up as KmsAccessDenied and other Kms* errors.
• AWS service producers (SNS subscriptions, S3 event notifications, EventBridge) can write to an encrypted queue only when it uses a customer managed key whose key policy lets the service principal call kms:GenerateDataKey and kms:Decrypt. The policy of the aws/sqs key can't be edited, so these integrations fail with it. A queue encrypted with aws/sqs also can't invoke a Lambda function in another account.
• Queue policy (resource policy; up to 8,192 bytes, 20 statements, 50 principals, 10 conditions): grants another account ("Principal": {"AWS": "<account ID or role ARN>"}) or a service principal (sns.amazonaws.com, s3.amazonaws.com) the actions it needs. For service principals add aws:SourceArn (topic or bucket ARN) and/or aws:SourceAccount to prevent the confused-deputy problem.
• Cross-account access needs both sides: the queue policy in the owner account and an IAM policy on the caller in its own account.
• Actions that can't be granted cross-account: CreateQueue, DeleteQueue, SetQueueAttributes, AddPermission, RemovePermission, ListQueues, ListQueueTags, TagQueue, UntagQueue, StartMessageMoveTask, CancelMessageMoveTask, ListMessageMoveTasks.
• Enforce TLS with a Deny statement on "aws:SecureTransport": "false". Private access from a VPC goes through an interface endpoint (condition aws:SourceVpce): see notes/vpc-guide.md.
• ABAC: queue tags can drive access through aws:ResourceTag/<key> conditions. A queue policy that denies every principal can be removed with the AWS managed policy SQSUnlockQueuePolicy (added 2024-11-15).

Developer angle

Queue policy for S3 event notifications, protected against the confused deputy:

{"Version": "2012-10-17",
 "Statement": [{"Sid": "AllowUploadsBucket", "Effect": "Allow",
   "Principal": {"Service": "s3.amazonaws.com"},
   "Action": "sqs:SendMessage",
   "Resource": "arn:aws:sqs:eu-west-1:111122223333:uploads-queue",
   "Condition": {"ArnLike": {"aws:SourceArn": "arn:aws:s3:::uploads-bucket"},
                 "StringEquals": {"aws:SourceAccount": "111122223333"}}}]}

• For a consumer role in another account, the statement uses "Principal": {"AWS": "arn:aws:iam::444455556666:role/OrderConsumer"} with sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility and sqs:GetQueueAttributes on the queue ARN; the role also needs the same actions in its own IAM policy.

Key policy statement on the customer managed key of an encrypted queue fed by SNS (S3 and EventBridge use s3.amazonaws.com and events.amazonaws.com):

{"Sid": "AllowSnsToUseTheKey", "Effect": "Allow", "Principal": {"Service": "sns.amazonaws.com"},
 "Action": ["kms:GenerateDataKey", "kms:Decrypt"], "Resource": "*"}

# switch to the customer managed key; a longer reuse period means fewer KMS calls
aws sqs set-queue-attributes --queue-url "$QURL" \
  --attributes KmsMasterKeyId=alias/orders-cmk,KmsDataKeyReusePeriodSeconds=3600

• The queue policy is the Policy attribute, passed as a JSON string (SetQueueAttributes, or Policy on CreateQueue).

Traps

• Trap: "An IAM policy in the queue's account is enough for another account's role." → Reality: the queue policy must allow it and the role's own IAM policy too.
• Trap: "S3 can send events to a queue encrypted with aws/sqs once the queue policy allows it." → Reality: it also needs a customer managed key whose policy allows s3.amazonaws.com.
• Trap: "SSE hides message attributes." → Reality: only the body is encrypted.
• Trap: "Turning on SSE encrypts the existing backlog." → Reality: only messages sent afterwards.
• Trap: "Grant the partner account SetQueueAttributes in the queue policy." → Reality: that action can't be shared cross-account.
• Trap: "Consumers need kms:GenerateDataKey." → Reality: consumers need kms:Decrypt.

Practice questions

1. A developer switched an SQS queue to SSE-KMS with the aws/sqs key, and the S3 event notifications it receives stopped arriving. The queue policy already allows s3.amazonaws.com to call sqs:SendMessage for the bucket, and the data must stay encrypted with KMS. What should the developer do?

   • A. Add the bucket's ARN as an aws:SourceArn condition and the bucket owner's account as an aws:SourceAccount condition to the queue policy.
   • B. Turn off server-side encryption on the queue.
   • C. Encrypt the queue with a customer managed key whose key policy lets s3.amazonaws.com use it.
   • D. Attach an IAM policy that grants kms:* on the aws/sqs key to the bucket owner's role.

2. An application in account 444455556666 must receive and delete messages from a queue in account 111122223333. Which TWO actions are required? (Select TWO.)

   • A. Grant the consumer role sqs:SetQueueAttributes in the queue policy.
   • B. Add a queue policy statement in account 111122223333 that allows the consumer role from account 444455556666 to call sqs:ReceiveMessage and sqs:DeleteMessage.
   • C. Attach an IAM policy to the consumer role in account 444455556666 that allows sqs:ReceiveMessage and sqs:DeleteMessage on the queue's ARN.
   • D. Create an IAM user in account 111122223333 and store its access keys in the application.
   • E. Set the queue policy's principal to "*" with no conditions.

7. SNS topics and subscriptions

Concept

• Amazon SNS is push-based publish/subscribe: a publisher sends one message to a topic, and SNS delivers a copy to every subscription whose filter policy (if any) matches. The publisher doesn't know who listens, and subscribers come and go without touching it.
• Subscription protocols: sqs, lambda, firehose, http/https, email, email-json, sms and application (mobile push endpoints).
• A standard topic gives high throughput, best-effort ordering and at-least-once delivery. It doesn't keep messages for later readers: SNS retries according to a delivery policy and then drops the message, unless the subscription has a dead-letter queue. (FIFO topics can archive: §9.)

Exam facts

• Quotas: 100,000 standard and 1,000 FIFO topics per account; 12,500,000 subscriptions per standard topic and 100 per FIFO topic (Firehose: 5 per topic per subscription owner); 5,000 pending subscriptions per account. Topic names have 1–256 characters (letters, digits, -, _); type and name can't change after creation. CreateTopic is idempotent: an existing name returns that topic's ARN.
• Message size: default maximum 256 KiB; the topic attribute MaximumMessageSize accepts 1,024 bytes to 1 MiB. Above 256 KiB a topic may have at most 100 subscriptions, all SQS, Lambda or Firehose. Beyond 1 MiB: the SNS Extended Client Library (Java, Python; payload in S3, up to 2 GB). ⚠️ Changed: a topic can accept messages up to 1 MiB once MaximumMessageSize is raised (since 2026-09-18, standard and FIFO topics). Older material may say SNS messages are limited to 256 KB. For SQS subscribers the topic's limit applies, not the queue's own MaximumMessageSize.
• Billing: on standard topics each 64 KB chunk published is billed as one request (1 MiB = 16 requests); FIFO topics bill per message (minimum 1 KB) plus per GB of payload.
• Publishing: Publish (one message; Subject under 100 characters for email, MessageAttributes, MessageStructure=json for a different payload per protocol, which requires a top-level default key) and PublishBatch (up to 10 entries whose combined size stays within MaximumMessageSize). Publish throughput is a soft per-account, per-Region quota (for example 30,000 messages/s in US East (N. Virginia)); above it calls are throttled.
• Confirmation: HTTP/S and email endpoints, and endpoints in another account, must confirm with ConfirmSubscription (the token is valid for 2 days). When the owner of a queue subscribes it, the subscription is confirmed at once. ReturnSubscriptionArn returns the ARN even while the subscription is pending.
• Delivery retries: AWS-managed endpoints (SQS, Lambda) get 100,015 attempts over 23 days; email, SMS and mobile push get a fixed 50 attempts over 6 hours; HTTP/S uses a delivery policy you can customise at topic or subscription level (numRetries 0–100, delays up to 3,600 s, total retry time capped at 3,600 s, throttlePolicy.maxReceivesPerSecond). HTTP 5XX and 429 responses are retried; other codes are permanent failures.
• Client-side errors (endpoint deleted, a policy blocks SNS) aren't retried: with a DLQ the message lands there, without one it's lost.
• Subscription DLQ: RedrivePolicy on the subscription (not the topic) with the deadLetterTargetArn of an SQS queue in the same account and Region; standard topic → standard queue, FIFO topic → FIFO queue. The DLQ's queue policy must let SNS send, and an encrypted DLQ needs a customer managed key that SNS may use.
• Raw message delivery (RawMessageDelivery=true; SQS, HTTP/S and Firehose subscriptions) drops the JSON envelope (Type, MessageId, TopicArn, Signature…). SQS raw delivery passes at most 10 message attributes; a message with more is discarded as a client error.
• Lambda subscribers are invoked asynchronously (payload up to 1 MiB); retries and destinations then follow Lambda's asynchronous rules: see notes/lambda-guide.md.
• Encryption: SSE with a symmetric KMS key (KmsMasterKeyId) is off unless configured, and it encrypts the body only (not subject, attributes or metadata). Publishers need kms:GenerateDataKey* and kms:Decrypt. AWS services that publish to an encrypted topic (CloudWatch alarms, EventBridge, S3, RDS…) need a customer managed key whose policy grants their service principal; for a few (Auto Scaling, CodePipeline, Elastic Beanstalk…) the key policy names an IAM role instead of a service principal.
• Topic policy: by default only the owner can publish or subscribe. Grant other accounts or service principals there, scoped with aws:SourceArn / aws:SourceAccount.
• Delivery status logging to CloudWatch Logs per protocol (SQSSuccessFeedbackRoleArn, LambdaFailureFeedbackRoleArn, …, sample rate 0–100 %). Metrics: NumberOfMessagesPublished, NumberOfNotificationsDelivered, NumberOfNotificationsFailed, NumberOfNotificationsFilteredOut.
• Mobile push: create a platform application (push-service credentials), then a platform endpoint per device token, then publish to the endpoint ARN or to a topic the endpoints subscribe to. Supported services: APNs, Firebase Cloud Messaging (FCM), ADM, Baidu, MPNS and WNS. ⚠️ Changed: Android push goes through FCM with HTTP v1 credentials (supported since 2024-01-18; SNS still names the platform and payload key GCM). Older material may say Google GCM.
• SMS goes through AWS End User Messaging SMS (new senders start in a sandbox; default spend threshold USD 1.00); email delivery is capped at 10 messages per second per subscription.
• ⚠️ Changed: SNS message data protection (data protection policies on topics) is closed to new customers since 2026-04-30. Older material may say it is available to every account.

Developer angle

TOPIC_ARN=$(aws sns create-topic --name order-events --query TopicArn --output text)
aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol https \
  --notification-endpoint https://hooks.example.com/orders --return-subscription-arn
aws sns publish --topic-arn "$TOPIC_ARN" --subject "Order placed" \
  --message '{"orderId":"o-1001","total":42.5}' \
  --message-attributes '{"eventType":{"DataType":"String","StringValue":"order_placed"}}'
# keep what the retry policy can't deliver
aws sns set-subscription-attributes --subscription-arn "$SUB_ARN" \
  --attribute-name RedrivePolicy \
  --attribute-value '{"deadLetterTargetArn":"arn:aws:sqs:eu-west-1:111122223333:order-events-dlq"}'
# accept payloads up to 1 MiB (only SQS, Lambda and Firehose subscriptions, at most 100)
aws sns set-topic-attributes --topic-arn "$TOPIC_ARN" \
  --attribute-name MaximumMessageSize --attribute-value 1048576

• HTTP/S DeliveryPolicy JSON (topic or subscription attribute): healthyRetryPolicy (minDelayTarget, maxDelayTarget, numRetries, numNoDelayRetries, numMinDelayRetries, numMaxDelayRetries, backoffFunction arithmetic/exponential/geometric/linear), throttlePolicy.maxReceivesPerSecond and requestPolicy.headerContentType; the total retry time must stay within 3,600 s.
• IAM: sns:Publish for publishers; sns:Subscribe and sns:SetSubscriptionAttributes for whoever manages subscriptions; scope Resource to the topic ARN.
• Errors: AuthorizationError (403), InvalidParameter (for example a message larger than MaximumMessageSize), NotFound, InvalidSecurity, KMSAccessDenied and the other KMS* codes on Publish; SubscriptionLimitExceeded and FilterPolicyLimitExceeded on Subscribe.

Traps

• Trap: "SNS keeps messages until every subscriber has read them." → Reality: a standard topic only retries; a failed delivery without a DLQ is dropped.
• Trap: "Attach the DLQ to the topic." → Reality: DLQs belong to subscriptions.
• Trap: "Give the SQS subscription a custom retry policy." → Reality: only HTTP/S delivery policies can be customised.
• Trap: "A partner's queue in another account receives messages right after Subscribe." → Reality: it stays pending until the queue owner confirms (unless the queue owner created the subscription).
• Trap: "Raise MaximumMessageSize to 1 MiB on a topic with email subscribers." → Reality: above 256 KiB only SQS, Lambda and Firehose subscriptions (at most 100) are allowed.
• Trap: "Stretch the HTTP delivery policy to survive a 3-hour outage." → Reality: total retry time is capped at 3,600 s; add a subscription DLQ.
• Trap: "Use SNS email subscriptions to email each customer." → Reality: subscriptions are fixed, confirmed endpoints; transactional email is Amazon SES (see notes/other-services-guide.md).

Practice questions

1. A company's SNS topic delivers order events to a partner's HTTPS webhook. The partner's endpoint is sometimes unavailable for up to 2 hours during maintenance, and no event may be lost. What should the developer do with the LEAST operational overhead?

   • A. Give the subscription an SQS dead-letter queue and redeliver its messages afterwards.
   • B. Set the subscription's delivery policy to 100 retries with a maximum delay of 3,600 seconds.
   • C. Enable raw message delivery on the HTTPS subscription.
   • D. Publish every event twice, five minutes apart.

2. A developer encrypted an SNS topic with the alias/aws/sns key, and CloudWatch alarm notifications stopped reaching the topic. Security requires the topic to stay encrypted. Which TWO actions fix the problem? (Select TWO.)

   • A. Enable raw message delivery on all subscriptions.
   • B. Configure the alarm action with an IAM user's access keys.
   • C. Turn off server-side encryption on the topic.
   • D. Create a customer managed KMS key and set it as the topic's KmsMasterKeyId.
   • E. Add a key policy statement that allows cloudwatch.amazonaws.com to use kms:GenerateDataKey* and kms:Decrypt.

8. SNS fan out and message filtering

Concept

• Fan-out: publish once to an SNS topic and subscribe one SQS queue per consuming service. Each service gets its own durable copy, reads at its own pace, retries and dead-letters on its own, and new services subscribe without any change to the publisher.
• Filter policies let each subscription receive only the messages it cares about, so one topic serves many consumers interested in different events, with no topic per event type and no filtering code.

Exam facts

• The queue policy must allow sns.amazonaws.com to call sqs:SendMessage on the queue, scoped with aws:SourceArn = the topic ARN (one statement per topic, or aws:SourceAccount when many topics of one account publish). Without it, deliveries fail; check NumberOfNotificationsFailed and the delivery status logs. An encrypted subscriber queue also needs a customer managed key that SNS may use (§6).
• Cross-account: a topic in account A can deliver to a queue in account B. The subscription stays pending until B's queue owner confirms it, unless B creates the subscription itself (A's topic policy must then allow B sns:Subscribe).
• Cross-Region: SNS delivers to SQS queues and Lambda functions in other Regions. When an opt-in Region is involved, the queue or function policy names the Regional principal sns.<region>.amazonaws.com. Run Subscribe in the topic's Region.
• One S3 event to several queues: S3 allows one notification destination per event type and overlapping prefix/suffix, so send it to an SNS topic and fan out from there (or use EventBridge): see notes/s3-guide.md.
• SNS → Firehose subscriptions archive messages to S3 and other Firehose destinations (SubscriptionRoleArn required).
• A standard topic forwards an optional MessageGroupId only to SQS standard subscriptions, where it activates fair queues (§1).
• Filter policy scope: FilterPolicyScope = MessageAttributes (default) or MessageBody (the payload must be a well-formed JSON object; nested keys allowed). ⚠️ Changed: filter policies can match the message body (since 2022-11-22). Older material may say SNS filters only on message attributes.
• Matching rules: keys are ANDed; the values listed for one key are ORed; the $or operator ORs whole conditions; string matching is case-sensitive unless equals-ignore-case is used. Attributes of type String, String.Array and Number are matched; Binary attributes are ignored.
• Operators: exact string, anything-but (also with a prefix), prefix, suffix, equals-ignore-case, numeric (exact, ranges, anything-but), exists, IP address matching (cidr) and wildcard. ⚠️ Changed: suffix, equals-ignore-case and $or were added on 2023-11-16. Older material may say only exact, prefix, anything-but, numeric and exists matching are available.
• No filter policy → the subscription receives everything. Messages that don't match aren't delivered and are counted in NumberOfNotificationsFilteredOut. (Lambda event source mappings and EventBridge rules have their own filters: see notes/lambda-guide.md and notes/monitoring-guide.md.)
• Limits: 5 keys per policy (leaf keys for body policies), 150 value combinations (the product of the array sizes), 256 KB per policy, 200 filter policies per topic and 10,000 per account; numeric values between −10⁹ and 10⁹ with 5 decimal places.
• Adding or changing a filter policy takes up to 15 minutes to apply everywhere.

Developer angle

QUEUE_ARN=$(aws sqs get-queue-attributes --queue-url "$REFUNDS_URL" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)
SUB_ARN=$(aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs \
  --notification-endpoint "$QUEUE_ARN" --attributes RawMessageDelivery=true \
  --return-subscription-arn --query SubscriptionArn --output text)
aws sns set-subscription-attributes --subscription-arn "$SUB_ARN" \
  --attribute-name FilterPolicyScope --attribute-value MessageBody
aws sns set-subscription-attributes --subscription-arn "$SUB_ARN" \
  --attribute-name FilterPolicy --attribute-value file://refunds-filter.json

refunds-filter.json (body scope; nested keys; AND across keys):

{
  "order": {"status": ["REFUNDED"], "total": [{"numeric": [">=", 100]}]},
  "channel": [{"anything-but": ["test"]}]
}

An attribute-scope policy mixes OR inside a key with $or across keys, for example {"eventType": ["order_placed", {"prefix": "order_ship"}], "$or": [{"priority": ["high"]}, {"amount": [{"numeric": [">", 1000]}]}]}.

• The queue policy that lets the topic deliver has the shape of the S3 example in §6, with "Principal": {"Service": "sns.amazonaws.com"}, "Action": "sqs:SendMessage" on the queue ARN and "Condition": {"ArnEquals": {"aws:SourceArn": "<topic ARN>"}}.
• Without raw delivery, the consumer parses the SNS envelope and reads its Message field; with raw delivery, the SQS body is the published message itself.

Traps

• Trap: "Create one topic per consumer to route events." → Reality: use one topic plus filter policies; otherwise the publisher must know every consumer.
• Trap: "Filter on a body field with the default scope." → Reality: the default scope reads message attributes; set FilterPolicyScope=MessageBody.
• Trap: "All values listed for one key must match." → Reality: any one of them matches (OR); different keys are ANDed.
• Trap: "A new filter policy applies to the very next message." → Reality: allow up to 15 minutes.
• Trap: "The subscription exists, so the queue will receive messages." → Reality: only if the queue policy (and, for encrypted queues, the KMS key policy) lets SNS write.
• Trap: "Filter on a Binary attribute." → Reality: binary attributes are ignored by filtering.

Practice questions

1. An order service publishes JSON payloads such as {"status":"REFUNDED","orderId":"o-17"} to an SNS topic without message attributes. A refunds queue subscribed to the topic must receive only refunded orders, and the publisher's code must not change. What should the developer do?

   • A. Set an attribute-based filter policy {"status": ["REFUNDED"]} on the subscription.
   • B. Create a separate topic for refunded orders and have the publisher send refunds to it.
   • C. Set FilterPolicyScope to MessageBody on the subscription with the policy {"status": ["REFUNDED"]}.
   • D. Subscribe a Lambda function to the topic that drops non-refund messages and forwards refunds to the queue.

2. A developer subscribes a new SQS queue to an SNS topic in the same account. The subscription shows as confirmed, but no messages arrive. The queue uses SSE-KMS with the aws/sqs key and has no queue policy. Which TWO changes are required? (Select TWO.)

   • A. Add a queue policy that allows sns.amazonaws.com to call sqs:SendMessage with aws:SourceArn set to the topic ARN.
   • B. Enable raw message delivery on the subscription.
   • C. Increase the queue's visibility timeout.
   • D. Grant sns:Publish on the topic to the queue's consumers.
   • E. Re-encrypt the queue with a customer managed key whose key policy allows sns.amazonaws.com to use kms:GenerateDataKey and kms:Decrypt.

9. SNS FIFO topics and message archiving

Concept

• An SNS FIFO topic keeps publish order within each message group, drops duplicate publishes and delivers to subscribed SQS queues. With SQS FIFO queues as subscribers it gives ordered, deduplicated fan-out: every consuming service sees the same sequence.
• FIFO topics can also archive what is published and let a subscriber replay a time window, for example after a downstream bug or to seed a new environment.

Exam facts

• The name ends in .fifo, set with FifoTopic=true at creation. Every Publish needs a MessageGroupId; MessageDeduplicationId is required unless ContentBasedDeduplication=true (SHA-256 of the body, attributes excluded). Duplicates within the 5-minute interval are accepted but not delivered. Publish returns a 128-bit SequenceNumber that keeps increasing within each message group.
• Subscribers: SQS FIFO queues (order and deduplication preserved) and SQS standard queues (lower cost, best-effort order, possible duplicates, no group ID passed). Email, SMS, mobile push and HTTP(S) endpoints can't subscribe; to reach Lambda, subscribe a queue and let the queue trigger the function. ⚠️ Changed: FIFO topics also deliver to SQS standard queues (since 2023-09-14). Older material may say only SQS FIFO queues can subscribe. Quotas: 1,000 FIFO topics per account, 100 subscriptions per FIFO topic; a FIFO topic's subscription DLQs are SQS FIFO queues.
• Throughput: each message group delivers at most 300 messages/s. With FifoThroughputScope=Topic (default), deduplication spans the whole topic and the topic handles up to 3,000 messages/s or 20 MB/s. With FifoThroughputScope=MessageGroup (since 2025-01-21), deduplication is per message group and per-topic throughput rises up to the account's Regional publish quota (for example 30,000 messages/s in US East (N. Virginia)); this change can't be reverted, and subscribed SQS FIFO queues should use high throughput mode too. ⚠️ Changed: SNS FIFO throughput is set per message group and per topic as above. Older material may say an SNS FIFO topic is limited like an SQS FIFO queue (300 messages/s, 3,000 with batching).
• Exactly-once end to end needs an SQS FIFO subscriber whose policy lets SNS deliver, consumers that delete before the visibility timeout, no network disruption and no filter policy (with filtering, delivery to that subscription is at-most-once).
• Archive: topic attribute ArchivePolicy = {"MessageRetentionPeriod": "<days>"} with 1–365 days; FIFO (application-to-application) topics only; extra cost. GetTopicAttributes returns BeginningArchiveTime. A topic with an active archive policy can't be deleted; deactivating the policy (set it to {}) deletes the archive.
• Replay: subscription attribute ReplayPolicy with PointType Timestamp, StartingPoint (required) and EndingPoint (optional). Replayed messages keep their MessageId and Timestamp and carry a Replayed attribute; the subscription's filter policy still applies. With an EndingPoint the subscription pauses afterwards until a new replay policy without an end resumes it. Encrypted topics need a key policy that lets sns.amazonaws.com decrypt the archive. Progress shows in ReplayStatus (Pending, In progress, Completed, Failed).
• ⚠️ Changed: SNS FIFO topics can store messages for up to 365 days and replay them (since 2023-10-26). Older material may say SNS never persists messages; standard topics still don't.

Developer angle

FIFO_TOPIC=$(aws sns create-topic --name price-updates.fifo \
  --attributes FifoTopic=true,ContentBasedDeduplication=true,FifoThroughputScope=MessageGroup \
  --query TopicArn --output text)
aws sns publish --topic-arn "$FIFO_TOPIC" \
  --message '{"sku":"P-214","price":19.99}' --message-group-id P-214
# keep 30 days of messages for replay
aws sns set-topic-attributes --topic-arn "$FIFO_TOPIC" \
  --attribute-name ArchivePolicy --attribute-value '{"MessageRetentionPeriod":"30"}'
# redeliver a two-hour window to one subscription
aws sns set-subscription-attributes --subscription-arn "$SUB_ARN" \
  --attribute-name ReplayPolicy \
  --attribute-value '{"PointType":"Timestamp","StartingPoint":"2026-10-10T08:00:00.000Z","EndingPoint":"2026-10-10T10:00:00.000Z"}'

• Without content-based deduplication, also pass --message-deduplication-id (SDK: MessageDeduplicationId) built from the business operation, such as P-214-v57, so retries reuse it.
• IAM: sns:SetTopicAttributes for the archive policy (topic owner) and sns:SetSubscriptionAttributes for the replay policy (subscriber).

Traps

• Trap: "Subscribe a Lambda function directly to the FIFO topic." → Reality: FIFO topics deliver only to SQS queues; put a queue in between.
• Trap: "A standard queue subscriber keeps the FIFO order." → Reality: only FIFO queues preserve order and deduplication.
• Trap: "Switch FifoThroughputScope back to Topic after the peak." → Reality: the change to MessageGroup is one-way.
• Trap: "Archive a standard topic to replay it later." → Reality: archiving exists for FIFO topics only.
• Trap: "Replays bypass filter policies." → Reality: replayed messages are filtered like new ones.
• Trap: "Use one message group ID for all products." → Reality: one group delivers at most 300 messages/s.

Practice questions

1. A pricing service publishes price changes that a wholesale service and a retail service must apply in order for each product. An analytics service also needs every change, doesn't care about order and should cost as little as possible. Which design meets these requirements?

   • A. An SNS FIFO topic with SQS FIFO queues for wholesale and retail and a Lambda function subscribed directly for analytics.
   • B. An SNS FIFO topic with SQS FIFO queues for wholesale and retail and an SQS standard queue for analytics.
   • C. An SNS standard topic with three SQS standard queues whose consumers sort by SentTimestamp.
   • D. A producer that writes each change separately to three SQS FIFO queues.

2. A developer finds that a consumer bug corrupted two hours of price updates read from an SQS FIFO queue subscribed to an SNS FIFO topic. The topic has had an archive policy with 30 days of retention for months. The updates must be processed again without asking the publisher to resend them. Which TWO actions should the developer take? (Select TWO.)

   • A. Call StartMessageMoveTask with the topic ARN as the source.
   • B. Purge the subscribed queue and wait for SNS to redeliver automatically.
   • C. Deploy the fixed consumer before the replay starts.
   • D. Set a ReplayPolicy on the queue's subscription with StartingPoint and EndingPoint around the two-hour window.
   • E. Change the topic's FifoThroughputScope to Topic.

10. Kinesis Data Streams shards and capacity modes

Concept

• Kinesis Data Streams (KDS) stores an ordered, replayable sequence of records. Consumers read by position and remove nothing; records expire only when the retention period ends, so several applications read the same data independently and can read it again.
• A stream is made of shards. Each record carries a partition key; KDS hashes the key with MD5 to a 128-bit number and stores the record in the shard whose hash key range contains it. Same key → same shard → strict order for that key; there is no ordering across shards.
• Capacity is either provisioned (you choose and change the shard count) or on-demand (KDS manages shards for you).

Exam facts

• Per shard (provisioned): writes up to 1 MB/s or 1,000 records/s; reads up to 2 MB/s shared by all standard consumers and at most 5 GetRecords calls per second; enhanced fan-out consumers get their own 2 MB/s each (§12).
• Sizing: shards = the larger of write KB/s ÷ 1,024 and read KB/s ÷ 2,048, rounded up, where read KB/s = write KB/s × the number of standard consumers; also check records/s ÷ 1,000.
• Record: partition key (1–256 Unicode characters) + data blob (any bytes, base64 on the wire). The maximum record size (key + data, before base64) is 1 MiB by default and can be raised per stream to 10 MiB with UpdateMaxRecordSize (or MaxRecordSizeInKiB on CreateStream). ⚠️ Changed: records can be up to 10 MiB once a stream opts in (since 2025-10-28; not in every Region). Older material may say records are limited to 1 MB. Shard limits don't change: large records draw on burst capacity, so keep them below about 2 % of traffic.
• PutRecords: up to 500 records and 10 MiB per request, partition keys included. ⚠️ Changed: 10 MiB per request (since 2025-10-28). Older material may say 5 MB.
• Sequence numbers are unique within the stream and increase over time within a shard. PutRecord with SequenceNumberForOrdering guarantees strictly increasing numbers for one producer and key; ExplicitHashKey overrides the key-to-shard mapping.
• Retention: records are immutable (no update, reordering or single delete) and expire only with the retention period: default 24 hours; IncreaseStreamRetentionPeriod up to 8,760 hours (365 days), DecreaseStreamRetentionPeriod down to 24 hours. Retention above 24 hours costs extra; after a decrease, older records become unreadable almost immediately.
• Provisioned mode: pay per shard-hour plus PUT payload units (25 KB chunks). Scale with UpdateShardCount (UNIFORM_SCALING; by default at most 10 scaling operations per rolling 24 hours, at most double or half the current count per call, 10,000 shards per stream) or adjust single shards with SplitShard and MergeShards. While resharding, the stream is UPDATING but stays readable and writable; parent shards stay readable until their records expire.
• On-demand Standard: no shard management. A new stream starts with 4 MB/s of write and 8 MB/s of read capacity and handles up to double its peak write throughput of the previous 30 days; jumping past double the previous peak within 15 minutes can throttle, and a single partition key still can't exceed one shard's 1 MB/s or 1,000 records/s. It scales up to 10 GB/s of writes in US East (N. Virginia), US West (Oregon) and Europe (Ireland) and 200 MB/s by default elsewhere; 50 on-demand streams per account by default; you pay per stream-hour plus per GB written and read. The course's "4,000 records per second" on-demand default is ⚠️ unverified.
• On-demand Advantage (an account-level setting for all on-demand streams in a Region): no per-stream charge and data prices at least 60 % lower, against a commitment of at least 25 MiB/s ingest and 25 MiB/s retrieval across the account's on-demand streams; adds warm throughput (UpdateStreamWarmThroughput, capacity ready before a known peak) and up to 50 enhanced fan-out consumers per stream; it can be switched off only after 24 hours. ⚠️ Changed: a third capacity mode, On-demand Advantage, exists (launch date ⚠️ unverified). Older material may say the only modes are provisioned and on-demand.
• UpdateStreamMode switches a stream between provisioned and on-demand at most twice per 24 hours, without downtime; the shard count carries over.
• Record distribution strategy (on-demand streams only): USER_PARTITION_KEY (default) or AUTO, which spreads records evenly and ignores partition keys and ExplicitHashKey (so no per-key ordering); set with UpdateStreamRecordDistributionStrategy. ⚠️ Changed: on-demand streams can ignore partition keys with the AUTO strategy (launch date ⚠️ unverified). Older material may say the partition key always decides the shard.
• Encryption: SSE with KMS (StartStreamEncryption, the aws/kinesis key or a customer managed key) encrypts records written after it's enabled. Producers need kms:GenerateDataKey, consumers kms:Decrypt; the stream ARN is passed as encryption context (kms:EncryptionContext:aws:kinesis:arn). Sharing an encrypted stream with another account through a resource policy needs a customer managed key. In transit: HTTPS.
• Default shard quota: 20,000 per account in US East (N. Virginia), US West (Oregon) and Europe (Ireland), 1,000 or 6,000 elsewhere; raise it through Service Quotas.

Developer angle

aws kinesis create-stream --stream-name clicks --shard-count 4            # provisioned
aws kinesis create-stream --stream-name telemetry \
  --stream-mode-details StreamMode=ON_DEMAND                                # on-demand
aws kinesis update-shard-count --stream-name clicks \
  --target-shard-count 8 --scaling-type UNIFORM_SCALING
aws kinesis increase-stream-retention-period --stream-name clicks --retention-period-hours 168
aws kinesis start-stream-encryption --stream-name clicks \
  --encryption-type KMS --key-id alias/aws/kinesis
STREAM_ARN=$(aws kinesis describe-stream-summary --stream-name telemetry \
  --query StreamDescriptionSummary.StreamARN --output text)
aws kinesis update-max-record-size --stream-arn "$STREAM_ARN" --max-record-size-in-ki-b 4096

Sizing example: 1,500 records/s of 2 KB each, read by three shared-throughput consumer applications.

• IAM: producers kinesis:PutRecord, kinesis:PutRecords (+ kms:GenerateDataKey); consumers kinesis:GetRecords, kinesis:GetShardIterator, kinesis:DescribeStreamSummary, kinesis:ListShards (+ kms:Decrypt).
• Errors: ProvisionedThroughputExceededException (§12), ResourceInUseException (a control-plane call such as UpdateShardCount while the stream isn't ACTIVE; reads and writes continue during resharding), LimitExceededException (too many streams CREATING, shard quota), ValidationException (UpdateShardCount on an on-demand stream), InvalidArgumentException, KMSAccessDeniedException.

Traps

• Trap: "Consumers delete records after reading them, as in SQS." → Reality: records stay until retention ends; consumers track their own position.
• Trap: "A stream keeps one global order." → Reality: order holds per shard, that is per partition key.
• Trap: "On-demand streams never throttle." → Reality: beyond double the previous peak within 15 minutes, or above 1 MB/s on a single key, they do.
• Trap: "Records can be 10 MiB by default." → Reality: 1 MiB unless the stream's maximum is raised.
• Trap: "Call UpdateShardCount on an on-demand stream." → Reality: provisioned only (ValidationException).
• Trap: "Lowering retention is harmless." → Reality: older records become unreadable almost immediately.

Practice questions

1. A developer is sizing a provisioned Kinesis data stream that receives 4,000 records per second of 1 KB each. Two consumer applications read every record with shared-throughput GetRecords calls. What is the minimum number of shards that avoids throttling?

   • A. 2
   • B. 4
   • C. 6
   • D. 8

2. A team ingests IoT telemetry into a provisioned stream. Traffic is unpredictable, the team spends hours splitting and merging shards, and next month's product launch is expected to push the peak to three times today's level within minutes. Which TWO actions meet these requirements with the LEAST operational overhead? (Select TWO.)

   • A. Switch the stream to on-demand capacity with UpdateStreamMode.
   • B. Call UpdateShardCount on the stream every few minutes during the launch.
   • C. Enable On-demand Advantage for the account and set the stream's warm throughput above the expected launch peak.
   • D. Raise the stream's retention period to 7 days.
   • E. Send each record with SequenceNumberForOrdering.

11. Kinesis producers consumers and the KCL

Concept

• Producers write with the SDK (PutRecord, PutRecords), with the Kinesis Producer Library (KPL) for high-volume batching, or with the Kinesis Agent (a Java service on Linux that tails log files into KDS or Firehose).
• Consumers either poll each shard themselves (GetShardIterator, then repeated GetRecords), use the Kinesis Client Library (KCL), which hides shards, leases and checkpoints, or are managed services: Lambda event source mappings, Firehose, Managed Service for Apache Flink.

Exam facts

• PutRecords can partly fail: the call returns HTTP 200 with FailedRecordCount > 0, and failed entries carry ErrorCode ProvisionedThroughputExceededException or InternalFailure. Retry only the failed entries, with backoff. Entries are processed independently, so PutRecords doesn't guarantee order; strict per-key order needs serial PutRecord calls (optionally with SequenceNumberForOrdering).
• Throttling arithmetic: PutRecord rounds each record up to the next KB; PutRecords rounds the total of the request.
• KPL: aggregates many small user records into one Kinesis record (better use of the 1,000 records/s limit), collects records into PutRecords calls, retries automatically, publishes CloudWatch metrics and works asynchronously (futures). It adds up to RecordMaxBufferedTime of delay, so latency-critical producers call the SDK directly; calling flushSync() after every write adds about 500 ms each time (it's meant for shutdown). Aggregated records must be de-aggregated: the KCL and Firehose do it automatically; consumers that call GetRecords themselves use the KPL's Java deaggregation library.
• Shard iterators: TRIM_HORIZON (oldest available record), LATEST (only new records), AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER, AT_TIMESTAMP. An iterator expires 5 minutes after it's issued; every GetRecords response carries NextShardIterator for the next call. GetShardIterator allows 5 calls per second per shard.
• GetRecords: up to 10,000 records or 10 MB per call, 5 calls per second per shard, 2 MB/s per shard shared; after a 10 MB response, calls in the next 5 seconds are throttled.
• KCL: a Java library (other languages through the MultiLangDaemon). Each KCL application keeps a lease table in DynamoDB, named after the application by default, with one row per shard (leaseKey = shard ID, checkpoint, leaseOwner, leaseCounter, parent and child shard IDs). A shard is leased to exactly one worker at a time; one worker can hold several leases, and workers beyond the number of shards sit idle.
• Checkpointing stores the last processed sequence number in the lease table, so a replacement worker resumes from there. Processing is at-least-once, so record handlers must be idempotent. After resharding, the KCL finishes parent shards before their children to keep order.
• Every KCL application needs a unique application name: two applications sharing a name share one lease table and split the shards between them instead of each reading everything.
• KCL 3.x (generally available since 2024-11-06) balances leases by worker CPU utilisation and adds a worker metrics table and a coordinator state table (KCL 3.5 and later can keep everything in the lease table). KCL creates its tables in DynamoDB on-demand mode by default; if they're provisioned, a ProvisionedThroughputExceededException on the lease table means the table needs more capacity. Delete the tables when you retire an application, because they keep costing money. KCL 2.0 and later consume through enhanced fan-out by default (§12).
• ⚠️ Changed: KCL 3.x (GA 2024-11-06) and KPL 1.x (GA 2024-12-15) are current; KCL 1.x and KPL 0.x entered maintenance mode on 2025-04-17 with an end-of-support date of 2026-01-30; KCL 2.x stays generally available. Older material may say KCL 1.x, with a single lease table, is the current library.
• Lambda reads shards through an event source mapping; records larger than 6 MiB go to its on-failure destination. Batch size, ParallelizationFactor (1–10 batches per shard), bisecting and destinations: see notes/lambda-guide.md.
• IAM for a KCL consumer: kinesis:GetRecords, kinesis:GetShardIterator, kinesis:DescribeStreamSummary, kinesis:ListShards (plus kinesis:RegisterStreamConsumer, kinesis:DescribeStreamConsumer and kinesis:SubscribeToShard for fan-out), read/write access to its DynamoDB tables and cloudwatch:PutMetricData.

Developer angle

# write one record (AWS CLI v2 expects base64 blobs unless told otherwise)
aws kinesis put-record --stream-name clicks --partition-key user-42 \
  --cli-binary-format raw-in-base64-out --data '{"page":"/cart"}'
# read one shard from a point in time
IT=$(aws kinesis get-shard-iterator --stream-name clicks \
  --shard-id shardId-000000000000 --shard-iterator-type AT_TIMESTAMP \
  --timestamp 2026-10-12T08:00:00Z --query ShardIterator --output text)
aws kinesis get-records --shard-iterator "$IT" --limit 500

Retrying only the failed entries of PutRecords:

import time
import boto3

kinesis = boto3.client("kinesis")

def put_all(stream_arn, records, attempts=6):
    """records: [{"Data": b"...", "PartitionKey": "user-42"}, ...], at most 500 per call."""
    pending = records
    for attempt in range(attempts):
        resp = kinesis.put_records(StreamARN=stream_arn, Records=pending)
        if resp["FailedRecordCount"] == 0:
            return
        pending = [rec for rec, res in zip(pending, resp["Records"]) if "ErrorCode" in res]
        time.sleep(min(0.1 * 2 ** attempt, 5))   # exponential backoff
    raise RuntimeError(f"{len(pending)} records still failing")

Traps

• Trap: "Run more KCL workers than shards to go faster." → Reality: each shard has one lease owner; extra workers idle.
• Trap: "Two applications can share an application name." → Reality: they'd share one lease table and split the shards between them.
• Trap: "HTTP 200 from PutRecords means everything was written." → Reality: check FailedRecordCount and the per-entry results.
• Trap: "PutRecords keeps the batch order." → Reality: there is no ordering guarantee inside the batch.
• Trap: "The KPL gives the lowest latency." → Reality: it buffers up to RecordMaxBufferedTime; latency-critical code calls the SDK directly.
• Trap: "Keep one shard iterator for hours." → Reality: it expires after 5 minutes; follow NextShardIterator.
• Trap: "Reprocess yesterday's data with LATEST." → Reality: use AT_TIMESTAMP or TRIM_HORIZON.

Practice questions

1. A company runs a KCL consumer application on 10 EC2 instances to process a stream with 6 shards. CloudWatch shows that 4 of the instances process nothing, and the team wants to cut cost without lowering throughput. What should the developer do?

   • A. Raise the write capacity of the application's DynamoDB lease table.
   • B. Register each instance as a separate enhanced fan-out consumer.
   • C. Change the application's initial position to LATEST.
   • D. Reduce the consumer fleet to 6 instances.

2. A developer's producer sends batches with PutRecords. The responses return HTTP 200, but FailedRecordCount is often above zero with ProvisionedThroughputExceededException entries, and downstream counts show missing events. Shard-level metrics show one shard throttled while the others are mostly idle. Which TWO actions should the developer take? (Select TWO.)

   • A. Treat every HTTP 200 response from PutRecords as a full success.
   • B. Resend only the entries that returned an ErrorCode, with exponential backoff.
   • C. Switch to PutRecord calls with SequenceNumberForOrdering.
   • D. Increase the stream's retention period.
   • E. Switch to a partition key with many more distinct values.

12. Kinesis throttling and enhanced fan out

Concept

• ProvisionedThroughputExceededException means a shard-level limit was hit: writes above 1 MB/s or 1,000 records/s on a shard, or reads above 5 GetRecords calls or 2 MB per second per shard. The cure depends on which side throttles and why.
• Hot shard: a few partition keys carry most of the traffic, so one shard throttles while the others idle. More shards don't help as long as the hot key stays on one shard.
• Too many readers: standard consumers share each shard's 2 MB/s and 5 calls/s. Enhanced fan-out (EFO) gives every registered consumer its own 2 MB/s per shard, pushed over HTTP/2.

Exam facts

• Write-side signals: WriteProvisionedThroughputExceeded (stream and shard level), PutRecords.ThrottledRecords, per-entry errors. Fixes: retry with exponential backoff and jitter (SDK retries: see notes/cli-sdk-policies-guide.md), spread the keys (higher cardinality, or a random suffix on a hot key at the cost of per-key order), split the hot shard (SplitShard) or add shards (UpdateShardCount), aggregate small records with the KPL, or move to on-demand.
• Read-side signals: ReadProvisionedThroughputExceeded. Fixes: EFO for each consumer application, fewer polling consumers per stream, polling no faster than needed (about once per second per shard), bigger batches.
• Lag: GetRecords.IteratorAgeMilliseconds (stream level; IteratorAgeMilliseconds per shard) shows how far behind the newest record a consumer is reading; alarm on the Maximum statistic. When it approaches the retention period, records expire unread. EFO consumers report SubscribeToShardEvent.MillisBehindLatest. Fixes: more parallelism (more shards and workers; for Lambda a higher ParallelizationFactor), faster processing, and longer retention as a safety net.
• Enhanced shard-level metrics (EnableEnhancedMonitoring, extra CloudWatch cost) show which shard is hot: IncomingBytes, IncomingRecords, WriteProvisionedThroughputExceeded, ReadProvisionedThroughputExceeded, IteratorAgeMilliseconds, OutgoingBytes, OutgoingRecords. Stream-level metrics are free and arrive every minute.
• On-demand streams throttle too: traffic above double the previous peak within 15 minutes, or one partition key above 1 MB/s or 1,000 records/s. On-demand splits shards evenly and doesn't isolate hot keys; for granular splits use provisioned mode, or the AUTO distribution strategy when per-key order doesn't matter (§10).
• Records above 1 MiB consume burst capacity and can throttle the writes that follow; for a steady flow of big payloads, send S3 pointers instead.
• EFO: RegisterStreamConsumer (up to 20 consumers per stream, 50 with On-demand Advantage; at most 5 being created at a time), then SubscribeToShard per shard: an HTTP/2 push subscription that lasts up to 5 minutes and is then renewed (the KCL does this for you). One SubscribeToShard call per second per consumer per shard; calling again within 5 seconds of a successful call returns ResourceInUseException. ⚠️ Changed: up to 50 EFO consumers per stream with On-demand Advantage. Older material may say 20 is always the maximum.
• EFO delivers in about 70 ms on average regardless of the number of consumers; shared-throughput reads average about 200 ms with one consumer and about 1,000 ms with five.
• EFO costs consumer-shard-hours plus per-GB retrieval (no premium under On-demand Advantage). Use it when several applications read the same stream or latency matters; one or two tolerant consumers share throughput more cheaply.
• Lambda event source mappings can read through an EFO consumer (see notes/lambda-guide.md). Encrypted streams can also hit KMSThrottlingException.

Developer angle

# find the hot shard (shard-level metrics cost extra)
aws kinesis enable-enhanced-monitoring --stream-name clicks \
  --shard-level-metrics IncomingBytes IncomingRecords WriteProvisionedThroughputExceeded IteratorAgeMilliseconds
# split it at a hash key inside its range (read the range from list-shards)
aws kinesis list-shards --stream-name clicks
aws kinesis split-shard --stream-name clicks --shard-to-split shardId-000000000003 \
  --new-starting-hash-key 255211775190703847597530955573826158592
# dedicated 2 MB/s per shard for a new consumer application
CONSUMER_ARN=$(aws kinesis register-stream-consumer --stream-arn "$STREAM_ARN" \
  --consumer-name fraud-scorer --query Consumer.ConsumerARN --output text)
# alarm when consumers fall more than 10 minutes behind
aws cloudwatch put-metric-alarm --alarm-name clicks-consumer-lag \
  --namespace AWS/Kinesis --metric-name GetRecords.IteratorAgeMilliseconds \
  --dimensions Name=StreamName,Value=clicks --statistic Maximum --period 60 \
  --evaluation-periods 5 --threshold 600000 --comparison-operator GreaterThanThreshold

• Write sharding for one very busy key: append a random suffix (device-17#0 … device-17#7) so its records spread over up to 8 shards; the price is that records of that device are no longer guaranteed to arrive in order.

Traps

• Trap: "Increase retention to stop ProvisionedThroughputExceededException." → Reality: retention adds no throughput.
• Trap: "Add shards to fix one hot partition key." → Reality: that key still lands on one shard; change the key.
• Trap: "Turn on EFO everywhere to save money." → Reality: EFO adds consumer-shard-hour and retrieval charges.
• Trap: "Call GetRecords more often to catch up." → Reality: extra calls hit the 5-per-second limit and throttle.
• Trap: "A high iterator age means producers are throttled." → Reality: it means consumers are falling behind.
• Trap: "EFO gives each consumer 2 MB/s for the whole stream." → Reality: 2 MB/s per shard, per consumer.

Practice questions

1. A company has five independent applications that read the same Kinesis data stream with shared-throughput GetRecords calls. ReadProvisionedThroughputExceeded is frequent and end-to-end latency is about one second. Each application needs lower latency with the LEAST operational overhead. What should the developer do?

   • A. Double the number of shards in the stream.
   • B. Put an SNS topic in front of the stream and subscribe one SQS queue per application.
   • C. Register each application as an enhanced fan-out consumer and read with KCL 2.x or later.
   • D. Make each application call GetRecords twice as often.

2. A Lambda function consumes a 4-shard stream. GetRecords.IteratorAgeMilliseconds has been rising for hours while WriteProvisionedThroughputExceeded stays at zero, and each invocation runs close to its timeout. Which TWO actions should the developer take? (Select TWO.)

   • A. Decrease the stream's retention period.
   • B. Add more retries to the producers.
   • C. Raise the event source mapping's ParallelizationFactor.
   • D. Increase the number of shards in the stream.
   • E. Switch the producers to random partition keys.

13. Amazon Data Firehose

Concept

• Amazon Data Firehose is a fully managed, serverless delivery service: point a Firehose stream at a destination and it buffers incoming records, optionally transforms, converts and compresses them, then writes them in batches with retries and backup. There are no shards, consumers or servers to manage.
• It is near real time (it buffers) and has no replay: nothing can read records back out of Firehose. When several applications need the same records, or a replay, put Kinesis Data Streams in front and make Firehose one of its consumers.

Exam facts

• ⚠️ Changed: the service is called Amazon Data Firehose (since 2024-02-09); endpoints, APIs, IAM actions, metrics and the firehose CLI didn't change (CreateDeliveryStream creates a "Firehose stream"). Older material may say Amazon Kinesis Data Firehose and "delivery stream".
• Sources: Direct PUT (PutRecord, or PutRecordBatch with up to 500 records or 4 MiB per call), a Kinesis data stream, or Amazon MSK. Many services write through Direct PUT: CloudWatch Logs subscriptions, EventBridge, SNS subscriptions, AWS IoT, WAF logs, API Gateway access logs, the Kinesis Agent.
• With a Kinesis data stream as the source, Firehose's own PutRecord/PutRecordBatch are disabled (write to the stream instead); Firehose starts reading at LATEST, calls GetRecords once per second per shard (twice with full source backup), and those reads count against the shard's read limits. It de-aggregates KPL records.
• Record size: 1,000 KiB before base64. Direct PUT throughput per stream: 500,000 records/s, 2,000 requests/s and 5 MiB/s in US East (N. Virginia), US West (Oregon) and Europe (Ireland); 100,000 records/s, 1,000 requests/s and 1 MiB/s elsewhere. Firehose raises it automatically when throttling persists, or you request an increase; a Kinesis stream source isn't bound by it.
• Destinations: Amazon S3, Amazon Redshift (data lands in S3, then Firehose issues a COPY), Amazon OpenSearch Service and OpenSearch Serverless, Apache Iceberg tables, Snowflake, Splunk, generic HTTP endpoints, and partners such as Datadog, Dynatrace, Elastic, MongoDB, New Relic and Sumo Logic. ⚠️ Changed: Firehose also reads from Amazon MSK and delivers to Apache Iceberg Tables, Snowflake and OpenSearch Serverless. Older material may say its only targets are S3, Redshift, OpenSearch, Splunk, a few partners and HTTP endpoints.
• Buffering hints: SizeInMBs 1–128 MiB (default 5; the API reference counts MiB despite the name) and IntervalInSeconds 0–900 (default 300); delivery happens when the first of the two is reached. ⚠️ Changed: the interval can be 0 (zero buffering, since 2023-12-26), and most streams without processing then deliver within about 5 seconds. Older material may say Firehose always buffers for at least 60 seconds.
• Lambda transformation: Firehose invokes the function synchronously with buffered batches (0.2–3 MB, default 1 MB; interval 0–900 s, default 60 s). The function must return every recordId with a result of Ok, Dropped or ProcessingFailed and base64 data. Invocations may run up to 5 minutes, and the 6 MB synchronous payload limit applies. Failed invocations are retried 3 times by default; unsuccessful records land in S3 under the processing-failed prefix.
• Format conversion (S3 destinations): JSON input → Apache Parquet or Apache ORC, using a table schema from the AWS Glue Data Catalog. Other formats (CSV, plain logs) must first be turned into JSON by a Lambda transformation.
• S3 options: compression GZIP, Snappy, ZIP or Hadoop-compatible Snappy; dynamic partitioning by keys taken from the records (inline jq parsing or Lambda; up to 50 keys; 500 active partitions by default; enabled only when the Firehose stream is created and never turned off); custom prefixes and an error output prefix; SSE-KMS for delivered objects; source record backup of all or only failed records.
• Failure handling: when the destination is down, Direct PUT data is kept for up to 24 hours (a Kinesis source keeps it for its own retention). Redshift COPY and OpenSearch requests are retried for a configurable 0–7,200 s (OpenSearch default 300 s, after which records go to an S3 error bucket); HTTP endpoints retry for the configured duration, then back the data up to S3.
• Pricing: per GB ingested, with each record rounded up to the next 5 KB, so many tiny records cost more than fewer, larger ones.
• Amazon Managed Service for Apache Flink runs Apache Flink applications (Java, Scala, Python, SQL; Studio notebooks for interactive queries) on streams from Kinesis Data Streams or MSK, with automatic scaling and checkpoints/snapshots; it can't read from Firehose. ⚠️ Changed: renamed from Kinesis Data Analytics for Apache Flink on 2023-08-30; the API and CLI namespace is still kinesisanalyticsv2. Older material may say Kinesis Data Analytics.
• ⚠️ Changed: Kinesis Data Analytics for SQL applications is discontinued (no new applications since 2025-10-15; applications deleted from 2026-01-27). Older material may say it is the SQL option for streams.
• ⚠️ Changed: on-demand Kinesis data streams can deliver straight to S3 general purpose buckets or to Apache Iceberg streaming tables (S3 Tables) with CreateChannel: no consumer code, no custom transformation (Iceberg delivery converts to Parquet itself), 5–15 minute data freshness, no use of the stream's read throughput, at most 2 deliveries per stream (launch date ⚠️ unverified). Older material may say a stream needs Firehose or custom consumers to land data in S3. Firehose remains the choice for transformation, format conversion, dynamic partitioning and the other destinations.

Developer angle

aws firehose create-delivery-stream --delivery-stream-name clicks-to-s3 \
  --delivery-stream-type DirectPut \
  --extended-s3-destination-configuration file://s3-destination.json
# Data is a base64 blob in AWS CLI v2: {"page":"/cart"} followed by a newline
aws firehose put-record --delivery-stream-name clicks-to-s3 \
  --record '{"Data":"eyJwYWdlIjoiL2NhcnQifQo="}'

s3-destination.json:

{"RoleARN": "arn:aws:iam::111122223333:role/FirehoseDeliveryRole", "BucketARN": "arn:aws:s3:::clickstream-raw",
 "Prefix": "clicks/", "ErrorOutputPrefix": "errors/", "CompressionFormat": "GZIP",
 "BufferingHints": {"SizeInMBs": 64, "IntervalInSeconds": 60}}

A transformation function that drops synthetic traffic and enriches the rest:

import base64
import json

def handler(event, context):
    out = []
    for record in event["records"]:
        payload = json.loads(base64.b64decode(record["data"]))
        if payload.get("synthetic"):
            out.append({"recordId": record["recordId"], "result": "Dropped",
                        "data": record["data"]})
            continue
        payload["source"] = "web"
        data = base64.b64encode((json.dumps(payload) + "\n").encode()).decode()
        out.append({"recordId": record["recordId"], "result": "Ok", "data": data})
    return {"records": out}

• The delivery role trusts firehose.amazonaws.com and needs s3:PutObject (plus bucket listing) on the destination, read access to a Kinesis stream source, lambda:InvokeFunction for a transformation and AWS Glue table read access for format conversion. Producers need firehose:PutRecord / firehose:PutRecordBatch. Add a newline after each JSON record (as above) so that Athena and other tools can split the objects.

Traps

• Trap: "Firehose delivers in milliseconds and can replay." → Reality: near real time, no replay.
• Trap: "Turn on dynamic partitioning for an existing Firehose stream." → Reality: it can be enabled only at creation (and can't be disabled later).
• Trap: "Managed Service for Apache Flink reads a Firehose stream." → Reality: it reads Kinesis Data Streams or MSK.
• Trap: "Call Firehose PutRecord on a stream whose source is a Kinesis data stream." → Reality: that's disabled; write to the Kinesis stream.
• Trap: "The transformation may return only the records it keeps." → Reality: every recordId must come back (Dropped for discarded ones).
• Trap: "Convert CSV straight to Parquet." → Reality: conversion reads JSON; turn CSV into JSON with Lambda first.
• Trap: "Firehose always waits at least 60 seconds." → Reality: IntervalInSeconds can be 0.

Practice questions

1. A web application emits JSON clickstream events. Analysts query them in Amazon Athena and want columnar files in S3 with the LEAST operational overhead. What should the developer do?

   • A. Write the events to a Kinesis data stream and run a KCL application on EC2 that writes Parquet files to S3.
   • B. Send the events to a Firehose stream with record format conversion to Apache Parquet.
   • C. Write JSON files to S3 and convert each file to Parquet with a Lambda function triggered by S3 event notifications.
   • D. Send the events to a Firehose stream and have a Managed Service for Apache Flink application read from it and write Parquet.

2. A developer's Firehose stream with a Lambda transformation delivers almost nothing to S3; most records appear under the processing-failed prefix. The function decodes each record, adds fields and returns a plain list of the modified JSON objects, and CloudWatch Logs show that some invocations with large batches time out after 5 minutes. Which TWO changes are needed? (Select TWO.)

   • A. Return an object for every input record with its original recordId, a result of Ok, Dropped or ProcessingFailed, and base64-encoded data.
   • B. Set the destination buffering interval to 0 seconds.
   • C. Enable dynamic partitioning on the S3 destination.
   • D. Keep each invocation under 5 minutes, for example by lowering the Lambda buffering size hint.
   • E. Return the transformed JSON as plain text in the data field.

14. Choosing between SQS SNS Kinesis and Firehose

Concept

• Four questions settle most scenarios: pull or push? (consumers poll SQS and KDS; SNS pushes), one consumer or many? (a queue serves one logical consumer; SNS and KDS serve many), is order or exactly-once required? (FIFO variants, or KDS per partition key), must data be replayed, or loaded somewhere without code? (KDS replays; Firehose delivers).
• The services combine: SNS + SQS for durable fan-out, KDS + Firehose to analyse in real time and archive to S3, SQS in front of Lambda to absorb bursts at a controlled concurrency.

Exam facts

• Buffer work between components and scale workers on backlog → SQS standard (idempotent consumers).
• Order per entity or no duplicates → SQS FIFO; when several services need the same ordered events → SNS FIFO + SQS FIFO.
• One event, several independent consumers each with its own backlog and retries → SNS + one SQS queue per consumer.
• Notify people or devices (email, SMS, mobile push) → SNS standard topics.
• High-volume data read by several applications in real time, replay, or per-key ordered processing → Kinesis Data Streams (EFO for many readers).
• Load into S3, Redshift, OpenSearch, Splunk, Snowflake or an HTTP endpoint with optional transformation and no code → Amazon Data Firehose.
• SQL or Java analytics on a stream → Managed Service for Apache Flink reading KDS or MSK.
• Delay up to 15 minutes → SQS delay or message timer; longer schedules → EventBridge Scheduler (see notes/monitoring-guide.md). Payloads above 1 MiB → S3 + pointer (SQS or SNS extended clients, up to 2 GB) or KDS records up to 10 MiB.
• Smoothing a burst before Lambda → SQS between producer and function (Lambda polls at a controlled concurrency); SNS → Lambda invokes asynchronously for every message as it arrives.
• Per-message retries and DLQs are SQS features; a stream consumer must handle a bad record itself or it holds up its shard.
• Elsewhere: event routing across many AWS and SaaS sources with rules and a schema registry → EventBridge (see notes/monitoring-guide.md); a Kafka-compatible API → MSK, and multi-step workflows with retries and branching → Step Functions (see notes/other-services-guide.md); change capture on a table → DynamoDB Streams (see notes/dynamodb-guide.md).

Developer angle

One publish, two durable consumers: a worker queue and an S3 archive through Firehose (the queue policy and the Firehose role must allow SNS):

aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs --notification-endpoint "$QUEUE_ARN"
aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol firehose \
  --notification-endpoint arn:aws:firehose:eu-west-1:111122223333:deliverystream/order-archive \
  --attributes SubscriptionRoleArn=arn:aws:iam::111122223333:role/SnsToFirehoseRole

Traps

• Trap: "Several consumers on one SQS queue each get every message." → Reality: consumers compete; use SNS fan-out (one queue each) or KDS.
• Trap: "SNS keeps messages for subscribers that join later." → Reality: standard topics keep nothing; only FIFO topics with an archive policy can replay.
• Trap: "Use Firehose for real-time processing by several applications." → Reality: that's Kinesis Data Streams; Firehose only delivers.
• Trap: "Choose FIFO everywhere to be safe." → Reality: lower throughput and higher cost; use it only when order or deduplication is required.
• Trap: "SNS → Lambda absorbs any burst." → Reality: every message becomes an invocation; buffer with SQS when concurrency must stay controlled.
• Trap: "Use a stream for independent work items that need individual retries and a DLQ." → Reality: that's what SQS offers out of the box.

Practice questions

1. A company's analytics teams each need to read the last three days of click events at different times, and new teams may join and start from the beginning of that window. Events arrive at thousands per second and must be readable within a second. Which solution meets these requirements?

   • A. Amazon Kinesis Data Streams with a retention period of at least 72 hours.
   • B. An SQS standard queue with a 14-day retention period shared by all teams.
   • C. An SNS topic with one SQS queue per team.
   • D. An Amazon Data Firehose stream that delivers the events to S3 every 60 seconds.

2. An order service publishes an event for every order. An inventory service must process every event with retries and must not lose any, an archive of all events must land in S3 without custom code, and a fraud service will be added next quarter without changes to the order service. Which TWO components should the developer use? (Select TWO.)

   • A. A Kinesis data stream read by a KCL application that writes the archive files to S3.
   • B. An SNS topic for the order events with an SQS queue subscribed for the inventory service.
   • C. A Firehose stream that subscribes to the order events topic and delivers to S3.
   • D. An SQS FIFO queue shared by the inventory and fraud services.
   • E. A scheduled job that copies order rows from the database to S3 every hour.

Quick decision rules

SQS (§1–§6)

• If you see a message processed twice while a slow job is still running, think visibility timeout shorter than the processing time: raise it or extend it with ChangeMessageVisibility (heartbeat).
• If you see many empty ReceiveMessage responses or high request cost, think long polling (WaitTimeSeconds or ReceiveMessageWaitTimeSeconds = 20).
• If you see a message that fails forever and wastes consumers, think DLQ with maxReceiveCount (at least 5 for Lambda) and an alarm on the DLQ's ApproximateNumberOfMessagesVisible.
• If you see "move the DLQ messages back after the fix without code", think StartMessageMoveTask with MaxNumberOfMessagesPerSecond (up to 500).
• If you see messages expiring early in a standard DLQ, think original enqueue timestamp: give the DLQ a longer retention than its source.
• If you see "make new messages invisible for N minutes", think DelaySeconds on the queue or a message timer per message (standard queues only), at most 15 minutes.
• If you see payloads larger than 1 MiB, think S3 plus a pointer (SQS or SNS Extended Client Library, up to 2 GB).
• If you see order per customer, account or device plus parallel processing, think FIFO with MessageGroupId set to that entity.
• If you see duplicates caused by producer retries, think FIFO with a stable MessageDeduplicationId (or content-based deduplication when the body is identical).
• If you see FIFO traffic above 3,000 messages per second, think high throughput mode (DeduplicationScope=messageGroup, FifoThroughputLimit=perMessageGroupId) and many group IDs.
• If you see duplicates on a queue that must stay standard, think idempotent consumer (conditional write on a business key).
• If you see S3, SNS or EventBridge failing to write to an encrypted queue, think customer managed KMS key whose key policy allows that service principal.
• If you see cross-account queue access, think queue policy in the owner account plus an IAM policy in the caller's account.
• If you see one tenant's burst delaying all others on a standard queue, think fair queues (MessageGroupId = tenant ID).

SNS (§7–§9)

• If you see one event that several services must process independently, think SNS topic with one SQS queue per service.
• If you see a subscribed queue that receives nothing, think queue policy for sns.amazonaws.com with aws:SourceArn, and the KMS key policy if the queue is encrypted.
• If you see subscribers that each need only some events, think filter policies (FilterPolicyScope=MessageBody when the fields are in the payload).
• If you see an HTTP/S subscriber that can be down for more than an hour, think subscription DLQ (HTTP/S retries stop at 3,600 s).
• If you see consumers stripping the SNS JSON envelope, think RawMessageDelivery=true.
• If you see ordered, deduplicated fan-out, think SNS FIFO topic → SQS FIFO queues.
• If you see "redeliver last week's events to one subscriber", think FIFO topic ArchivePolicy plus a subscription ReplayPolicy.

Kinesis and Firehose (§10–§13)

• If you see real-time data read by several applications, or replay, think Kinesis Data Streams.
• If you see write throttling concentrated on one shard, think partition key distribution first, then SplitShard.
• If you see read throttling with several consumer applications, think enhanced fan-out (2 MB/s per shard per consumer).
• If you see IteratorAgeMilliseconds rising, think consumers falling behind: more shards, workers or ParallelizationFactor.
• If you see KCL instances sitting idle, think more workers than shards.
• If you see unpredictable stream traffic and shard-management toil, think on-demand capacity mode.
• If you see PutRecords returning 200 with FailedRecordCount > 0, think resend only the failed entries with backoff.
• If you see "load into S3, Redshift, OpenSearch or Splunk with no code", think Amazon Data Firehose.
• If you see "JSON to Parquet or ORC on the way to S3", think Firehose record format conversion with a Glue Data Catalog schema.
• If you see SQL or Java analytics on a stream, think Managed Service for Apache Flink reading KDS or MSK (never Firehose).

Changed vs older material

Learn the Current column; the course or older practice material may still show the left-hand value. Rows already in notes/current-facts.md are marked there; the others await confirmation in this module's /ingest review.