Overview
You want one message delivered to multiple SQS queues. Sending it individually to every queue causes problems: the application can crash in between, there can be delivery failures, and you must change code when you add more queues later. The fan-out pattern fixes this: push once to an SNS topic and subscribe as many SQS queues as you want to that topic. See 07-Amazon-SNS and 02-SQS-Standard-Queues.
Example: a buying service sends one message to an SNS topic. The fraud service and the shipping service each read all messages from their own SQS queue.
SNS and SQS fan-out
Benefits
- Fully decoupled, with no data loss.
- SQS gives data persistence, delayed processing and retries of work.
- You can add more SQS queues as subscribers over time.
- The SQS queue access policy must allow the SNS topic to write to the queue.
- Cross-region delivery works: an SNS topic in one region can send to SQS queues in other regions if security allows it.
S3 Events to Multiple Queues
S3 event rules have a limitation: for a combination of event type (for example object created) and prefix (for example images/) you can have only one S3 event rule. To send the same S3 event notification to multiple SQS queues, use fan-out: S3 object created event, to an SNS topic, to many SQS queues. You can also subscribe other targets such as email or Lambda functions. See 02-S3-Event-Notifications.
SNS to S3 through Kinesis Data Firehose
SNS has a direct integration with Kinesis Data Firehose (KDF). The buying service sends data to an SNS topic, Firehose receives it, and Firehose writes to Amazon S3 or any other KDF-supported destination. See 10-Amazon-Data-Firehose.
SNS FIFO Fan-Out
| Feature | Detail |
|---|---|
| Ordering | Ordering by message group ID (same as SQS FIFO) |
| Deduplication | Deduplication ID or content-based deduplication |
| Subscribers | SQS FIFO queue (the course says "for now"). The course also says both SQS standard and FIFO queues can be subscribers. |
| Throughput | Same as the SQS FIFO queue (limited) |
Use it when you need fan-out + ordering + deduplication: the buying service sends to an SNS FIFO topic, which fans out to two SQS FIFO queues read by the fraud and shipping services. See 05-SQS-FIFO-Queues.
Message Filtering
A JSON policy applied to an SNS topic subscription that filters which messages are delivered.
- A subscription without a filter policy receives every message (default).
- Example: a buying service publishes transactions with an order number, product (a pencil), quantity and state (placed). A queue subscribed with a filter policy State = Placed receives only placed orders. Another queue with a canceled filter gets only canceled orders. The canceled filter can also feed an email subscription. A declined-orders queue is another option, and a queue with no filter gets everything.
Combining fan-out, message filtering, FIFO queues and FIFO topics gives many possibilities, and the instructor says the exam will test you on all of them.
Questions & Answers
Q1: How do you deliver one message to many SQS queues reliably?
Answer:
Publish once to an SNS topic and subscribe the queues (fan-out). Make sure each queue's access policy allows the topic to write.
Q2: How do you send one S3 event type and prefix to multiple queues?
Answer:
S3 allows only one event rule per event type and prefix, so send the event to an SNS topic and fan out to the queues.
Q3: How do you need ordering and deduplication with fan-out?
Answer:
Use an SNS FIFO topic with SQS FIFO queues.
Q4: How do you send only some messages to a subscriber?
Answer:
Set a subscription filter policy (JSON) on that subscription.
Q5: How do you persist SNS messages in S3?
Answer:
Subscribe Kinesis Data Firehose to the topic and let it write to S3.
Check yourself
Quiz
Flashcards
Click the card (or focus it), then use ← → to move and Space to flip.