SQS - Standard Queues

Overview

At the core of Amazon SQS is a queue inside a simple queuing service. The queue holds messages. A producer sends messages into the queue (one or many producers). A message can be anything, e.g. "process this order" or "process this video". Consumers poll the queue ("do you have any message for me?"), process the message, then delete it from the queue. You can have multiple consumers.

A queue is a buffer that decouples producers from consumers.

Standard queues are the oldest SQS offering. SQS is one of the first AWS services, over 10 years old, so how it works is well settled. It is a fully managed service used to decouple applications.

Standard Queue Attributes

Messages are short-lived. After you send a message, it must be read by a consumer and deleted after processing within the retention period, otherwise it is lost. A FIFO queue handles the ordering limitation (see 05-SQS-FIFO-Queues).

Producing Messages

• Producers send messages with an SDK, using the SendMessage API.
• The message is persisted in the queue until a consumer reads and deletes it, which signals it was processed.
• Example: to process an order in your own time, send a message with the order ID, customer ID and any attributes such as the address. The consumer deals with the message itself.

Consuming Messages

• Consumers are applications you write. They can run on EC2 instances, on-premises servers, or AWS Lambda (covered later in the course).
• A consumer polls SQS and can receive up to 10 messages at a time.
• Your code processes them, e.g. inserts the orders into an RDS database.
• The consumer then deletes each message using the DeleteMessage API. This ensures no other consumer sees it, so processing is complete.

Multiple consumers

• Multiple consumers receive and process messages in parallel. Each gets a different set of messages per poll.
• If a message is not processed fast enough by one consumer, other consumers receive it. This is why there is at least once delivery and best-effort ordering.
• Consumers must delete messages when done, otherwise other consumers see them.
• To raise processing throughput, add consumers (horizontal scaling).

SQS with Auto Scaling Group

Consumers run on EC2 instances in an Auto Scaling group (ASG) and poll the queue. The ASG scales on the queue length metric, the ApproximateNumberOfMessages CloudWatch metric available on every SQS queue. A CloudWatch alarm fires when the queue length goes over a level and increases ASG capacity by X. The more messages (e.g. an order surge), the more EC2 instances. See 06-SQS-with-Auto-Scaling-Group and ASG.

Decoupling Application Tiers

Example: video processing.

• One big front-end application takes the request, processes the video and writes it to S3. Processing may be very long and slow the website.
• Decoupled: the front-end sends a message into an SQS queue for each processing request. A back-end processing application in its own ASG receives the messages, processes the videos and inserts them into an S3 bucket.
• Front-end and back-end scale independently.
• The queue has unlimited throughput and unlimited messages, so the architecture is robust and scalable.
• Each tier can use the optimal EC2 instance type, e.g. instances with a GPU for the video back-end.

SQS Security

Hands-On Walkthrough

Create the queue

1. Open the SQS console and Create queue.
2. Two queue types: Standard and FIFO. Choose standard and name it Demo Queue (FIFO later).
3. Configuration items (covered in later lectures): visibility timeout, delivery delay, wait time, retention period. Retention period stays at 4 days. Max message size is 1 MiB, the maximum allowed in SQS.
4. Encryption: can be disabled, but is enabled by default with an Amazon SQS key (SSE-SQS), similar to SSE-S3 in S3. The other option is KMS, where you choose the key (the default AWS key alias is alias/AWS/SQS, as heard in the recording) and a data key reuse period (e.g. 5 minutes) to limit the number of KMS API calls. Keep SSE-SQS.
5. Access policy: answer prompts for who can send (only the queue owner, or a list of accounts, users and roles) and who can receive. A JSON document is generated, similar to an S3 bucket policy. It is a resource policy for SQS.
6. Redrive and dead-letter queue are covered later. Click Create queue.

Send and receive

1. Top right, click Send and receive messages. Send is at the top, receive at the bottom.
2. The queue shows 0 messages available. Enter hello world! in the message body and send. Messages available becomes 1.
3. Click Poll for messages. The message appears with a message ID. Open the message details to see metadata: hash of the message, sender, receive count (1 the first time), size in bytes, and the body. Message attributes (key/value) can be set and read, but are out of scope for the exam.
4. The message was received a second time because it was not processed in time. After 30 seconds it went back into the queue. Polling again shows receive count 3.
5. Select the message and Delete. This tells SQS it was successfully processed. Messages available returns to 0 and polling does not return it again.
6. Send hello world, hello world 2, hello world 3. After refresh, 3 messages are available. One poll returns all three. Delete them to signal they were processed.

Other queue options

• Edit the queue configuration.
• Purge: deletes all messages in the queue. Type purge to confirm. Helpful in development, not in production.
• Monitoring: shows how many messages are in the queue and the approximate age of the oldest message, which can be another way to scale an ASG reading from the queue.
• Access policy, encryption (SSE-SQS in the demo, editable), and dead-letter queue redrive status (relevant if a DLQ is set up).

Questions & Answers