SQS with Auto Scaling Group

Overview

EC2 instances in an Auto Scaling group (ASG) poll messages from an SQS queue. The ASG scales automatically based on the queue size. See ASG and 02-SQS-Standard-Queues.

Scaling on Queue Length

1. The CloudWatch metric ApproximateNumberOfMessages (queue length) shows how many messages are left in the queue.
2. Set a CloudWatch alarm, e.g. over 1,000 messages means processing is lagging.
3. The alarm triggers a scaling action in the ASG, adding EC2 instances.
4. Messages are processed faster and the queue size decreases.

It works in both directions: scale up or scale down.

Pattern 1: SQS as a Buffer for Database Writes

Scenario: your biggest marketing campaign. Customers place orders stored in Amazon RDS or Aurora (OLTP) or DynamoDB (NoSQL). Writing very fast and suddenly into the database can cause errors if the database is overloaded, and you lose customer transactions.

Solution: use SQS as a buffer.

1. The front-end application writes each transaction request as a message into the SQS queue (enqueue) instead of writing to the database. SQS is, per AWS, infinitely scalable with no throughput issue.
2. Messages are durably stored, so none are lost.
3. A second ASG dequeues the messages and inserts them into the databases.
4. Only after a message is known to be inserted into the database is it deleted from the queue.

This guarantees every transaction is written. It works only if the client does not need confirmation that the write reached the database. Once the message is in SQS, it will eventually be written.

Pattern 2: Decoupling Application Tiers

Instead of one application receiving a request, processing it and returning a response, requests arrive at a front-end web application, which sends them to an SQS queue. The back-end processing job receives and handles the messages in its own time. Each tier scales accordingly.

Questions & Answers