Simple Queue Service (SQS)

SNS is a fully managed, secure, durable, and available hosted queue.

Since it’s hosted in the AWS Public Zone it’s available from outside AWS but services in a VPC need to either have internet access or be able to reach a VPC Endpoint for SQS.

It’s used for application decoupling.

SQS conceptually has a One-to-one architecture: at the end of the queue one type of workload is picking messages. If you need One-to-many couple it with SNS.

It supports ABAC access control.

Retention: they have a default value of 4 days, which can be extended to up to 14 days.

Messages must be explicitly deleted from the queue.

Latenct: for both publish and receive latency is < 10ms.

Message size: 256 KiB is the maximum allowed size for messages.

Encryption at rest: SSE-SQS or SSE-KMS.

Apart from IAM policies, Queue policies are resource-based policies that regulate access to the queue (be sure to include publishers and subscribers) and can also grant cross-account access.

Visibility Timeout

The amount of time (default 30s; min 0, max 12h) that a message is invisible to other consumers after a first consumer has polled it.

It can be set per queue or for single messages. You can change the visibility timeout for a message by using the ChangeMessageVisibility API.

Tipically a consumer polls for messages, receives one and starts processing it. Assuming the default visibility timeout is maintained, if the consumer does not delete the message from the queue after successfully processing it within the timeout, another consumer might process that same message. Hence the at-least-once QoS for standard queues.

A strategy that can be put in place is that the consumer, once it realizes that it’s taking too long to process the message, can alter the visibility timeout to a longer value (within the allowed duration of 12h) and buy time.

Delay Queues

Delay queues set a default visibility delay to messages: the message enters the queue but it’s not visible to any subscribers for a certain amount of time.

It defaults to 0, messages are usually immediately visible to subscribers, delay must be enabled manually. It can range from 0 to 15 minutes.

You can either configure the queue to have a non-zero delay or set a delay per message when sending it using the DelaySeconds attribute.

*This feature is NOT SUPPORTED on FIFO queues.

Long Polling

Long polling is a polling mode that allows to save API calls and be sure that an empty response is not a false positive.

This can be set in both the queue configuration and per request.

By default (short polling), when polling for new messages (ReceiveMessage request), only a subset of servers is queried (based on a weighted random distribution) and a message is returned. Messages could still be present in servers that were not queried, so to be (kinda) sure no message is present in a queue you need to query more than once. Also, if no message is sent to the queue by a producer for a long time, you’re still paying for many API calls, or you have to handle rarely populated queues within your app.

With Long Polling you send a single ReceiveMessage request and specify the WaitTimeSeconds parameter. When this happens:

  • All the servers are queried

  • AWS will not return a respose unless:

    • WaitTimeSeconds have passed without messages being available ⇒ True no messages response.

    • A message has been sent to the queue ⇒ you get that message.

The resulting number of API calls decreases drastically.

The maximum allowed value for WaitTimeSeconds is 20 (20s).

Large Messages

Use the Extended Client Library for Java and Python. It uses S3 to store data.

You can also pass the storage url in the payload.

Standard Queues

They offer unlimited throughput and an unlimited number of messages in a queue.

QoS: SQS Standard offers at-least-once delivery.

Ordering: it provides best effort ordering.

Messages are sent using the SendMessage API call.

FIFO Queues

FIFO queues guarantee the ordering of messages and an exactly-once QoS with deduplication features. Their name MUST contain the .fifo suffix.

QoS: SQS Standard offers exactly-once delivery.

Ordering: it provides strict ordering.

They do NOT support visibility delay.

Two fields are required in a Message:

  • A Deduplication ID: It allows the application to tell SQS what messages (even with different payload) should be treated as unique.

  • A Group ID to partition the flow of messages. Ordering happens for each Group ID, this means that messages with different Group IDs might be sent out of order.
    You can’t request to receive messages with a specific message group ID.

They have a limited throughput. For each Group ID ou can achieve:

  • Up to 300 msg/s without batching

  • Up to 3000 msg/s with batching

Consumers

Consumers poll for messages and, after processing, they have to delete them for the queue.

Example consumer types:

  • Applications running on EC2

  • Servers

  • Lambda Functions

A typical example involves workloads running on EC2 instances managed by an ASG. You can set a target tracking policy in the ASG for the queue backlog length (ApproximateNumberOfMessages in CloudWatch, after setting an alarm for it) and have consumer instances scale accordingly.

Further on this, you can *benefit of the inifinite scalability of SQS Standard queues if you have a storage backend or a database that doesn’t scale inifinitely by creating:

  1. A first ASG that enqueues incomin requests.

  2. An SQS Standard queue that can potentially scale indefinitely.

  3. A second ASG that dequeues messages reading them from the Standard Queue and writing contents to the database (delete messages after that), and scales according to the queue size.

  4. A database tier for storing data.

Dead-Letter Queues

An optional queue where messages can be sent if their processing fails repeatedly. It helps to handle recurring errors in a customized way.

Specialized workloads can subscribe to that queue.

Each message has a ReceiveCount that increments each time it’s delivered.

When active this option lets you specify a maxReceiveCount. Messages whose ReceiveCount reach above that threashold will be routed to the dead-letter queue.

This prevents pointless recurring execution. And you can set an alarm for each message that enters this queue.

A crucial point is that usually, upon entering a queue (from another queue or on creation) messages receive a queue timestamp. Based on that timestamp the message expires. When a message is moved into the dead-letter queue the timestamp is NOT updated, meaning that it will expire according to the creation date in the original queue. So the retention period of dead-letter queues should always be longer than that of the origin queue.

Security

Encryption:

  • In-flight using HTTPS for API calls

  • At-rest using SSE or KMS

  • Client-side

Access control:

  • IAM permissions (identity-based)

  • SQS Access Policies (resource-based)

Billing

You are billed per request: a request can return from 0 to 10 messages (see long polling).

Use Cases

  • You have a frontend that uploads video files in a bucket and then publishes the object URL to a SNS topic. 3 SQS queues are set as subscribers to that topic in a Fanout architecture, specifically each SQS queue will be dedicated to a specific transcoding resolution (say 480p, 720p, 1080p). Message are safely stored in SQS until a pool of workers, one per resolution, picks the message, download the video from S3 and transcodes it to its set resolution. It then uploads the video to a bucket dedicated to transcoded videos, maybe one bucket for each transcoding resolution.