Amazon SQS is a highly available distributed queue system
A queue is a temporary repository for messages awaiting for processing and acts as a buffer between the component producer and the consumer
Amazon SQS
offers a reliable, highly-scalable, hosted queue for storing messages in transit between computers.
Provides fault tolerant, loosely coupled, flexibility of distributed components of applications to send & receive without requiring each component to be concurrently available.
Helps build distributed application with decoupled components.
Requires no administrative overhead and little configuration.
Supports the HTTP over SSL (HTTPS) and Transport Layer Security (TLS) protocols for security.
SQS provides two types of Queues – Standard & FIFO
SQS queue retains messages for four days, by default.
Queues can configured to retain messages for 1 minute to 14 days after the message has been sent.
Messages are stored until they are processed or deleted.
SQS can delete a queue without notification if actions hasn’t been performed on it for 30 consecutive days.
SQS allows the deletion of the queue with messages in it
Stores all message queues and messages within a single, highly-available AWS Region with multiple redundant Availability Zones.
Queues can be shared anonymously or with specific AWS accounts
Queue sharing can also be restricted by IP address and time of day
SSE encrypts messages as soon as Amazon SQS receives them, then protects them using keys managed in AWS KMS
Queue URLs
Queue is identified by a unique queue name within the same AWS account
SQS assigns each queue with a Queue URL identifier for e.g. http://sqs.us-east-1.amazonaws.com/123456789012/queue2
Queue URL is needed to perform any operation on the Queue
Message ID
Message IDs are useful for identifying messages,
Each message receives a system-assigned message ID that SQS returns to with the SendMessage response.
To delete a message, the message’s receipt handle instead of the message ID is needed.
Message ID can be of is 100 characters max
Receipt Handle
When a message is received from a queue, a receipt handle is returned with the message which is associated with the act of receiving the message rather then the message itself.
Receipt handle is required, not the message id, to delete a message or to change the message visibility.
If a message is received more than once, each time its received, a different receipt handle is assigned and the latest should be used always.
We have 2 types Standard Queue & FIFO Queue
Standard Queue:
Provide best-effort ordering which ensures that messages are generally delivered in the same order as they are sent.
More than one copy of a message might be delivered out of order
Nearly-unlimited number of transactions per second (Processes billions of messages per day )
Available in all the region
Supported by all AWS services
FIFO Queues
First-in-first-out delivery and exactly-once processing .
Limited to 300 transactions per second per API action
Available in limited regions only
Currently not supported by all AWS services like: CloudWatch Events, S3 Event Notifications, SNS Topic Subscriptions, Auto Scaling Lifecycle Hooks, AWS IoT Rule Actions, AWS Lambda Dead Letter Queues
SQS is a distributed queuing system. Messages are NOT pushed to receivers. Receivers have to poll or pull messages from SQS.
SNS is a distributed publish-subscribe system. Messages are pushed to subscribers as and when publishers send them to SNS.
If you wants to change the queue type (FIFO or Standard) then you need to recreate it.
To manage large Amazon SQS messages, you can use S3 and the Amazon SQS Extended Client Library for Java.
This is especially useful for storing and consuming messages up to 2 GB.
Unless your application requires repeatedly creating queues and leaving them inactive or storing large amounts of data in your queues, consider using Amazon S3 for storing your data.
Delay Queue
Delay queues let you postpone the delivery of new messages to consumers for a number of seconds, for example, when your consumer application needs additional time to process messages.
If you create a delay queue, any messages that you send to the queue remain invisible to consumers for the duration of the delay period.
The default (minimum) delay for a queue is 0 seconds. The maximum is 15 minutes.
Message timers
let you specify an initial invisibility period for a message added to a queue.
For example, if you send a message with a 45-second timer, the message isn't visible to consumers for its first 45 seconds in the queue. The default (minimum) delay for a message is 0 seconds. The maximum is 15 minutes.
Redundant infrastructure
offers reliable and scalable hosted queues for storing messages
is engineered to always be available and deliver messages
provides ability to store messages in a fail safe queue
highly concurrent access to messages
At-Least-Once delivery
ensures delivery of each message at least once
stores copies of the messages on multiple servers for redundancy and high availability
might deliver duplicate copy of messages, if the servers storing a copy of a message is unavailable when you receive or delete the message and the copy of the message is not deleted on that unavailable server.
Applications should be designed to be idempotent with the ability to handle duplicate messages and not be adversely affected if it processes the same message more than once
Message Attributes
SQS message can contain up to 10 metadata attributes.
take the form of name-type-value triples
can be used to separate the body of a message from the metadata that describes it.
Helps process and store information with greater speed and efficiency because the applications don’t have to inspect an entire message before understanding how to process it.
Message Sample
behavior of retrieving messages from the queue depends on whether short (standard) polling, the default behavior, or long polling is used
With short polling,
SQS samples only a subset of the servers (based on a weighted random distribution) and returns messages from just those servers.
A receive request might not return all the messages in the queue. But a subsequent receive request would return the message
With Long polling,
request persists for the time specified and returns as soon as the message is available thereby reducing costs and time the message has to dwell in the queue.
Long polling doesn’t return a response until a message arrives in the message queue or the long poll times out.
Long polling makes it inexpensive to retrieve messages from your Amazon SQS queue as soon as the messages are available.
Batching
SQS allows send, receive and delete batching which helps club up to 10 messages in a single batch while charging price for a single message.
helps lower cost and also increases the throughput.
Configurable settings per queue
All queues don’t have to be alike
Order
makes a best effort to preserve order in messages does not guarantee first in, first out delivery of messages (UPDATE – SQS now offers FIFO queues which maintain order and Exactly-Once Processing)
can be handled by placing sequencing information within the message and performing the ordering on the client side.
Loose coupling
removes tight coupling between components
Multiple writers and readers
supports multiple readers and writers interacting with the same queue as the same time
locks the message during processing, using Visibility Timeout, preventing it to be processed by any other consumer
Variable message size
supports message in any format up to 256KB of text.
messages larger than 256 KB can be managed using the S3 or DynamoDB, with SQS holding a pointer to the S3 object.
Access Control
Access can be controlled for who can produce and consume messages to each queue
Delay Queues
delay queue allows the user to set a default delay on a queue such that delivery of all messages enqueued is postponed for that time duration.
Dead Letter Queues
Dead letter queue is a queue for messages that were not able to be processed after a maximum number of attempts
A DLQ is just like any other Amazon SQS queue.
Messages can be sent to and received from it like any other SQS queue.
You can create a DLQ from the Amazon SQS API and the SQS console.
PCI Compliance
supports the processing, storage, and transmission of credit card data by a merchant or service provider, and has been validated as being PCI-DSS (Payment Card Industry – Data Security Standard) compliant
Work Queues
Decouple components of a distributed application that may not all process the same amount of work simultaneously.
Buffer and Batch Operations
Add scalability and reliability to the architecture and smooth out temporary volume spikes without losing messages or increasing latency.
Request Offloading
Move slow operations off of interactive request paths by enqueueing the request.
Fan-out
Combine SQS with SNS to send identical copies of a message to multiple queues in parallel for simultaneous processing.
Auto Scaling
SQS queues can be used to determine the load on an application, and combined with Auto Scaling, the EC2 instances can be scaled in or out, depending on the volume of traffic.
Appropriate for:
Service-to-service communication
Asynchronous work items
State change notifications
Inappropriated for:
Selecting specific messages (This can lead to a scenario where there are messages in the queue that no one is polling for and are never consumed)
Large messages (As message sizes grow, it’s best to use a dedicated storage system, such as Amazon S3, and pass a reference to an object in that store in the message itself.)
Behaviour
SQS does not delete the message once it is received by a consumer,
because the system is distributed, there’s no guarantee that the consumer will actually receive the message.
Consumer should explicitly delete the message from the Queue once it is received and successfully processed.
As the message is still available on the Queue, other consumers would be able to receive and process and this needs to be prevented.
SQS handles the above behavior using Visibility timeout.
SQS blocks the visibility of the message for the Visibility timeout period, which is the time during which SQS prevents other consuming components from receiving and processing that message.
Consumer should delete the message within the Visibility timeout. If the consumer fails to delete the message before the visibility timeout expires, the message is visible again for other consumers.
Visibility timeout considerations
clock starts ticking once SQS returns the message
should be large enough to take into account the processing time for each of the message
default Visibility timeout for each Queue is 30 seconds and can be changed at the Queue level
When receiving messages, a special visibility timeout for the returned messages can be set without changing the overall queue timeout using the receipt handle.
Can be extended by the consumer, if the consumer thinks it won’t be able to process the message within the current visibility timeout period. SQS restarts the timeout period using the new value.
A message’s Visibility timeout extension applies only to that particular receipt of the message and does not affect the timeout for the queue or later receipts of the message.
SQS has an 120,000 limit for the number of inflight messages per queue i.e. message received but not yet deleted and any further messages would receive an error after reaching the limit.
Component 1 sends Message A to a queue, and the message is redundantly distributed across the SQS servers.
When Component 2 is ready to process a message, it retrieves messages from the queue, and Message A is returned. While Message A is being processed, it remains in the queue but is not returned to subsequent receive requests for the duration of the visibility timeout.
Component 2 deletes Message A from the queue to avoid the message being received and processed again once the visibility timeout expires.
Priority Queue Pattern
Use SQS to prepare multiple queues for the individual priority levels.
Place those processes to be executed immediately (job requests) in the high priority queue.
Prepare numbers of batch servers, for processing the job requests of the queues, depending on the priority levels.
Queues have a message “Delayed Send” function, which can be used to delay the time for starting a process.
Job Observer Pattern
Enqueue job requests as SQS messages.
Have the batch server dequeue and process messages from SQS.
Set up Auto Scaling to automatically increase or decrease the number of batch servers, using the number of SQS messages, with CloudWatch, as the trigger to do so