AWS SQS Case Study

Task Description📄

✍🏻 Research on usecases of AWS SQS and create a blog, Article for the same.

What is Amazon SQS ??

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables us to decouple and scale microservices, distributed systems, and server less applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, we can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.

SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

Standard Queues

Unlimited Throughput: Standard queues support a nearly unlimited number of transactions per second (TPS) per API action.

At-Least-Once Delivery: A message is delivered at least once, but occasionally more than one copy of a message is delivered.

Best-Effort Ordering: Occasionally, messages might be delivered in an order different from which they were sent.

FIFO Queues

High Throughput: By default, FIFO queues support up to 300 messages per second (300 send, receive, or delete operations per second). When we batch 10 messages per operation (maximum), FIFO queues can support up to 3,000 messages per second. If we require higher throughput, we can enable high throughput mode for FIFO (offered as a Preview in select regions) on the Amazon SQS console, which will support up to 30,000 messages per second with batching, or up to 3,000 messages per second without batching.

Exactly-Once Processing: A message is delivered once and remains available until a consumer processes and deletes it. Duplicates aren’t introduced into the queue.

First-In-First-Out Delivery: The order in which messages are sent and received is strictly preserved (i.e. First-In-First-Out).

Functionality

• Unlimited queues and messages: Create unlimited Amazon SQS queues with an unlimited number of message in any region
• Payload Size: Message payloads can contain up to 256KB of text in any format. Each 64KB ‘chunk’ of payload is billed as 1 request. For example, a single API call with a 256KB payload will be billed as four requests. To send messages larger than 256KB, you can use the Amazon SQS Extended Client Library for Java, which uses Amazon S3 to store the message payload. A reference to the message payload is sent using SQS.
• Batches: Send, receive, or delete messages in batches of up to 10 messages or 256KB. Batches cost the same amount as single messages, meaning SQS can be even more cost effective for customers that use batching.
• Long polling: Reduce extraneous polling to minimize cost while receiving new messages as quickly as possible. When our queue is empty, long-poll requests wait up to 20 seconds for the next message to arrive. Long poll requests cost the same amount as regular requests.
• Retain messages in queues for up to 14 days.
• Send and read messages simultaneously.
• Message locking: When a message is received, it becomes “locked” while being processed. This keeps other computers from processing the message simultaneously. If the message processing fails, the lock will expire and the message will be available again.
• Queue sharing: Securely share Amazon SQS queues anonymously or with specific AWS accounts. Queue sharing can also be restricted by IP address and time-of-day.
• Server-side encryption (SSE): Protect the contents of messages in Amazon SQS queues using keys managed in the AWS Key Management Service (AWS KMS). SSE encrypts messages as soon as Amazon SQS receives them. The messages are stored in encrypted form and Amazon SQS decrypts messages only when they are sent to an authorized consumer.
• Dead Letter Queues (DLQ): Handle messages that have not been successfully processed by a consumer with Dead Letter Queues. When the maximum receive count is exceeded for a message it will be moved to the DLQ associated with the original queue. Set up separate consumer processes for DLQs which can help analyze and understand why messages are getting stuck. DLQs must be of the same type as the source queue (standard or FIFO).

Benefits

Eliminate administrative overhead

AWS manages all ongoing operations and underlying infrastructure needed to provide a highly available and scalable message queuing service. With SQS, there is no upfront cost, no need to acquire, install, and configure messaging software, and no time-consuming build-out and maintenance of supporting infrastructure. SQS queues are dynamically created and scale automatically so we can build and grow applications quickly and efficiently.

Reliably deliver messages

Use Amazon SQS to transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be available. SQS lets us decouple application components so that they run and fail independently, increasing the overall fault tolerance of the system. Multiple copies of every message are stored redundantly across multiple availability zones so that they are available whenever needed.

Keep sensitive data secure

We can use Amazon SQS to exchange sensitive data between applications using server-side encryption (SSE) to encrypt each message body. Amazon SQS SSE integration with AWS Key Management Service (KMS) allows us to centrally manage the keys that protect SQS messages along with keys that protect our other AWS resources. AWS KMS logs every use of our encryption keys to AWS CloudTrail to help meet our regulatory and compliance needs.

Scale elastically and cost-effectively

Amazon SQS leverages the AWS cloud to dynamically scale based on demand. SQS scales elastically with our application so we don’t have to worry about capacity planning and pre-provisioning. There is no limit to the number of messages per queue, and standard queues provide nearly unlimited throughput. Costs are based on usage which provides significant cost saving versus the “always-on” model of self-managed messaging middleware.

Using Amazon SQS with other AWS infrastructure web services

Amazon SQS message queuing can be used with other AWS Services such as Redshift, DynamoDB, RDS, EC2, ECS, Lambda, and S3, to make distributed applications more scalable and reliable. Below are some common design patterns:

• Work Queues: Decouple components of a distributed application that may not all process the same amount of work simultaneously.
• Buffer and Batch Operations: Add scalability and reliability to our architecture, and smooth out temporary volume spikes without losing messages or increasing latency.
• Request Offloading: Move slow operations off of interactive request paths by enqueing the request.
• Fanout: Combine SQS with Simple Notification Service (SNS) to send identical copies of a message to multiple queues in parallel.
• Priority: Use separate queues to provide prioritization of work.
• Scalability: Because message queues decouple our processes, it’s easy to scale up the send or receive rate of messages — simply add another process.
• Resiliency: When part of your system fails, it doesn’t need to take the entire system down. Message queues decouple components of our system, so if a process that is reading messages from the queue fails, messages can still be added to the queue to be processed when the system recovers.

AWS Services Used: Amazon SQS, Amazon S3, Amazon DynamoDB, Amazon RDS, & AWS Elastic Beanstalk.

The BMW Group is using AWS for its connected car application that collects sensor data from BMW 7 Series cars to give drivers dynamically updated map information. BMW built its new car-as-a-sensor (CARASSO) service in only six months leveraging Amazon SQS, Amazon S3, Amazon DynamoDB, Amazon RDS, and AWS Elastic Beanstalk.

The BMW Group is using AWS for its new connected-car application that collects sensor data from BMW 7 Series cars to give drivers dynamically updated map information. BMW Group is one of the leading manufacturers of premium cars and mobility services in the world, with brands such as Rolls Royce, BMW, and Mini. BMW built its new car-as-a-sensor (CARASSO) service in only six months leveraging Amazon Simple Storage Service (Amazon S3), Amazon Simple Queue Service (Amazon SQS), Amazon DynamoDB, Amazon Relational Database Service (Amazon RDS), and AWS Elastic Beanstalk. By running on AWS, CARASSO can adapt to rapidly changing load requirements that can scale up and down by two orders of magnitude within 24 hours. By 2018 CARASSO is expected to process data collected by a fleet of 100,000 vehicles traveling more than eight billion kilometers.

All these services enable BMW to focus more on making quality software, leaving behind the problem of scaling the resources, security and deployment.

Thank You for reading !!!!!!