[ad_1]
|
Time positive does fly! I wrote concerning the manufacturing launch of Amazon Easy Queue Service (SQS) again in 2006, with no inkling that I might nonetheless be running a blog fifteen years later, or that this service would nonetheless be rising quickly whereas remaining elementary to the structure of so many several types of web-scale purposes.
The primary beta of SQS was introduced with little fanfare in late 2004. Regardless that now we have added many options since that beta, the unique description (“a dependable, extremely scalable hosted queue for buffering messages between distributed utility parts”) nonetheless applies. Our clients consider SQS as an infinite buffer within the cloud and use SQS queues to implement the connections between the useful components of their utility structure.
Through the years, now we have labored arduous to maintain the SQS interface easy and straightforward to make use of, despite the fact that there’s quite a lot of complexity inside. With a view to make SQS scalable, sturdy, and dependable, messages are saved in a fleet that consists of hundreds of servers in every AWS Area. Inside a area, we save three copies of every message, taking care to distribute the messages throughout storage nodes and Availability Zones. Along with this built-in redundant storage, SQS is self-healing and resilient to host failures & community interruptions. Regardless that SQS is now 15 years previous, we proceed to enhance our scaling and cargo administration purposes with the intention to at all times depend on SQS to deal with your mission-critical workloads.
SQS runs at an unbelievable scale. For instance:
Amazon Product Achievement – Throughout Prime Day 2021, site visitors to SQS set a brand new document, peaking at 47.7 million messages per second.
Rapid7 – Amazon buyer Rapid7 makes in depth use of SQS. In accordance with Jeff Myers (Platform Software program Architect):
Amazon SQS gives us with a easy to make use of, extremely performant, and extremely scalable messaging service with none administration complications. It’s a essential element of our structure that has allowed us to scale to deal with 10s of billions of messages per day.
You may go to the Amazon SQS dwelling web page to examine different high-scale use instances from NASA, BMW Group, Capital One, and different AWS clients.
Serverless Workplace Hours
You should definitely be part of us as we speak (July 13th) for Serverless Workplace Hours at Midday PT on Twitch.television. Rumor has it that there will probably be cake!
Again in Time
Right here’s a fast recap of among the main SQS milestones:
2006 – Manufacturing launch. An infinite variety of queues per account and gadgets per queue, with every merchandise as much as eight KB in size. Pay as you go pricing based mostly on messages despatched and knowledge transferred.
2007 – Extra capabilities together with management of the visibility timeout and entry to the approximate variety of queued messages.
2009 – SQS in Europe, and extra management over queue entry through AddPermission and RemovePermission. This launch additionally marked the debut of what we then known as the Entry Coverage Language, which has developed into as we speak’s IAM Insurance policies.
2010 – A brand new free tier (100Ok requests/month then, since expanded to 1 million requests/month), configurable most message size (as much as 64 KB), and configurable message retention time.
2011 – Extra CloudWatch Metrics for every SQS queue. Later that yr we added batch operations (SendMessageBatch and DeleteMessageBatch), delay queues, and message timers.
2012 – Assist for lengthy polling, together with SDK help for request batching and client-side buffering.
2013 – Assist for even bigger payloads (256 KB) and a 50% worth discount.
2014 – Assist for useless letter queues, to simply accept messages which have develop into caught as a result of a timeout or to a processing error inside your code.
2015 – Assist for prolonged payloads (as much as 2 GB) utilizing the Prolonged Consumer Library.
2016 – Assist for FIFO queues with exactly-once processing and deduplication and one other worth discount.
2017 – Assist for server-side encryption of messages and price allocation tags.
2018 – Assist for Amazon VPC Endpoints utilizing AWS PrivateLink and the flexibility to invoke Lambda capabilities.
2019 – Assist for Tag-on-Create and X-Ray tracing.
2020 – Assist for 1-minute metrics for extra fine-grained queue monitoring, a brand new console expertise, and outcome pagination for the ListQueues and ListDeadLetterSourceQueues capabilities.
2021 – Tiered pricing so that you just lower your expenses as your utilization grows, and a Excessive Throughput mode for FIFO queues.
At the moment, SQS stays easy, scalable, cost-effective, and extremely dependable.
From the AWS Heroes
We requested among the AWS Heroes to mirror on the success of SQS and to share a few of their success tales. Right here’s what we realized:
Eric Hammond (Serverless Hero) makes use of AWS Lambda Lifeless Letter Queues. They put alarms on the queues and ship inner emails as alerts when issues must be investigated.
Tom McLaughlin (Serverless Hero) has been utilizing SQS since 2015. He mentioned “My favourite use case is anytime somebody desires a queue and I don’t wish to handle a queuing platform. Which is at all times.”
Ken Collins (Serverless Hero) was not fully positive how lengthy he had been utilizing SQS, and provided a spread of two to eight years! He makes use of it to energy the Lambdakiq gem, which implements ActiveJob on SQS & Lambda.
Kyuhyun Byun (Serverless Hero) has been utilizing SQS to energy a push system that should maintain huge quantities of site visitors, and tells us “Because of SQS, I don’t contemplate constructing a queuing system anymore.”
Prashanth HN (Serverless Hero) has been utilizing SQS since 2017, and considers himself “late to the occasion.” He used SQS as a part of his first serverless utility, and finds that it’s best for connecting providers with differing throughput.
Ben Kehoe (Serverless Hero) instructed us that “I first noticed the facility of SQS when a colleague confirmed me retain state in a fleet of EC2 Spot situations by storing the state in SQS when an occasion was getting shut down, and having new situations verify the queue for state on startup.”
Jeremy Daly (Serverless Hero) began utilizing SQS in 2010 as a light-weight queue that fed a facial recognition course of on user-supplied photos. At the moment, he typically makes use of it as a buffer to throttle requests to downstream providers that aren’t but serverless.
Casey Lee (Container Hero) additionally began to make use of SQS in 2010 as a alternative for ActiveMQ between loosely coupled Java providers. Casey implements auto scaling based mostly on queue depth, and has discovered it to be an correct strategy to deal with the load.
Vlad Ionecsu (Container Hero) started his AWS journey with SQS again in 2014. Vlad discovered that the API was very straightforward to know, and used SQS to energy his thesis mission.
Sheen Brisals (Serverless Hero) began to make use of SQS in 2018 whereas constructing a proof-of-concept that additionally used Lambda and S3. Sheen appreciates the flexibility to regulate the traits of every queue to create a great match for the message processing capabilities, and in addition makes use of each excessive and low precedence queues.
Gojko Adzic (Serverless Hero) started to make use of SQS in 2013 as a job dispatch for exporters in MindMup. This on-line mind-mapping utility permits massive teams of customers to collaborate, and requires strict ordering of updates to every doc. Gojko used FIFO queues to permit them to course of messages for various paperwork in parallel, with sequential order inside every doc.
Sebastian Müller (Serverless Hero) began to make use of SQS in 2016 by constructing a notification heart for website-builder jimdo.com . The middle ensures that clients are stored conscious of occasions (orders, help messages, and phone requests) on a well timed foundation.
Luca Bianchi (Serverless Hero) first used SQS in 2012. He decoupled a pair of microservices operating on AWS Elastic Beanstalk, and in addition created a fan-out processing system for a gamification platform. At the moment, his favourite SQS use case shops inference jobs and makes them out there to a employee course of operating on Amazon SageMaker.
Peter Hanssens (Serverless Hero) makes use of SQS to dump duties that don’t must be processed instantly. A number of years in the past, whereas helping some knowledge scientists, he created an event-driven batch-processing system that used a Lambda perform to verify a queue each 5 minutes and fireplace up EC2 situations to construct fashions, whereas maintaining strict management over the utmost variety of operating situations.
Serkan Ozal (Serverless Hero) has been utilizing SQS since 2013 or so. He focuses on asynchronous message processing and counts on the flexibility of SQS to deal with an enormous variety of occasions. He additionally makes makes use of of the message visibility function in order that he can re-process messages if essential.
Matthieu Napoli (Serverless Hero) has been utilizing SQS for about 5 years, beginning out with EC2, and as a alternative for different queues. As he says, “Paired with Lambda, it offers huge parallelism out of the field with out having to suppose an excessive amount of about it. Plus the built-in failure dealing with makes it very strong.”
As you possibly can see, there are a mess of use instances for SQS.
SQS Assets
In case you are not already utilizing SQS, then it’s time to get within the queue and get began. Listed here are some assets that will help you discover your means:
Completely happy queueing!
— Jeff;
[ad_2]
Source link
