Message Queues : importance ,need and implementation using Amazon SQS
What is message queuing?
Message queuing allows applications to communicate by sending messages to each other. The message queue provides temporary message storage when the destination program is busy or not connected.
Message queues
A message queue provides an asynchronous communications protocol, which is a system that puts a message onto a message queue and does not require an immediate response to continuing processing. Email is probably the best example of asynchronous communication. When an email is sent, the sender continues to process other things without needing an immediate response from the receiver. This way of handling messages decouples the producer from the consumer so that they do not need to interact with the message queue at the same time.
Decoupling and Scalability
Decoupling describes how much one piece of a system relies on another piece of the system. Decoupling is the process of separating functions so that they are more self-contained.
A decoupled system is achieved when two or more systems are able to communicate without being connected. The systems can remain completely autonomous and unaware of other functions. Decoupling is often a sign of a computer system that is well structured because it is easier to maintain,
If one process in a decoupled system fails to process messages from the queue, other messages can still be added to the queue and be processed when the system has recovered. You can also use a message queue to delay processing — for example, a producer posts messages to a queue. At the appointed time, the consumers start and process the messages in the queue. A queued message can be stored-and-forwarded, and the message can be redelivered until it is processed.
What is Amazon SQS service?
Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. Get started with SQS in minutes using the AWS console, Command Line Interface or SDK of your choice, and three simple commands.
SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.
Benefits
Eliminate administrative overhead
AWS manages all ongoing operations and underlying infrastructure needed to provide a highly available and scalable message queuing service. With SQS, there is no upfront cost, no need to acquire, install, and configure messaging software, and no time-consuming build-out and maintenance of supporting infrastructure. SQS queues are dynamically created and scale automatically so you can build and grow applications quickly and efficiently.
Reliably deliver messages
Use Amazon SQS to transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be available. SQS lets you decouple application components so that they run and fail independently, increasing the overall fault tolerance of the system. Multiple copies of every message are stored redundantly across multiple availability zones so that they are available whenever needed.
Keep sensitive data secure
You can use Amazon SQS to exchange sensitive data between applications using server-side encryption (SSE) to encrypt each message body. Amazon SQS SSE integration with AWS Key Management Service (KMS) allows you to centrally manage the keys that protect SQS messages along with keys that protect your other AWS resources. AWS KMS logs every use of your encryption keys to AWS CloudTrail to help meet your regulatory and compliance needs.
Scale elastically and cost-effectively
Amazon SQS leverages the AWS cloud to dynamically scale based on demand. SQS scales elastically with your application so you don’t have to worry about capacity planning and pre-provisioning. There is no limit to the number of messages per queue, and standard queues provide nearly unlimited throughput. Costs are based on usage which provides significant cost saving versus the “always-on” model of self-managed messaging middleware.
How SQS is used by the industry
Environmental Monitoring Solutions (EMS) Case Study
EMS Delivers 500% ROI through IoT Capabilities Using AWS
With our AWS IoT–enabled Fuelsuite solution, customers manage their petrol stations proactively rather than reactively… to dramatically improve efficiencies and detect fuel leaks early to minimize environmental impacts.
Russell Dupuy Founder and Managing Director, EMS
About Environmental Monitoring Solutions (EMS)
Environmental Monitoring Solutions (EMS) is based in Victoria, Australia. Launched 25 years ago, the company specializes in solutions that help petrol retailers gather and analyze data on the performance of their petrol stations. Its solutions provide remote monitoring and 24/7 support services — helping customers boost sales, reduce maintenance expense, and decrease the risk of accidents. Today, EMS operates with a team of 30 personnel.
The Challenge
EMS customers such as Viva Energy (Shell), PUMA Energy, BP, and 7-Eleven typically own and operate hundreds of petrol stations across Australia. The stations need to operate highly efficiently because profit margins are small. Yet, at the same time, they have to offer great customer experiences, ensure employee safety, and minimize their environmental impact. Sometimes accidents do occur, and a typical EMS customer is likely to incur annual costs of AU$15 million (US$12.13 million) for cleaning up underground petrol tank leaks or vehicle fuel tank contamination. To help customers maximize efficiencies while addressing the need for service excellence, safety, and environmental protection, EMS developed Fuelsuite, which enables customers to switch from legacy technologies that are largely manual and unify station management — significantly reducing costs. With Fuelsuite, customers can monitor inventories, deliveries, and prices. The solution also raises alarms in event of possible environmental incidents, such as underground tank overfills. After the successful launch of Fuelsuite, EMS focused on product development. It looked to connect sensors in the stations’ underground tanks and pumps and, regardless of the configurations of those sensors, collect all their data at 30-second intervals. The data would then be aggregated on a cloud-computing infrastructure and displayed via a web-enabled interface in Fuelsuite in near-real time. Russell Dupuy, founder and managing director of EMS, says, “Our job was to find an IT partner and cloud-computing provider that could help us re-engineer our Fuelsuite technology and deliver an innovative off-the-shelf product that was user-friendly and easily customizable.”
Why Amazon Web Services
EMS researched the market for cloud-computing providers, focusing specifically on providers of Internet of Things (IoT) technology. Through IoT, EMS would be able to collect all the sensor data and deliver it to the Fuelsuite interface. In 2015, when EMS first started looking at cloud providers and their IoT services, Dupuy found that Amazon Web Services (AWS) was way ahead of Microsoft Azure. “From what I’d read, I was more confident in the AWS IoT roadmap,” says Dupuy. “We were investing millions of dollars in this project, so I had to be sure of the provider we were going to work with.”
EMS then engaged with AWS Partner Network (APN) Advanced Consulting Partner DiUS, which organized a two-day workshop with key stakeholders to agree on a shared vision for Fuelsuite. “The expertise that DiUS has in AWS and IoT gave us a lot of confidence,” says Dupuy. Just as importantly, DiUS agreed to the plan of designing, supporting productization, and launching the IoT-enabled Fuelsuite solution for a major EMS customer retail network within 12 months. “DiUS showed that it was possible to meet our schedule by rolling out the platform at first with just enough AWS IoT capabilities for customers, and then extending those capabilities down the line.”
DiUS and EMS designed the new version of Fuelsuite so that data from the sensors is aggregated by a custom-built physical device, called Fuelscan®, which is situated inside the petrol stations. Fuelscan then wirelessly transmits the data to the AWS Cloud, where it gets processed and delivered via the Fuelsuite web-based interface. So that EMS is able to offer customized functionality for different customers, DiUS designed the solution to take advantage of AWS IoT Device Management to securely onboard, organize, monitor, and remotely manage the Fuelscan devices. It also uses Amazon Simple Queue Service (Amazon SQS) to queue messages from the AWS Cloud to Fuelscan devices and relies on Amazon Elastic Compute Cloud (Amazon EC2) to process the messages to and from the devices. In addition, the sensor data is stored from the devices in an Amazon Relational Database Service (Amazon RDS) instance, and Amazon Elasticsearch Service is used to keep a historical record of all Fuelscan data. Finally, the collected data is archived in Amazon Simple Storage Service (Amazon S3) buckets.
The Benefits
At present, more than 1,000 petrol stations in Australia are benefiting from the IoT-enabled Fuelsuite solution. Station operators get near-real-time data on the performance of their stations, including how much petrol is being sold and how much is in the underground storage tanks. It also includes data on the pressure inside the hoses connecting the petrol pumps to the automobiles, and on the temperature and petrol level inside the underground tanks. “With our AWS IoT–enabled Fuelsuite solution, customers manage their petrol stations proactively rather than reactively — gaining a complete picture of petrol station performance to dramatically improve efficiencies and detect fuel leaks early to minimize environmental impacts,” says Dupuy.
The potential savings from proactive rather than reactive management are significant. Likewise, operators can use the performance data on fuel sales to optimize the distribution of the most heavily used petrol pumps. “We expect that customers gain a 500 percent return on their annual costs of the AWS IoT–enabled Fuelsuite solution,” says Dupuy. Furthermore, EMS expects the return on investment to increase over time. “The AWS IoT Core platform is extensible so that we can incorporate more data from the petrol stations.”
Dupuy says that without AWS and DiUS, EMS would have found it challenging to take advantage of IoT. “The expertise of DiUS and the total reliability of the AWS infrastructure meant we could develop and roll out our IoT-enabled Fuelsuite in 12 months and be sure it would meet the needs of our customers.”
EMS, in partnership with DiUS, is now developing Fuelsuite further, adding new services for the fuel industry. It will also be launching the product internationally. Dupuy says, “An advantage of AWS is that machine learning, deep learning, and other new artificial intelligence services can easily be applied to a customer’s aggregated data that is collected through the Fuelsuite solution.”
