Intro to AWS Monitoring & Messages Processing Tool
CloudWatch | CloudTrail | AWS Config | SQS | SNS | EventBridge
So in this article, we will have a simple review and summarize what AWS monitoring and logging services are, how they differ from each other, and when to use them.
We wouldn’t be diving into any technical concepts or setup. This is just a very high-level overview of these services, just a one-stop document with the needs to know about Monitoring and logging on AWS.
CloudWatch
CloudWatch is an AWS monitoring service to monitor applications. It’s used primarily for log and metrics analysis. CloudWatch offers features that are used to do the following 3 main things: collect, monitor, and analyze application health.
Collect:
- things like your application logs
- and other general AWS service logs such as maintenance, errors, etc.
Monitor:
- using metrics graphs that help track things like CPU and RAM usage for certain applications running on EC2 or a backend API application.
- Alarms that you can set to alert you by sending text or email notifications when something out of the ordinary or wrong is happening with your application, like CPU usage above 90% or network throughput exceeding your desired value.
Analyze:
- using Cloudwatch insights, you can perform queries like we do with SQL to get insights 🤷🏾♂️ 😎😅
There are a couple more components and features that Cloudwatch offers, such as Cloudwatch rules (kinda like CRON jobs) and EventBridge (your application events).
CloudTrail
CloudTrail unlike Cloudwatch; is more of an auditing tool for AWS accounts, a monitoring tool for users and resources. To put it simply:
“CloudTrail analyze WHO performed WHAT actions and WHEN on your resources”
I didn’t come up with this great and straightforward way to explain CouldTrail, all credit goes to the “Be a Better Dev” channel and a great video post on YouTube. CloudTrail features 3 main event types: management, insights (here’s that word again 🤷🏾♂️😅), and data.
Management events:
They are mostly administrative actions events such as the creation of new resources, updating existing resources, deleting, who signs in and granting access, and things like that.
All these events are created automatically upon your AWS account creation, you can go to Cloudtrail and review the events there for yourself.
Data events:
These are a bit deeper contexts with higher throughput, like when a lambda function is invoked or a query is made to the DynamoDB table. Important note: Data events don’t come with default settings, you have to enable them on a particular AWS service to capture data events.
Insights events:
They allow you to detect when anything out of the ordinary is happening concerning access and/or usage of your application using the AWS machine learning algorithm. An example might be: that you have a backend API app that typically receives a thousand calls per hour. If all of a sudden, it's receiving100,000 calls per hour, cloudTrail will then flag it as an insight event, which will also be reported to Cloudwatch.
CloudTrail also offers the option to export and archive data to a storage like S3 or S3 Glacier for a low-cost retention of the access data; which you can then use later for compliance reporting and such.
AWS Config
To put it briefly, AWS Config provides a detailed view of the configuration of AWS resources in your AWS account. This includes how the resources are related to one another and how they were configured in the past so that you can see how the configurations and relationships change over time. (Source: AWS Config)
Honestly, all of these services discussed above are very similar to each other, with similar principles but they are not the same nor have the same function. You can leverage their features and functionality to accomplish specific tasks and objectives as shown below:
Now let’s talk about message-processing tools. Imagine a scenario where you have an e-commerce application hosted on AWS. You have a customer who makes an order, which calls an “/order API” on your app and triggers an “OrdersServices” on AWS. That order then goes through a series of steps or event stages before completion, such as (pending, reviewed, processed, shipped, and delivered). You have an “AnalyticsServicer” which might be running as a lambda function that needs to capture, analyze, and generate customer order statistics or metrics for your business.
Message processing services or tools are used to help better “process these events situation. AWS provides a few options with different use case scenarios such as SQS, SNS, EventBridge, and Kinesis. They can be very confusing and seem like they all do similar things but not the same thing. So let's try to clarify what each of them does and when to use it.
SQS — Simple Queue Service
Amazon SQS is a synchronic communication service. It allows Applications to publish messages to a queue, which is essentially just a temporary holding area to store messages that will be processed later. This way of doing things helps decoupling applications from one another while still letting them communicate and exchange data.
In the example above, SQS will help decouple our 2 applications: “OrdersServices” and “AnalyticsServicer”. SQS works based on these 3 concepts: queues, messages, and polling:
Queues:
These are the basis of SQS, you are going to create queues directly on the console ou through IaC with Couldformation or other tools. Queues are where your application is going to be publishing messages (temporary holding pools).
Messages:
Messages are messages 🤷🏾♂️😎😂 well they have to be in a certain format tho; like a raw JSON that can contain any data, there are some limitations on size. you can look it up for more details.
Polling:
This is essentially the mechanism that allows the subscriber to retrieve messages to be processed from the queue, like when the“AnalyticsServicer” application retrieves a message published on a queue by the “OrdersServices” application.
SQS is generally best used for one-to-one decoupling of services or applications.
SNS — Simple Notification Service
SNS is very similar to SQS but has different functions and operates differently. It works based on these 3 concepts: topics, messages, and publish/subscribe:
Topics:
Unlike queues, topics don’t hold all the orders coming in. Think of it as classification by order theme like location, categories, price range, order stage, etc. When a new event occurs in the application, the application will then publish a message about that event to a topic, that topic will then deliver a copy of that message to other services or applications that have subscribed to the topic.
so In our example, let’s say you have a service that needs to route all the canceled orders, that service will then subscribe to a topic cancel, so whenever a cancel order event happens, a message will be published in the cancelOrder topic and a copy of that message will be delivered to the subscriber of to the cancelOrder topic.
Messages:
Messages are similar to SQS messages, they can contain any date in a raw JSON format. there is a size limit as well.
Publish/Subscribe aka Pub/Sub:
Pub/Sub is an asynchronous and scalable messaging service that decouples services producing messages (services or applications where an event happened or is happening) from services processing those messages (services or applications that need to know about the event happening or has happened). Not much to say here, it's pretty straightforward.
Important Note: From an architecture perspective, it’s generally n to a good idea to have an SNS directly connected to a service or application to avoid loss of data. meaning if the subscriber app is down, it won’t get messages being published in the topic, and by the time it’s up and running, it’ll not have all the messages that it missed receiving while it was down.
To address this, adding an SQS owned by the subscriber that seat in between them and the SNS topic they subscribed to.
Note Update: Amazon SNS now supports in-place message archiving and replay for FIFO topics. You might not need to add an SQS anymore.
EventBridge
EventBridge is a fairly new feature on AWS. Same concept as SQN and SNS but a little different. It works based on these 4 concepts: message bus, events, rules, and targets.
Message Bus:
Message bus is just like a topic in SNS, same thing and the same concept.
Events:
Events are essentially just the same as those we discussed previously based on our example such as shipped, pending, delivered, etc.
Rules:
In a much simpler way, rules are just what sort or classify incoming events to match and send them to their corresponding target.
Targets:
A target is a resource or destination endpoint where EventBridge sends an event when that event matches the event pattern defined for a rule. They allow the filtering of messages as well (kinda like subscribing, in context to SNS).
So in a bulk view, EventBridge is very similar to SNS and uses the same principle. The one thing that makes it stand out is the automatic integration feature with third-party applications; without having to write custom code. Some of these third-party app are PagerDuty, DataDog, Shopify
However, Only up to five targets can be defined for each rule. As of 2024, this is one of the biggest limitations to keep in mind as you are thinking about which of these services to use:
This diagram is from a post by Mrudhula Balasubramanyan. It demonstrates how to use event-driven architectures for coordinating applications that need to exchange large messages across AWS accounts and Regions. The Payload Offloading Java Common Library enables AWS’s messaging and automation and uses Amazon S3 as the intermediate data store. These components can simplify the solution implementation and improve your applications’ scalability, fault tolerance, and performance.
So yeah, pretty cool stuff. There are plenty of online resources where you can learn more about the things I discussed here. As mentioned earlier, I didn’t come up alone with all these great and straightforward ways to explain these AWS services, credit goes to the “Be a Better Dev” channel on YouTube. Check it out, they have some great video posts.
If you enjoyed this article, you might also like What is Apache Kafka “In Simple English”
