AWS Project : Custom Notifications for your workflow
Generate email notifications in case of any error (or any custom event log) for an AWS resource. Using AWS CloudWatch, SNS and Lambda services.
Problem statement
Generally when we use or run any specific AWS resource (for any service like lambda, S3 etc.) the execution logs are generated (provided that it has required access for CloudWatch). These logs can be viewed on AWS CloudWatch. If there is any error or any custom message/text we are logging/printing, all can be viewed in respective logs. So, if we have any automated or scheduled process on AWS, it becomes very redundant to manually go and check the logs for any error or any specific message. It is quite a manual task and takes can take some time to check individual logs and search for stuffs.
Therefore along with automating our workflow, we can automate the log filtering (searching for specific things) and notification process (sending an email or SMS).
Project implementation overview
So, this time we are going to create an automatic notification workflow to track logs of our AWS service and send email notification based on some patterns found in the execution logs of that service.
“Logger” Lambda function
AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers.
The “logger” lambda function is the resource for which we will be tracking the execution logs. It will be having CloudWatch access to put the log events within a particular log group. Each log group is dedicated to a particular resource. It consists of log streams i.e. group of log events.
Each log stream and log event is marked by a timestamp for differentiation.
CloudWatch log group
Amazon CloudWatch is a monitoring and observability service. It provides you with data and actionable insights to monitor your applications, respond to system-wide performance changes, and optimize resource utilization.
Then we will put a subscription filter on the CW log group of “logger” lambda function. It will enable us to filter the multiple log events (Ex : Pick those events with ERROR text in it) and send the log stream to another lambda function (i.e. “mailer” lambda function) for processing.
“Mailer” Lambda function
It will get the log streams from CloudWatch and it will process it to create a suitable message based on the log events. Then it will publish the message to an SNS topic.
SNS topic
Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.
We will create an SNS topic and subscribe to it using our email id. So, as soon as the “mailer” lambda function will publish the message to it, it will send an email (with custom subject line) to the mail id(s) which have subscribed to it.
Step-1 : Create “logger” lambda function
First we will create a lambda function for which we will track the logs and in case of any ERROR or CRITICAL message we will send an email notification.
On AWS console search for “Lambda” service. Then click on Create Function and create a lambda function with Python-3 runtime. Keep other settings as default.
Note : The new default lambda role will have a CloudWatch access (to create CloudWatch log group and put events. If we use a different role for lambda function, we need to ensure that it should have CloudWatch accesses (can add CloudWatch full access policy for time being).
Then add the below code and click on Deploy to save the function.
import json
import loggingdef lambda_handler(event, context):
# TODO implement
logging.critical(‘Custom critical message’)
print(pandas.DataFrame())
print(‘This line will not be printed as error occurs in previous line’)
Note : Maintain proper indentation in code.
Here logging library is used to log or print the text we want.
It is better than using print() because:
- We can add severity levels like critical, debug, info etc.
- We can log to files, sockets, pretty much anything, all at the same time.
- We can easily turn it on/off and get less/more info without changing the logging statements.
There are a lot of good blogs on web to see about advantages of logging.
So, a line starting with [CRITICAL] will be seen in the lambda function logs.
Then as we have not imported pandas library (also it is not pre-installed), it will throw us an error line stating with [ERROR].
The default file name will be lambda_function.py.
Test the lambda function
Click on the Test. If you are running the function for first time, the add an event name (Ex: test), save it and click on Test.
We can see the full log for this run in CloudWatch console. Go to CloudWatch and click on log groups. Search for a log group with lambda function name. Then click on the latest log stream.
Or select the Monitor tab and click on View in CloudWatch logs. Then select the latest log stream.
This shows the expected critical and error message.
Step 2 : Create an SNS topic
Now we will create a SNS topic to which we will publish our message from “mailer” lambda function.
Go to the Simple Notifications Service dashboard on AWS console. Click on Topics and create a topic.
Choose Standard type and provide a name to the topic. Then add a display name which will be the subject line of the email we will send.
Then select the topic and get the Topic ARN to be used in mailer lambda function.
Create subscription
Now let’s subscribe this topic from our mail id.
Click on the topic and Create subscription. Add Protocol as Email and Endpoint as your own email id.
Note : You can use this website to generate and use a temporary email address for testing purpose https://temp-mail.org/en/.
Then add subscription and just check your mailbox and confirm the subscription.
Step 3 : Create “mailer” lambda function
Now we will create a lambda function that will
- Collect the CloudWatch log streams for “logger” lambda function
- Create message body as required from the log data
- Then publish the message to SNS topic which will be delivered to our email.
Add the following code and click on Deploy to save.
import json
import boto3 #AWS SDK for Python
import base64 #To decode the message
import gzip #To decompress the message#create sns boto3 client to interact with SNS topic
sns_client = boto3.client(‘sns’)def lambda_handler(event, context):
# TODO implement
#decompress and decode the event data
decompressed_decoded_event = json.loads(gzip.decompress(base64.b64decode(event[‘awslogs’][‘data’])))
messages = ‘’
for event in decompressed_decoded_event[‘logEvents’]:
messages += event[‘message’]
messages += ‘\n — — — — — — — — — — — — — — — — — — — — — — — — — — -\n’
#create body of the message
body = ‘’’
LogGroup : {}, \n
LogStream : {}, \n
Message : {}
‘’’.format(decompressed_decoded_event[‘logGroup’], decompressed_decoded_event[‘logStream’], messages)
print(body) def send_message(body):
sns = sns_client.publish(
TopicArn = ‘arn:aws:sns:ap-south-1:<AWS_account_number>:sks-mailer’,
Message = body,)
send_message(body)
The lambda will get the new log events from CloudWatch as the event parameter which will be in the form of JSON (or dictionary). First we need to decode this message and then decompress.
Just put the TopicArn which we have previously copied.
Add permission to access SNS
As our lambda function will try to publish the message to an SNS topic, we need to provide permissions to it’s IAM role to access SNS. For the time being we can add SNSFullAccess policy.
Click on the Configurations tab and click on Permissions. Click on the Role link in another tab.
Click on Add permissions, Add policy and search “AmazonSNSFullAccess” policy. Click on the checkbox and Add policy. Then the policy will be added in the lambda function role.
Step 4 : Create subscription filter for log group
Now we will create a subscription filter for the “logger” lambda CloudWatch log groups. This will stream the logs with specific patterns into the “mailer” lambda function.
Click on the “logger” lambda log group and go to Subscription filter tab. And “create lambda subscription filter”.
Choose the “mailer” lambda function.
Log format keep as JSON. This filter pattern means “select every log event with ERROR or CRITICAL string. Give a name to the filter.
For more information on filter patterns, have a look at this page : https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html#matching-terms-events.
Then select the latest log stream (it will consist of the error and critical log) under Select data to test. Then click on Test pattern to check if it is picking correct texts.
Click on Start streaming.
Then go inside the “mailer” lambda function and click on the CloudWatch trigger.
Scroll down to see the trigger and click on Details.
Here we can see the filter name, pattern etc.
Step 5 : Let’s test our architecture
Now just select the “logger” lambda function and click on Test. Now we can see the CloudWatch logs for the “logger” function and “mailer” function to see if these executed as per our expectations.
If everything goes fine then we can go and check mailbox to see a similar message from AWS SNS.
Done!!
So, in this way we can create custom notifications for our AWS resource logs.
Alternative approaches
- We can directly add the CloudWatch log group (of “logger” lambda function) as a trigger to the “mailer” lambda function. In this way as soon as the “logger” lambda function runs, a new log stream is created which triggers the “mailer” lambda function. Then the “mailer” lambda function can get the messages, process it and publish to SNS topic. But sometimes if there is not much of a time gap between subsequent runs of a lambda function, the new log events get added to the previous log stream. So, as no new log stream get created, the “mailer” lambda function will not get triggered.
- We can send notifications based on CloudWatch alarms. We can create custom metrics or use pre-built metrics to detect anomalies and based on that CW alarm will be triggered. This will send a notification to the client. But this approach does not provide much customization like picking up messages based on patterns, custom message body etc.
If you liked this project, kindly do clap. Feel free to share it with others.
Follow me for more such interesting contents and projects.
Connect with me at LinkedIn.
Thank you!!
