By default, when managing a large number of EC2 instances, you don't get a lot of visibility into how your instances are behaving. Any monitoring beyond what you get in the console needs to be configured and set up by the AWS user using services like Cloudwatch or Cloudtrail, etc.
Publish an AWS Health event formatted as a Slack API chat.postMessage message to an SNS topic. This currently supports AWS Health notifications and not issues because issues are not sent as aws.health events. This is intended to be used with aws-sns-to-slack-publisher. Service Interface. Event Type: AWS CloudWatch Event - Health Event. In CloudWatch, you define alarms to send a message to an SNS topic if the monitoring data gets out of normal bounds. Finally, you connect a Lambda function to the SNS topic to trigger a function execution. The Lambda function calls the Slack API to send a message. The following figure shows the data flow. SSE protects Amazon SNS messages by using keys managed in AWS Key Management Service (AWS KMS). To manage SSE using AWS Management Console or the AWS Service Development Kit (SDK), see Enabling Server-Side Encryption (SSE) for an Amazon SNS Topic in the Amazon Simple Notification Service Getting Started Guide. The AWS Cloud Development Kit (AWS CDK) is an open-source software development framework to define cloud infrastructure in code and provision it through AWS CloudFormation. It offers a high-level object-oriented abstraction to define AWS resources imperatively using the power of modern programming languages.
EC2 instances can be started or stopped for various reasons by anyone with access to the EC2 console, and your team might want to know about it.
The AWS Cloud Development Kit (AWS CDK) is an open-source software development framework to define cloud infrastructure in code and provision it through AWS CloudFormation. It offers a high-level object-oriented abstraction to define AWS resources imperatively using the power of modern programming languages.
In Zuar's case we manage hundreds of EC2 instances across three regions running our products Mitto, Rapid Portal and Custom Portal. We wanted to get notifications in our #devops slack channel anytime an instance state changed.
If you're familiar with Terraform feel free to skip everything below! Simply Edit and use this code.
Overview
AWS Cloudwatch allows you to create rules which will publish to an SNS topic any time a specific EC2 instance or list of instances has a state change. You can subscribe emails to this SNS topic and receive emails anytime there's a new SNS publication. In this case, we'll subscribe a custom Lambda function, that formats the slack message, and then makes a POST request to our slack channel.
Flow
Aws Sns Slack Lambda
- A user on your AWS account stops an EC2 instance.
- Your Cloudwatch Event Rule is triggered and publishes to your SNS Topic
- SNS invokes your Lambda function with the message Cloudwatch published
- Your Lambda function formats the message for slack and does a POST request to your webhook.
Results
The messages in your slack channel should look something like this:
Slack Webhook
First and foremost, you need a Slack channel with an incoming webhook. It should give you a link similar to this which we will use later in our Lambda function: https://hooks.slack.com/services/UNIQUE_ID/UNIQUE_ID/UNIQUE_ID .. You can test that the link is working using curl like this:
Lambda Function
I won't get into the specifics of Lambda deployment here. It'll need an execution role, a policy, and a policy attachment, some of which the console will create for you. Later you will add a trigger for SNS. Our Lambda uses Python3.6 and there are 4 environment variables you will need to set. ACCESSKEY, SECRETKEY, REGION and SLACK_HOOK. Below is our lambda function:
NOTE: When packaging this for deployment, be sure to include requests as a dependency.
Simple Notification Service (SNS)
Still with me? Next you'll need to create an SNS topic, and then create a subscription with your Lambda function. In the SNS console you would click 'Create Topic'. All you need here is a name. Once that's done, you'll go to 'Subscriptions' and create a subscription. enter the ARN for the topic you just created, select 'AWS Lambda' as the protocol, and then select the ARN for the Lambda we created.
Cloudwatch Rule
Aws Sns To Slack Cable
In the Cloudwatch console. Click on 'Rules' under events in the menu on the left. Click on 'Create Rule'. On the left select 'Event Pattern', service 'EC2', event type 'EC2 Instance State-change Notification'. Select 'Any state' and 'Any Instance' to publish to SNS on changes to any instance in this region.
On the right select 'SNS Topic' at the top then select the SNS topic you created. Below that select 'input transformer'. In the first textarea use:
and in the Second textarea use:
Our Lambda function will expect the format above.
Save your changes, and it's time to do some testing.
Testing and Troubleshooting
To test everything we just created, simply go to the EC2 console and start and stop an instance you're not using in the region you created your resources in. If you don't get a message in your slack channel, check the cloudwatch logs for your Lambda function and ensure that the function was invoked. If the function was not invoked, you might need an SNS trigger on your Lambda function. Also double check you have all the permissions, roles, and policies you need for Lambda to receive events and be invoked by SNS.
Aws Sns To Slack Download
As we all know, things go wrong. That’s why monitoring and alerting are essential topics. Wouldn’t it be nice, if problems in your AWS account would show up in Slack? So you can react quickly while using your favorite messaging tool. In this blog post, you will learn how you can turn CloudWatch Alarms into Slack messages like this:
How it works
On AWS, everything sends monitoring data (CPU utilization, estimated monthly charges, …) to CloudWatch. In CloudWatch, you define alarms to send a message to an SNS topic if the monitoring data gets out of normal bounds. Finally, you connect a Lambda function to the SNS topic to trigger a function execution. The Lambda function calls the Slack API to send a message. The following figure shows the data flow:
To deploy the components in the figure, you will use the Serverless Application Model (SAM). If you are not interested in implementing this on your own, give our Slack chatbot a try. Never miss an alert from your AWS infrastructure with marbot!
Implementing the Lambda function
Aws Chat
You will use Node.js to implement the Lambda function. To send a request to the Slack API, you have to make an HTTPS request. The request module is easy to use, but I wanted a variant of the module that returns promises to avoid callback hell. That’s why I used request-promise-native. The Slack webhook URL is passed in as an environment variable that you define later in the CloudFormation template.
Messages delivered from SNS to the Lambda function will look like this:
You need to convert the format into the Slack message format.
Aws Sns To Slack Test
Finally, each Lambda function needs a handler function. The handler function takes 3 parameters: