Incident Bot on Azure Functions
This blog is part of the series where we discuss DevOps concepts from Ground Zero for an audience that has limited starting knowledge. This article comes in the Intermediate Series since it involves understanding the working of serverless application aka IncidentBot on Azure Functions.
Some of the earlier blogs in the Cloud Series are as follows.
TweetBot
Azure Functions-GitHub Actions
NodeJs+Nginx
What are Azure Functions?
Functions is the flagship offering of Azure as part of FAAS(Functions as a service)/Serverless category where app-teams don’t have to take care of additional complexity of managing the infrastructure. Generally used for deploying stateless micro-services in Python, Java, C# etc; Functions comes in three plans — Consumption, Premium and Dedicated Plan. These plans can be used depending on type of workload- dev, stage and production. Consumption is the pay as you use model, recommended for running POC’s and experiments.
Automated Incident Management
For running any large scale operations, the end-2-end incident management system should have these 3 key elements- monitor, detect, alert without any human involvement. The nature of automated system should be independent of the metrics being collected between business, system and application anomalies. Ideally an orchestrator connecting the source of alerts with notification systems such as PagerDuty, ServiceNow, JIRA depending on the agreed SOP with the operations team is pivotal to scale operations in production.Additionally the system should cater to- reducing noise, driving actions and improving the end to end MTTD and MTTR metrics.
The below example is one such use-case in orchestrator connecting Azure Alerts with JIRA using Azure Monitor Action Groups and Azure Functions.
Creating the Function-App
The Azure Function App can be created from either of these options -
i) From the Azure Portal- https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-function-app-portal
ii) From a generic Terraform module- https://github.com/innovationnorway/terraform-azurerm-function-app.
Once the Function-app is created, the following resources will show up in the Azure Portal. An application insights resource is needed for debugging and logging needs.
Creating a Java Module
In order to create a Java Module, we need to break the overall incident management functionality into three parts-
i) HTTP Trigger Class- The Class which gets called when the actual web hook is triggered.
ii) JIRA Service Class- — The Service Class which is responsible for authenticating and creating an issue with the JIRA Provider.
iii) JIRA Model — The Model Class which which maintains the data elements needed to create a JIRA task/issue/epic.
Deploying the Azure Functions
The following medium blog can be used to deploy the code to Azure Functions using GitHub Actions.
Post the Azure Function is deployed, the endpoint can be triggered by using a GET operation on the given endpoint- https://medium-blog.azurewebsites.net/api/IncidentBot . This endpoint will create a JIRA incident/task.
Creating the Azure Monitor Action Group
i) Click on Monitor on the Azure Portal and select Action Groups from it.
ii) Create a new Action Group ( jira-incident bot) and add the above webhook url as the notification system.
Creating the Alert Rule
i) On the left navigation tab from the given resource, click on the Alerts and from there Create Alert Rules.
ii) Select the Alert Metric, Alert Frequency and other metrics needed for a normal alert creation.
iii) Select the Action Group as created above , so the next time Alert is triggered, the web-hook on the Function-App is triggered.
End 2 End Testing
i) Hit an endpoint on the Azure VM which results in a 404 message.
ii) Alert Condition changes to Fired as shown below
The following issue on JIRA board is created and assigned to the on-call person.
Additional Resources
For feedback, please drop a message to amit[dot]894[at]gmail[dot]com or reach out to any of the links at https://about.me/amit_raj.
