Questions? Feedback? powered by Olark live chat software
Hopp over navigasjon

Using Azure Automation to take action on Azure Alerts

Posted on 28 september, 2015

Sr. Program Manager, Automation and Integration

In this blog post, I will talk about how to use Azure Automation runbooks to initiate actions in response to Azure alerts.

Webhooks allow the user to route the Azure Alert notifications to other systems for post-processing or custom notifications. For details on Azure Alert notifications see documentation.

Are you wondering how could you take some action besides getting an email notification when the alert fires (for example, calling a script to perform automated remediation)? Then you don’t have to wait any longer, Azure Automation now provides a capability within the Azure Alerts pane to do just that. You can now have any Azure alert call an Azure Automation webhook when the alert fires.

For example, when the value of a specified metric crosses an assigned threshold, the alert rule becomes active and can trigger an Automation runbook to perform some actions. The runbook is triggered by invoking an Automation webhook, which allows you to start a particular runbook in Azure Automation through a single HTTP request. New to Azure Automation? Learn more here.

Here are the detailed steps for configuring an automation runbook with Azure Alerts. Scenario:  When the disk read on my virtual machine (VM) is more than 1MBs/sec over the last hour, I want to get notified via email and also and want to restart the VM (which is done by invoking an Automation webhook).

Detailed steps:

  1. Set the alert threshold: In the Azure preview portal, click Browse, select the VM resource you're interested in monitoring, then add the condition (metric: disk read; condition: greater than; threshold: 1MB/over last hour) in the alerts screen.

  2. Author the runbook: Create a runbook in Azure Automation to react to  the alerts. I will walk through a sample runbook(below) to get the webhook data and VM context so that you can take action on it.

  3. Create webhook: Create a new webhook linked to a runbook in the Azure preview portal. Copy the URL of the webhook (make sure you keep track of your webhook URLs, because once you have created your webhook, the url will no longer be viewable from the Azure portal).

  4. Update the Alert threshold with the webhook URL: Now go to appropriate resources in Azure which you want to monitor and alert on, and in the Create/Update Alerts screen, you can add the webhook URL

  5. Once the alert condition is met and triggers the runbook, it sends the alert context to the runbooks.  As you can see, the alert context contains details like subscriptionID, resourceGroupName, resourceName, resourceType, resourceId and timestamp, which are required for the runbook to identify the resource it will be working with. This is passed into the runbook as part of the webhookData object and is available in the webhook.RequestBody property. You can convert this to a JSON object and extract the alert context. See the sample output screen below.

    Click Browse, go to Automation Accounts –> Runbooks in the portal, to see the runbook job status and troubleshoot as needed.

Example of Alert context:

{"WebhookName":"AzureAlertTest","RequestBody":"{\"status\":\"Resolved\",\"context\":{\"id\":\"/subscriptions/<subscriptionID>/resourceGroups/Group/providers/microsoft.insights/alertrules/AlertVM\",\"name\":\"AlertVM\",\"description\":\"\",\"condition\":{\"metricName\":\"Diskread\",\"metricUnit\":\"Count\",\"metricValue\":\"361.811648\",\"threshold\":\"1073741\",\"windowSize\":\"5\",\"timeAggregation\":\"Average\",\"operator\":\"GreaterThan\"},\"subscriptionId\":\<subscriptionID> \",\"resourceGroupName\":\"Group\",\"timestamp\":\"2015-09-22T22:51:19.4549592Z\",\"resourceName\":\"Testvm\",\"resourceType\":\"microsoft.classiccompute/virtualmachines\",\"resourceId\":\"/subscriptions/<subscriptionID>/resourceGroups/Group/providers/Microsoft.ClassicCompute/virtualMachines/TestVM\",\"portalLink\":\"\"},\"properties\":{}}","RequestHeader":{"Connection":"Keep-Alive","Host":"<webhookURL>"}

Following is a sample runbook to get the webhook data and restart VM with comments. You can also access the Runbook Gallery for a library of already existing runbooks that could be modified to work with a webhook so they could be initiated by an Azure Alert.

Sample PS script Runbook

param ( 
    [object]$WebhookData
)

if ($WebhookData -ne $null) {  
    # Collect properties of WebhookData.
    $WebhookName    =   $WebhookData.WebhookName
    $WebhookBody    =   $WebhookData.RequestBody
    $WebhookHeaders =   $WebhookData.RequestHeader
       
    # Information on the webhook name that called This
    Write-Output "This runbook was started from webhook $WebhookName."
       
    # Obtain the WebhookBody containing the AlertContext
    $WebhookBody = (ConvertFrom-Json -InputObject $WebhookBody)
    Write-Output "`nWEBHOOK BODY"
    Write-Output "============="
    Write-Output $WebhookBody
       
    # Obtain the AlertContext
    $AlertContext = [object]$WebhookBody.context

    # Some selected AlertContext information
    Write-Output "`nALERT CONTEXT DATA"
    Write-Output "==================="
    Write-Output $AlertContext.name
    Write-Output $AlertContext.subscriptionId
    Write-Output $AlertContext.resourceGroupName
    Write-Output $AlertContext.resourceName
    Write-Output $AlertContext.resourceType
    Write-Output $AlertContext.resourceId
    Write-Output $AlertContext.timestamp
      
    # Act on the AlertContext data, for example, restart the VM.
       
    # Authenticate to your Azure subscription using OrganizationId to be able to restart that Virtual Machine. For authenticating to Azure using Azure Active Directory, please see the blog.
    $cred = Get-AutomationPSCredential -Name "<AutomationCredentialAssetName>"
    Add-AzureAccount -Credential $cred
    Select-AzureSubscription -subscriptionName "Visual Studio Ultimate with MSDN"
       
    Restart-AzureVM -ServiceName $AlertContext.resourceName -Name $AlertContext.resourceName
}
else 
{
    Write-Error "This runbook is meant to only be started from a webhook." 
}


When the runbook runs, you can see the alertcontext as the runbook output.

New to Automation?

Are there solutions you want to see, features you want added, or do you have feedback on existing features?  Request them on Azure Automation UserVoice.

Just getting started with Azure Automation?  Learn about the service here, and follow Azure Automation on Twitter.

Happy automating!