Forward Azure Automation diagnostic logs to Azure Monitor

Azure Automation can send runbook job status and job streams to your Log Analytics workspace. This process does not involve workspace linking and is completely independent and allows you to perform simple investigations. Job logs and job streams are visible in the Azure portal, or with PowerShell for individual jobs. With Azure Monitor logs for your Automation account, you can:

  • Get insights into the status of your Automation jobs.
  • Trigger an email or alert based on your runbook job status (for example, failed or suspended).
  • Write advanced queries across your job streams.
  • Correlate jobs across Automation accounts.
  • Use customized views and search queries to visualize your runbook results, runbook job status, and other related key indicators or metrics through an Azure dashboard.
  • Get the audit logs related to Automation accounts, runbooks, and other asset create, modify and delete operations.

Using Azure Monitor logs, you can consolidate logs from different resources in the same workspace where it can be analyzed with queries to quickly retrieve, consolidate, and analyze the collected data. You can create and test queries using Log Analytics in the Azure portal and then either directly analyze the data using these tools or save queries for use with visualization or alert rules.

Azure Monitor uses a version of the Kusto query language (KQL) used by Azure Data Explorer that is suitable for simple log queries. It also includes advanced functionality such as aggregations, joins, and smart analytics. You can quickly learn the query language using multiple lessons.

Azure Automation diagnostic settings

You can forward the following platform logs and metric data using Automation diagnostic settings support:

Data types Description
Job Logs Status of the runbook job in the Automation account.
Job Streams Status of the job streams in the runbook defined in the Automation account.
DSCNodeStatus Status of the DSC node.
AuditEvent All resource logs that record customer interactions with data or the settings of the Azure Automation service.
Metrics Total jobs, total update, deployment machine runs, total update deployment runs.

Configure diagnostic settings in Azure portal

You can configure diagnostic settings in the Azure portal from the menu for the Automation account resource.

  1. In the Automation account menu, under Monitoring select Diagnostic settings.

    Screenshot showing selection of diagnostic setting option.

  2. Click Add diagnostic setting.

    Screenshot showing selection of add diagnostic setting.

  3. Enter a setting name in the Diagnostic setting name if it doesn't already have one.

    You can also view all categories of Logs and metrics.

    Screenshot showing all categories of logs and metrics.

    • Logs and metrics to route : For logs, choose a category group or select the individual checkboxes for each category of data you want to send to the destinations specified. Choose AllMetrics if you want to store metrics into Azure Monitor logs.

    • Destination details : Select the checkbox for each destination. As per the selection of each box, the options appear to allow you to add additional information.

      Screenshot showing selections in destination details section.

      • Log Analytics : Enter the Subscription ID and workspace name. If you don't have a workspace, you must create one before proceeding.

      • Event Hubs: Specify the following criteria:

        • Subscription: The same subscription as that of the Event Hub.
        • Event Hub namespace: Create Event Hub if you don't have one yet.
        • Event Hub name (optional): If you don't specify a name, an event hub is created for each log category. If you are sending multiple categories, specify a name to limit the number of Event Hubs created. See Azure Event Hubs quotas and limits for details.
        • Event Hub policy (optional): A policy defines the permissions that the streaming mechanism has. See Event Hubs feature.
      • Storage: Choose the subscription, storage account, and retention policy. Screenshot showing the storage account.

      • Partner integration: You must first install a partner integration into your subscription. Configuration options will vary by partner. For more information, see Azure Monitor integration.

  4. Click Save.

After a few moments, the new setting appears in your list of settings for this resource, and logs are streamed to the specified destinations as new event data is generated. There can be 15 minutes time difference between the event emitted and its appearance in Log Analytics workspace.

Query the logs

To query the generated logs:

  1. In your Automation account, under Monitoring, select Logs.

  2. Under All Queries, select Automation Jobs.

    Screenshot showing how to navigate to select Automation jobs.

  3. Select one of the queries you want to execute and click Run.

  4. To execute a custom query, close the Queries window and paste your custom query in the new query window and click Run.

    The output of the query is displayed in Results pane.

  5. Click New alert rule to configure an Azure Monitor alert for this query.

    Screenshot showing how to query logs.

Azure Monitor log records

Azure Automation diagnostics create the following types of records in Azure Monitor logs, tagged as AzureDiagnostics. The tables in the below sections are examples of records that Azure Automation generates and the data types that appear in log search results.

Job logs

Property Description
TimeGenerated Date and time when the runbook job executed.
RunbookName_s Name/names of the runbook.
Caller_s Caller that initiated the operation. Possible values are either an email address or system for scheduled jobs.
Tenant_g GUID (globally unique identifier) that identifies the tenant for the caller.
JobId_g GUID that identifies the runbook job.
ResultType Status of the runbook job. Possible values are:
- New
- Created
- Started
- Stopped
- Suspended
- Failed
- Completed
Category Classification of the type of data. For Automation, the value is JobLogs.
OperationName Type of operation performed in Azure. For Automation, the value is Job.
Resource Name of the Automation account
SourceSystem System that Azure Monitor logs use to collect the data. The value is always Azure for Azure diagnostics.
ResultDescription Runbook job result state. Possible values are:
- Job is started
- Job Failed
- Job Completed
CorrelationId Correlation GUID of the runbook job.
ResourceId Azure Automation account resource ID of the runbook.
SubscriptionId Azure subscription GUID for the Automation account.
ResourceGroup Name of the resource group for the Automation account.
ResourceProvider Name of the resource provider. The value is MICROSOFT.AUTOMATION.
ResourceType Resource type. The value is AUTOMATIONACCOUNTS.

Job streams

Property Description
TimeGenerated Date and time when the runbook job was executed.
RunbookName_s Name of the runbook.
Caller_s Caller that initiated the operation. Possible values are either an email address or system for scheduled jobs.
StreamType_s Type of job stream. Possible values are:
-Progress
- Output
- Warning
- Error
- Debug
- Verbose
Tenant_g GUID that identifies the tenant for the caller.
JobId_g GUID that identifies the runbook job.
ResultType The status of the runbook job. Possible values are:
- In Progress
Category Classification of the type of data. For Automation, the value is JobStreams.
OperationName Type of operation performed in Azure. For Automation, the value is Job.
Resource Name of the Automation account.
SourceSystem System that Azure Monitor logs use to collect the data. The value is always Azure for Azure diagnostics.
ResultDescription Description that includes the output stream from the runbook.
CorrelationId Correlation GUID of the runbook job.
ResourceId Azure Automation account resource ID of the runbook.
SubscriptionId Azure subscription GUID for the Automation account.
ResourceGroup Name of the resource group for the Automation account.
ResourceProvider Resource provider. The value is MICROSOFT.AUTOMATION.
ResourceType Resource type. The value is AUTOMATIONACCOUNTS.

Audit events

Property Description
TenantID GUID that identifies the tenant for the caller.
TimeGenerated (UTC) Date and time when the runbook job is executed.
Category AuditEvent
ResourceGroup Resource group name of the Automation account.
Subscription Id Azure subscription GUID for the Automation account.
ResourceProvider MICROSOFT.AUTOMATION
Resource Automation Account name
ResourceType AUTOMATIONACCOUNTS
OperationName Possible values are Update, Create, Delete.
ResultType Status of the runbook job. Possible value is: Completed.
CorrelationId Correlation GUID of the runbook job.
ResultDescription Runbook job result state. Possible values are Update, Create, Delete.
Tenant_g GUID that identifies the tenant for the caller.
SourceSystem System that Azures Monitor logs use to collect the data. The value is always Azure for Azure diagnostics.
clientInfo_IpAddress_s {scrubbed}
clientInfo_PrincipalName_s {scrubbed}
clientInfo_TenantId_g Tenant ID of the client.
clientInfo_Issuer_s
clientInfo_ObjectId_g Object ID of the client.
clientInfo_AppId_g AppID of the client.
clientInfo_ClientRequestId_g RequestID of the client
targetResources_Resource_s Account, Job, Credential, Connections, Variables, Runbook.
Type AzureDiagnostics
_ResourceId Azure Automation account resource ID of the runbook.

View Automation logs in Azure Monitor logs

Now that you started sending your Automation job streams and logs to Azure Monitor logs, let's see what you can do with these logs inside Azure Monitor logs.

To see the logs, run the following query: kusto AzureDiagnostics | where ResourceProvider == "MICROSOFT.AUTOMATION"

Sample queries for job logs and job streams

Find all jobs that are completed with error

In addition to scenarios like alerting on failures, you can find when a runbook job has a non-terminating error. In these cases, PowerShell produces an error stream, but the non-terminating errors don't cause your job to suspend or fail.

  1. In your Log Analytics workspace, click Logs.
  2. In the query field, type:
    AzureDiagnostics | where ResourceProvider == "MICROSOFT.AUTOMATION" and Category == "JobStreams" and StreamType_s == "Error" | summarize AggregatedValue = count () by JobId_g. 
    
  3. Click Search.

View job streams for a job

When you're debugging a job, you might also want to look into the job streams. The following query shows all the streams for a single job with GUID 2ebd22ea-e05e-4eb9-9d76-d73cbd4356e0:

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.AUTOMATION" and Category == "JobStreams" and JobId_g == "2ebd22ea-e05e-4eb9-9d76-d73cbd4356e0"
| sort by TimeGenerated asc
| project ResultDescription

View historical job status

Finally, you might want to visualize your job history over time. You can use this query to search for the status of your jobs over time.

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.AUTOMATION" and Category == "JobLogs" and ResultType != "started"
| summarize AggregatedValue = count() by ResultType, bin(TimeGenerated, 1h)

Log Analytics Historical Job Status Chart

Find logs reporting errors in the automation jobs.

AzureDiagnostics 
| where ResourceProvider == "MICROSOFT.AUTOMATION" 
| where StreamType_s == "Error" 
| project TimeGenerated, Category, JobId_g, OperationName, RunbookName_s, ResultDescription, _ResourceId 

Find Azure Automation jobs that are completed

AzureDiagnostics 
| where ResourceProvider == "MICROSOFT.AUTOMATION" and Category == "JobLogs" and ResultType == "Completed" 
| project TimeGenerated, RunbookName_s, ResultType, _ResourceId, JobId_g

Find Azure Automation jobs that are failed, suspended, or stopped

AzureDiagnostics 
| where ResourceProvider == "MICROSOFT.AUTOMATION" and Category == "JobLogs" and (ResultType == "Failed" or ResultType == "Stopped" or ResultType == "Suspended") 
| project TimeGenerated, RunbookName_s, ResultType, _ResourceId, JobId_g 

List all runbooks & jobs that completed successfully with errors

AzureDiagnostics 
| where ResourceProvider == "MICROSOFT.AUTOMATION" and Category == "JobStreams" and StreamType_s == "Error" 
| project TimeGenerated, RunbookName_s, StreamType_s, _ResourceId, ResultDescription, JobId_g 

Send an email when a runbook job fails or suspends

The following steps explain how to set up email alerts in Azure Monitor to notify when something goes wrong with a runbook job.

To create an alert rule, create a log search for the runbook job records that should invoke the alert as described in Query the logs. Click the +New alert rule to configure the alert rule.

  1. In your Automation account, under Monitoring, select Logs.

  2. Create a log search query for your alert by entering a search criteria into the query field.

    AzureDiagnostics | where ResourceProvider == "MICROSOFT.AUTOMATION" and Category == "JobLogs" and (ResultType == "Failed" or ResultType == "Suspended")   
    

    You can also group by the runbook name by using:

    AzureDiagnostics | where ResourceProvider == "MICROSOFT.AUTOMATION" and     Category == "JobLogs" and (ResultType == "Failed" or ResultType == "Suspended") | summarize AggregatedValue = count() by RunbookName_s 
    
  3. To open the Create alert rule screen, click +New alert rule on the top of the page. For more information on the options to configure the alerts, see Log alerts in Azure

Azure Automation diagnostic audit logs

You can now send audit logs also to the Azure Monitor workspace. This allows enterprises to monitor key automation account activities for security & compliance. When enabled through the Azure Diagnostics settings, you will be able to collect telemetry about create, update and delete operations for the Automation runbooks, jobs and automation assets like connection, credential, variable & certificate. You can also configure the alerts for audit log conditions as part of your security monitoring requirements.

Difference between activity logs and audit logs

Activity log is a platform login Azure that provides insight into subscription-level events. The activity log for Automation account includes information about when an automation resource is modified or created or deleted. However, it does not capture the name or ID of the resource.

Audit logs for Automation accounts capture the name and ID of the resource such as automation variable, credential, connection and so on, along with the type of the operation performed for the resource and Azure Automation would scrub some details like client IP data conforming to the GDPR compliance.

Activity logs would show details such as client IP because an Activity log is a platform log that provides detailed diagnostic and auditing information for Azure resources. They are automatically generated for activities that occur in ARM and gets pushed to the activity log resource provider. Since Activity logs are part of Azure monitoring, it would show some client data to provide insights into the client activity.  

Sample queries for audit logs

Query to view Automation resource audit logs

AzureDiagnostics 
| where ResourceProvider == "MICROSOFT.AUTOMATION" and Category == "AuditEvent" 

Query to Monitor any variable update, create or delete operation

AzureDiagnostics 
| where ResourceProvider == "MICROSOFT.AUTOMATION" and Category == "AuditEvent" and targetResources_Resource_s == "Variable" 

Query to Monitor any runbook operation like create, draft or update

AzureDiagnostics 
| where ResourceProvider == "MICROSOFT.AUTOMATION" and Category == "AuditEvent" and targetResources_Resource_s contains "Runbook" 

Query to Monitor any certificate creation, updating or deletion

AzureDiagnostics 
| where ResourceProvider == "MICROSOFT.AUTOMATION" and Category == "AuditEvent" and targetResources_Resource_s contains "Certificate" 

Query to Monitor any credentials creation, updating or deletion

AzureDiagnostics 
| where ResourceProvider == "MICROSOFT.AUTOMATION" and Category == "AuditEvent" and targetResources_Resource_s contains "Credential" 

Filter job status output converted into a JSON object

Recently we changed the behavior of how the Automation log data is written to the AzureDiagnostics table in the Log Analytics service, where it no longer breaks down the JSON properties into separate fields. If you configured your runbook to format objects in the output stream in JSON format as separate columns, it is necessary to reconfigure your queries to parse that field to a JSON object to access those properties. This is accomplished using parse json to access a specific JSON element in a known path.

For example, a runbook formats the ResultDescription property in the output stream in JSON format with multiple fields. To search for the status of your jobs that are in a failed state as specified in a field called Status, use this example query to search the ResultDescription with a status of Failed:

AzureDiagnostics
| where Category == 'JobStreams'
| extend jsonResourceDescription = parse_json(ResultDescription)
| where jsonResourceDescription.Status == 'Failed'

Log Analytics Historical Job Stream JSON format

Next steps