We are announcing the much-awaited public preview of diagnostic logs for Azure Stream Analytics through integrations with Azure Monitoring. You can now examine late or malformed data that causes unexpected behaviors. This helps remediate errors caused by data that does not conform to the expectations of the query.
Diagnostic logs provide rich insights into all operations associated with a streaming job. They are turned off by default and can be enabled in the “Diagnostic logs” blade under “Monitoring”. These are different from Activity logs that are always enabled and provide details on management operations performed.
Examples of data handling errors that diagnostic logs can help with include:
- Data conversion and serialization errors in cases of schema mismatch.
- Incompatible types including constraints such as allow null and duplicates.
- Truncation of strings and issues with precision during conversion.
- Expression evaluation errors such as divide by zero, overflow etc.
An example of non-conforming data being written to Azure storage is illustrated below:
{
Diagnostic:"Encountered error trying to write 3 events: …",
Timestamp:"7/25/2015 12:27:44Z",
Source:"Output1",
Output:"Output1",
Error:
{
Type:"System.InvalidOperationException",
Description:"The given value “hello world” of type string from the data source cannot be converted to type decimal of the specified target column [Amount].",
},
EventData:
{
SomeValue:”hello world”,
Count:1
}
}
Errors are sampled by error type and source as shown above.
Immediate access to the actual data that causes errors enables you to either quickly remediate problems or ignore the non-conforming data to make progress.
Persisting event data and operational metadata (such as occurrence time and occurrence count) in an Azure Storage artifacts enables easier diagnosis and faster troubleshooting of issues. This data can also be analyzed offline using Azure Log Analytics. Routing this data to EventHub makes it possible to set up a Stream Analytics job to monitor another Stream Analytics job!
It should be noted that the usage of services such as Azure Storage, EventHub, and Log Analytics for analyzing non-conforming data will be charged based on the pricing model for those services.
We are excited for you to try it out our diagnostic logs. Detailed steps on using this capability can be found in the documentation page.