Data Integration solutions can be complex with many moving parts involving complex data factories with multiple pipelines. Monitoring provides data to ensure that your data factory pipelines stay up and running in a healthy state. It also helps you to stave off potential problems or troubleshoot past ones. In addition, you can use monitoring data to gain deep insights about your application. This knowledge can help you to improve application performance or maintainability, or automate actions that would otherwise require manual intervention.
Azure Data Factory (ADF) integration with Azure Monitor allows you to route your data factory metrics to Operations and Management (OMS) Suite.
Now, you can monitor the health of your data factory pipelines using ‘Azure Data Factory Analytics’ OMS service pack available in Azure marketplace.
Azure Data Factory OMS pack provides you a summary of overall health of your Data Factory, with options to drill into details and to troubleshoot unexpected behavior patterns. With rich, out of the box views you can get insights into key processing including:
- At a glance summary of data factory pipeline, activity and trigger runs
- Ability to drill into data factory activity runs by type
- Summary of data factory top pipeline, activity errors
You can also dig deeper into each of the pre-canned view, look at the Log Analytics query, edit it as per your requirement. You can also raise alerts via OMS.
You can route metrics from different data factories to the same OMS account and do monitoring across data factories as well. Simply enable the diagnostics settings for your data factory and route data to your Log Analytics workspace. Get more information and detailed steps on enabling the Azure Data Factory OMS service pack.
Our goal is to continue adding features and improve the usability of Data Factory tools. Get started building pipelines easily and quickly using Azure Data Factory. If you have any feature requests or want to provide feedback, please visit the Azure Data Factory forum.