Skip Navigation

Solution architecture: Anomaly detection with machine learning

Microsoft Azure’s IT Anomaly Insights can help automate and scale anomaly detection for IT departments to quickly detect and fix issues.

This solution is built on the Azure managed services: Event Hubs, Stream Analytics, Storage, Data Factory, Azure SQL Database, Machine Learning Studio, Service Bus, Application Insights and Power BI. These services run in a high-availability environment, patched and supported, allowing you to focus on your solution instead of the environment they run in.

Anomaly detection with machine learningMicrosoft Azure’s IT Anomaly Insights can help automate and scale anomaly detection for IT departments to quickly detect and fix issues.Machine Learning(Anomaly Detection)Service Bus topics(Publish/subscribe capabilities)Visual Studio Application Insights(Monitoring and telemetry)Event Hub(Event queue)Table Storage(Big Data store)Stream Analytics(Realtime analytics)MetadataSave ML outputScore each datasetPublish anomalies detectedPower BI Azure SQL DB(Anomaly detection results)Data FactoryTime series data

Implementation guidance

Products/Description Documentation

Event Hubs

This is the entry point of the pipeline, where the raw timeseries data is ingested.

Stream Analytics

Stream Analytics performs aggregation at 5-minute intervals, and aggregates raw data points by metric name.

Storage

Azure Storage stores data aggregated by the Stream Analytics job.

Data Factory

Data Factory calls the Anomaly Detection API at regular intervals (every 15 minutes by default) on the data in Azure Storage. It stores the results in a SQL database.

Azure SQL Database

SQL Database stores the results from the Anomaly Detection API, including binary detections and detection scores. It also stores optional metadata sent with the raw data points to allow for more complicated reporting.

Machine Learning Studio

This hosts the Anomaly Detection API. Note that the API itself is stateless and requires historical data points to be sent in each API call.

Service Bus

Detected anomalies are published to a service bus topic to enable consumption by external monitoring services.

Application Insights

Application Insights allows for monitoring of the pipeline.

Power BI

Power BI provides dashboards showing the raw data, as well as detected anomalies.

Related solution architectures