{"id":169,"date":"2021-06-30T00:00:00","date_gmt":"2021-06-30T00:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/blog\/advancing-safe-deployment-with-aiops-introducing-gandalf"},"modified":"2025-06-25T04:59:34","modified_gmt":"2025-06-25T11:59:34","slug":"advancing-safe-deployment-with-aiops-introducing-gandalf","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/advancing-safe-deployment-with-aiops-introducing-gandalf\/","title":{"rendered":"Advancing safe deployment with AIOps\u2014introducing Gandalf"},"content":{"rendered":"\n
\n

Changes to Azure services and the Azure platform itself are both inevitable and beneficial, to ensure continuous delivery of updates, new features, and security enhancements. However, change is also a primary cause of service regressions that can contribute towards reliability issues\u2014for hyperscale cloud providers, indeed for any IT service provider. As such, it is critical to catch any such problems as early as possible during the development and deployment rollout, to minimize any impact on the customer experience. As part of our ongoing Advancing Reliability<\/a> blog series, today I\u2019ve asked Principal Program Manager Jian Zhang<\/strong> from our AIOps team to introduce how we\u2019re increasingly leveraging machine learning to de-risk these changes, ultimately to improve the reliability of Azure.\u201d<\/em>\u2014Mark Russinovich, CTO, Azure<\/p>\n<\/blockquote>\n\n\n\n

\n\n\n\n

This post includes contributions from Principal Data Scientists Ken Hsieh and Ze Li, Principal Data Scientist Manager Yingnong Dang, and Partner Group Software Engineering Manager Murali Chintalapati.<\/em><\/p>\n\n\n\n

In our earlier blog post \u201cAdvancing safe deployment practices<\/a>\u201d Cristina del Amo Casado described how we release changes to production, for both code and configuration changes, across the Azure<\/a> platform. The processes consist of delivering changes progressively, with phases that incorporate enough bake time to allow detection at a small scale for most regressions missed during testing.<\/p>\n\n\n\n

The continuous monitoring of health metrics is a fundamental part of this process, and this is where AIOps plays a critical role\u2014it allows the detection of anomalies to trigger alerts and the automation of correcting actions such as stopping the deployment or initiating rollbacks.<\/p>\n\n\n\n

In the post that follows, we introduce how AI and machine learning are used to empower DevOps engineers, monitor the Azure deployment process at scale, detect issues early, and make rollout or rollback decisions based on impact scope and severity.<\/p>\n\n\n\n

Why AIOps for safe deployment<\/h2>\n\n\n\n

As defined by Gartner<\/a>, AIOps enhances IT operations through insights that combine big data, machine learning, and visualization to automate IT operations processes, including event correlation, anomaly detection, and causality determination. In our earlier post, “Advancing Azure service quality with artificial intelligence: AIOps<\/a>,” we shared our vision and some of the ways in which we are already using AIOps in practice, including around safe deployment. AIOps is well suited to catching failures during deployment rollout, particularly because of the complexities of cross-service dependencies, the scale of hyperscale cloud services, and the variety of different customer scenarios supported.<\/p>\n\n\n\n

Phased rollouts and enriched health signals are used to facilitate monitoring and decision making in the deployment process, but the volume of signals and level of complexity involved in deployment decision making exceeds what any human could reasonably reason over, across thousands of ever-evolving service components, spanning more than 200 datacenters in more than 60 regions. Some latent issues won\u2019t manifest for several days after their deployment, and global issues that span different clusters but manifest only minutely in any individual cluster are hard to detect with just a local watchdog. While loose coupling allows most service components to be deployed independently, their deployments could have intricate impacts on each other. For example, a simple change in an upstream service could potentially impact a downstream service if it breaks the contract of API calls between the two services.<\/p>\n\n\n\n

These challenges call for automated monitoring, anomaly detection, and rollout impact assessment solutions to facilitate deployment decisions at velocity.<\/p>\n\n\n\n

\"Gandalf<\/figure>\n\n\n\n

Figure 1: Gandalf safe deployment<\/em><\/p>\n\n\n\n

Gandalf safe deployment service: An AIOps solution<\/h2>\n\n\n\n

Rising to the challenge described above, the Azure Compute Insights team developed the \u201cGandalf\u201d safe deployment service\u2014an end-to-end, continuous monitoring system for safe deployment. We consider this part of the Gandalf AIOps solution suite, which includes a few other intelligent monitoring services. The code name Gandalf was inspired by the protagonist from The Lord of the Rings, as shown in Figure 1, it serves as a global watchdog, which makes intelligent deployment decisions based on signals collected. It works in tandem with local watchdogs, safe deployment policies, and pre-qualification tests, all to ensure deployment safety and velocity.<\/p>\n\n\n\n

As illustrated in Figure 2, the Gandalf system monitors rich and representative signals from Azure, performs anomaly detection and correlation, then derives insights to support deployment decision making and actions.<\/p>\n\n\n\n

\"Gandalf<\/figure>\n\n\n\n

Figure 2: Gandalf system overview<\/em><\/p>\n\n\n\n

Data sources<\/h3>\n\n\n\n

Gandalf monitors signals across performance, failures, and events as described below. It pre-processes the data to structure them around a unified data schema to support downstream data analytics. It also leverages a few other analytics services within Azure for health signals, including our Virtual Machine failure categorization service and near real-time failure attribution processing service. Signal registration with Gandalf is required when any new service components are onboarded, to ensure complete coverage.<\/p>\n\n\n\n