Advancing Azure Virtual Machine availability transparency | Azure Blog and Updates

“All through our Advancing Reliability weblog sequence we’ve defined varied strategies utilized by the Azure platform to forestall technical points from impacting prospects’ digital machines (VMs) or different sources—like host resiliency with Mission Tardigrade, cautious secure deployment practices profiting from ML-based AIOps insights, in addition to predicting and mitigating hardware failures with Mission Narya. Regardless of these efforts, when working on the scale of Azure we all know that there’ll inevitably be some failures that influence buyer sources—so once they do, we try for transparency in how we talk to impacted prospects. So for right now’s publish within the sequence, I’ve requested Principal Software program Engineering Supervisor Nick Swanson to spotlight current enhancements within the house—particularly, how we floor extra detailed root trigger statements by way of Azure useful resource well being.”— Mark Russinovich, CTO, Azure

The prevailing Azure useful resource well being function lets you diagnose and get help for service issues that have an effect on your Azure sources. It experiences on the present and previous well being of your sources, exhibiting any time ranges that every of your sources have been unavailable. However we all know that our prospects and companions are significantly fascinated by “the why” to grasp what brought about the underlying technical challenge, and in enhancing how they will obtain communications about any points—to feed into monitoring processes, to elucidate hiccups to different stakeholders, and in the end to tell enterprise selections.

Introducing root causes for VM points—in Azure useful resource well being

We not too long ago shipped an enchancment to the useful resource well being expertise that may improve the knowledge we share with prospects about VM failures, with further context on the foundation trigger that led to the problem. Now, along with getting a quick notification when a VM’s availability is impacted, prospects can count on a root trigger to be added at a later level as soon as our automated Root Trigger Evaluation (RCA) system identifies the failing Azure platform element that led to the VM failure. Let’s stroll via an instance to see how this works in apply:

1. At time T1, a server rack goes offline as a result of a networking challenge, inflicting VMs on the rack to lose connectivity. (Latest reliability enhancements associated to community structure shall be shared in a future Advancing Reliability weblog publish—watch this house!)
2. At time T2, Azure’s inner monitoring acknowledges that it’s unable to achieve VMs on the rack and begins to mitigate by redeploying the impacted VMs to a brand new rack. Throughout this time, an annotation is distributed to useful resource well being notifying prospects that their VM is at the moment impacted and unavailable.

    

   
   
   

   Determine 1: A screenshot of the Azure portal “useful resource well being” blade exhibiting the well being historical past of a useful resource.

3. At time T3, platform telemetry from the highest of rack swap, the host machine, and inner monitoring techniques, are all correlated collectively in our RCA engine to derive the foundation explanation for the failure. As soon as computed, the RCA is then printed again into useful resource well being together with related architectural resiliency suggestions that prospects can implement to attenuate the chance of influence sooner or later.

Determine 2: A screenshot of the Azure portal “well being historical past” blade exhibiting root trigger particulars for an instance of a VM challenge.

Whereas the preliminary downtime notification performance has existed for a number of years, the publishing of a root trigger assertion is a brand new addition. Now, let’s dive into the small print of how we derive these root causes.

Root Trigger Evaluation engine

Let’s take a more in-depth take a look at the prior instance and stroll via the small print of how the RCA engine works and the know-how behind it. On the core of our RCA engine for VMs is Azure Information Explorer (ADX), a giant knowledge service optimized for prime quantity log telemetry analytics. Azure Information Explorer permits the power to simply parse via terabytes of log telemetry from gadgets and companies that comprise the Azure platform, be part of them collectively, and interpret the correlated info streams to derive a root trigger for various failure situations. This finally ends up being a multistep knowledge engineering course of:

Part 1: Detecting downtime

The primary section in root trigger evaluation is to outline the set off beneath which the evaluation is executed. Within the case of Digital Machines, we need to decide root causes each time a VM unexpectedly reboots, so the set off is a VM transitioning from an up state to a down state. Figuring out these transitions from platform telemetry is simple in most situations, however extra difficult round sure sorts of infrastructure failure the place platform telemetry may get misplaced as a result of gadget failure or energy loss. To deal with these courses of failures, different strategies are required—like monitoring knowledge loss as a potential indication of a VM well being transition. Azure Information Explorer excels at the moment of sequence evaluation, and a extra detailed take a look at strategies round this may be discovered within the Microsoft Tech Neighborhood: Calculating downtime utilizing Window capabilities and Time Collection capabilities in Azure Information Explorer.

Part 2: Correlation evaluation

As soon as a set off occasion is outlined (on this case, a VM transitioning to an unhealthy state) the subsequent section is correlation evaluation. On this step we use the presence of the set off occasion to correlate telemetry from factors throughout the Azure platform, like:

• Azure host: the bodily blade internet hosting VMs.
• TOR: the highest of rack community swap.
• Azure Storage: the service which hosts Digital Disks for Azure Digital Machines.

Every of those techniques has their very own telemetry feeds that have to get parsed and correlated with the VM downtime set off occasion. That is executed via understanding the dependency graph for a VM and the underlying techniques that may trigger a VM to fail, after which becoming a member of all these dependent techniques’ well being telemetry collectively, filtered on occasions which can be comparatively near the VM transition in time. Azure Information Explorer’s intuitive and highly effective question language helps with this, with documented patterns like time window be part of for correlating temporal telemetry streams collectively. On the finish of this correlation course of, we have now a dataset that represents VM downtime transitions with correlated platform telemetry from all of the dependent techniques that would trigger or may have info helpful in figuring out what led to the VM failure.

Part three: Root trigger attribution

The subsequent step within the course of is attribution. Now that we’ve collected all of the related knowledge collectively in a single dataset, attribution guidelines get utilized to interpret the knowledge and translate it right into a customer-facing root trigger assertion. Going again to our unique instance of a TOR failure, after our correlation evaluation we would have many attention-grabbing items of data to interpret. For instance, techniques monitoring the Azure hosts may need logs indicating they misplaced connectivity to the hosts throughout this time. We would even have indicators associated to digital disk connectivity issues, and specific indicators from the TOR gadget concerning the failure. All these items of data are actually scanned over, and the express TOR failure sign is prioritized over the opposite indicators as the foundation trigger. This prioritization course of, and the foundations behind it, are constructed with area consultants and modified because the Azure platform evolves. Machine studying and anomaly detection mechanisms sit on high of those attributed root causes, to assist establish alternatives to enhance these classification guidelines in addition to to detect sample modifications within the price of those failures to feed again into secure deployment pipelines.

Part four: RCA publishing

The final step is publishing root causes to Azure useful resource well being, the place they turn into seen to prospects. That is executed in a quite simple Azure Capabilities utility, which periodically queries the processed root trigger knowledge in Azure Information Explorer, and emits the outcomes to the useful resource well being backend. As a result of info streams can are available with varied knowledge delays, RCAs can often be up to date on this course of to replicate higher sources of data having arrived resulting in a extra particular root trigger that what was initially printed.

Going ahead

Figuring out and speaking to our prospects and companions the foundation explanation for any points impacting them, is only the start. Our prospects might have to take these RCAs and share them with their prospects and coworkers. We need to construct on the work right here to make it simpler to establish and monitor useful resource RCAs, in addition to simply share them out. With a purpose to accomplish that, we’re engaged on backend modifications to generate distinctive per-resource and per-downtime monitoring IDs that we will expose to you, to be able to simply match downtimes to their RCAs. We’re additionally engaged on new options to make it simpler to e mail RCAs out, and finally subscribe to RCAs to your VMs. It will make it potential to enroll in RCAs straight in your inbox after an unavailability occasion with no further motion wanted in your half.