longquanwo
2019-11-20 20:46:37 +08:00
Summary of Impact: Between 00:57 UTC and 03:40 UTC, 20 November 2019, customers and services utilizing Azure Front Door (AFD) services were impacted by an infrastructure service failure. This resulted in a loss of connectivity to multiple services reliant on AFD. Azure Front Door services provide Edge caching and network entry point services to the Microsoft global network. This issue impacted a large percentage of Microsoft Services, though not all services were impacted. Many impacted services were able to initiate fail over from the AFD platform, providing immediate mitigation to their customers.
Preliminary Root Cause: During a recent periodic deployment, initial safety checks did not detect issue and prevent the roll out. Monitoring detected immediately once service failure was experienced. Engineers were alerted immediately, and immediately initiated roll back.
Mitigation: Engineers immediately initiated deployment rollback processes to correct the underlying Azure Front Door Issue, this was completed at 02:40 UTC 20 November 2019. Impacted services began recovering at that time.
A small subset of services may still be experiencing residual impact; customers will be communicated directly via Azure Service Health for those services.
A detailed root cause analysis will be published within 72 hours.