Live wire

Microsoft Says Months-Old Bug Caused Global 365 Outage

A July update to gateway servers hid a defect that load tests missed; service was restored on 13 November, Microsoft says.

Microsoft says a previously undetected software bug in an update to its Microsoft 365 networking gateway servers caused the global outage of 12 and 13 November, which hit Outlook, Exchange Online, Teams and Copilot, according to its preliminary report. The update went live in July and ran for more than three months without visible problems. A final report was expected later in the week.

How the bug was missed and triggered

The defect could overwhelm servers under certain traffic conditions and make them stop responding. Microsoft says its load tests ran on less powerful hardware with fewer processor cores than production, and that the defect is worse on machines with more cores and newer hardware. It cites two contributing factors: about 15 hours before the outage Teams automatically moved to the networking infrastructure used by Exchange Online, which raised traffic on those servers, and about 14 per cent of gateway server capacity was already offline from an unrelated issue. Whether an earlier Teams outage that day was related is unclear.

Timeline and recovery

The outage began at 11.20 am US Eastern time on 12 November and ended at 11.10 am on 13 November, when Microsoft confirmed services were fully restored. Traffic redirected from failing servers overwhelmed the others, the defect appeared across the fleet in under two hours, and at the worst point more than 80 per cent of servers were unresponsive. Safeguards meant to contain problems between regions failed, so the disruption spread worldwide. Teams, OneDrive and SharePoint could switch to other infrastructure and recovered sooner, but Exchange Online could not. Restarting servers caused them to be swamped by backed-up traffic again until engineers deployed a fix.

Effects and promises

In India, SpiceJet cancelled three Delhi flights and delayed two on 13 November, and IndiGo, Akasa Air and Air India reported problems with email support; Singapore's government email also went down. Microsoft says it has added capacity, improved monitoring and put in safeguards to stop traffic redirection from causing similar cascading failures. An executive apologised on social media. Microsoft earlier said it saw no sign of malicious activity, and this account gives no sign of an attack either.

Topics