outages
Latest stories covering outages, with summaries and links to related coverage.
2 articles · page 1 of 1
Connected topics
The August 17 outage, and the work ahead
news
4 min read
GitHub suffered a 7-hour, 47-minute outage on August 17 that disrupted github.com, authentication, GitHub Actions, APIs, pull requests, issues, and Copilot worldwide. GitHub said traffic had reached a new peak when a critical infrastructure component in its Central US data center failed to scale, creating capacity pressure that spread across its systems. Most services recovered that day after teams rerouted traffic and isolated affected infrastructure, while some Copilot services took longer because client-side retry loops increased traffic during recovery. The outage was GitHub’s second significant August incident, following an August 6 GitHub Actions failure. GitHub said neither event resulted from a code or configuration change; both were capacity failures. Monthly commits grew from 1.4 billion in April to 2.9 billion, increasing system demand. Since April, GitHub has added more than 3 million CPU cores, 120 petabytes of high-speed storage, and additional network capacity, while accelerating its migration to Azure. Azure now handles about 58% of GitHub platform load and half of Git operations, up from 12% of platform load in May. GitHub plans to limit retries and apply retry budgets and variable timeouts between services, review lower-priority CPU and memory alerts, isolate critical systems, and remove shared dependencies. It is also developing an architecture intended to scale read capacity linearly for large monorepos.
After outages, Amazon to make senior engineers sign off on AI-assisted changes
news
1 min read
Amazon's ecommerce division will require senior engineers to approve AI-assisted changes following a series of outages linked to these tools. The company identified a trend of incidents with a high blast radius associated with novel GenAI usage.