GitHub’s August 17 outage: how retries delayed recovery for hours

From 13:28 to 21:15 UTC on August 17, GitHub experienced an outage lasting 7 hours and 47 minutes. According to the final GitHub Status report, roughly 20% of web and API requests failed at the peak, while archive and raw-content downloads reached an error rate of approximately 50%. Issues, pull requests, Actions and Copilot were affected, along with several enterprise authentication and user-synchronisation paths.

This was not a bad software release. In GitHub’s subsequent account, the company explicitly said that no code or configuration change triggered the incident. Traffic reached a new peak and a critical component in the Central US data centre did not scale with it. Recovery was then prolonged not only by the original shortage, but by systems and clients trying to obtain an answer again.

The first concurrency ceiling was reached by an Istio sidecar, a small network proxy that runs beside the host service and carries its inbound and outbound requests. The service itself could still appear healthy. The autoscaling policy observed the host process rather than the sidecar limit, so it did not add instances when the network layer was already losing capacity.

Pressure from one constrained proxy then propagated. Four HAProxy nodes exhausted their flow limits — the number of network flows a load balancer can track and serve at once. The gateway authentication path degraded, spreading sign-in latency and failures to services that were not the original point of saturation.

That sequence matters more than the word “autoscaling.” Extra servers help only when the scaling signal covers the actual bottleneck. Host utilisation does not replace separate measurements for queues, sidecar concurrency, load-balancer flows and the share of traffic created by retries.

GitHub’s internal gateways used optimistic retry logic: when a response was delayed or missing, another attempt could hide a brief random failure from the user. Once a dependency is already saturated, however, each retry consumes another place in the queue and makes the original request less likely to finish.

Independent publication The Register highlighted a second loop: a delayed response from one internal endpoint exposed a latent retry bug in Visual Studio Code. Normal traffic to Copilot Token Service was 7–9 thousand requests per second; during recovery it rose to 70–100 thousand. GitHub described the amplification as approximately tenfold.

Copilot therefore remained impaired after most of GitHub had returned. The client did not create the original network ceiling, but it sustained load just when the service needed spare capacity to recover. The same “try again” behaviour that is helpful on an ordinary network became destructive during a mass failure.

Engineers redirected some traffic from Central US to Northern Virginia while isolating the saturated components. Pausing HAProxy on the four nodes that had exhausted their limits produced broad recovery. Returning all traffic at once was unsafe, however: accumulated retries could immediately consume the newly available capacity.

GitHub reduced gateway retries through a temporary code change, while load balancers returned HTTP 403 responses to inbound Copilot Token Service requests. That broke the loop, after which traffic was restored gradually by site. Most services recovered by 16:36 UTC, Actions remained degraded until about 18:03, Copilot Token Service until 21:02, and the overall incident closed at 21:15.

Git’s distributed design does soften part of the failure. An existing clone retains files, commit history and local branches. A developer can inspect code, commit, compare changes and build the project when the required dependencies and tools are already available.

GitHub has long provided more than a Git remote. Pull requests and their discussions, Issues, Actions, enterprise authentication, approval rules, Copilot tokens, archives and raw source files pass through platform services. A local commit cannot deliver a change to colleagues, start a hosted build or restore a secret issued only to an Actions workflow.

A continuity plan therefore starts with a dependency map, not the statement that “everything is local.” A team needs to know whether it can build and verify a critical fix without GitHub, where release packages and artefacts are stored, how reviewers communicate and which alternative remote path is permitted during a long outage. A repository backup preserves intellectual property; operational coordination has to be designed separately.

For a team operating its own service, a practical resilience review starts with one important path: sign-in, payment, order creation or release. In a test environment, the dependency is deliberately slowed while the team measures how many extra requests the client, gateway and internal services create together. This exposes a feedback loop that an ordinary total-outage test can miss.

1. Define a shared retry budget: the maximum additional request volume over an interval for the whole chain, rather than a harmless-looking limit inside every service.
2. Increase the delay between attempts and add random jitter so thousands of clients do not return in the same millisecond.
3. Open a circuit after a sustained series of failures and return an explicit temporary response while the dependency rebuilds capacity.
4. Observe sidecar concurrency, load-balancer flow limits, queue depth, retry share and recovery rate separately from host CPU and memory.
5. Test the independent working path: local builds, retained artefacts, manual-release procedure and an agreed backup channel for coordination.

The result should be more than library settings: a reproducible failure scenario, load boundaries, alerts and a decision about which feature is disabled first. During recovery, GitHub reduced gateway retries, began rejecting some Copilot Token Service requests with HTTP 403 responses, and then restored traffic gradually. Resilience here is not the ability to retry forever; it is knowing when to stop asking.