What Happened

In yesterday’s recent GitHub outage, there was a detail in the writeup that I didn’t say anything about: the autoscaling policy on the service with the saturated Istio sidecar.

Originally this was caused sidecar pod reaching its concurrency limits and failing correctly because of a misconfigured policy that watched host service but not sidecar limits.

I suspect readers familiar with what autoscaling , but here’s a brief summary ’t. The amount of compute and memory resources that a service requires depends ’s placed on that service. The relevant source external requests against the service, traffic. The volume of traffic varies over time. For example, for a company like GitHub, traffic during working hours than evening and weekends.

changes dynamically, compute and memory resources a service need is a function of load, general strategies. One strategy is to provision your service . The other strategy is to dynamically adjust the resources allocated to your service, current load; that’s called autoscaling.

Why It Matters

service to use autoscaling, define an autoscaling policy. In particular, metrics represent load, specify how resources should removed metric changes.

CPU utilization is a common metric used for autoscaling. a service can become saturated . For example, imagine a scenario thread-per-request with a threadpool, and the latency of your downstream requests increase, threads blocked. Here the service is saturated, and you’d benefit from spinning , actually low, because the threads are blocked waiting on I/O (this happened 2021). Now, additional autoscaling policy to handle such cases (, where they rapidly scaled number of threads). incoming request volume instead of CPU, if your service isn’t CPU-bound.

GitHub writeup, it sounds like the autoscaling policy for the impacted service used load metrics account service itself, sidecar.

In general, each service behaves differently under load, autoscaling policy is effectively bespoke. a a service responsible for the business logic, operational control system with custom parameters, that can really only be checked via load testing. ( testing services?) The service owners are also almost certainly not autoscaling experts. ’s not surprising a misconfigured autoscaling policy was a contributor here.

What Comes Next

But, while I think it’s worth discussing the particular defect with this policy, since it’s good for people autoscaling, I ’s fixate exclusion of other factors involved in this incident. refers component substitution fallacy – improve reliability efforts on identifying and fixing the defective components.

While, yes, you should identify defects uncovered by an incident, you should also recognize that:

component defects aren’t enough system, or your system . Don’t individual components: treat the interactions as first-class. In the GitHub outage, we see discussion of interactions between factors such as: changing traffic patterns (including scrapers), autoscaling policy, the Istio sidecar saturation, retry logic, HAProxy node saturation, and authentication traffic.

There’s also a multitude of details we don’t have because this is a rapidly disseminated public writeup, internal writeup. I speculated relationship between service owner and autoscaling policy, but I history here (did this policy predate sidecars, for example?). I’d problematic traffic. ( requests were they? Was it a sudden increase or a gradual ramp-up? traffic increased?).

You can’t get answers questions for public incident writeups, internal organization. It’s questions.

Explore more: Software & AI Guide