Random stalls of 100 ms or more on connections to Kubernetes nodes

Debugging network stalls on KubernetesTheo Julienne / The GitHub Blog, 2019

Services on GitHub's Kubernetes clusters saw occasional delays of 100 ms or more, even for database queries that the database itself finished in milliseconds. The team stripped the path down one layer at a time: a load-test client, then bare TCP SYN probes with hping3, then the tunnel between nodes on its own, then plain ICMP between hosts. The delay survived every simplification, and the timings showed the stalls arriving in groups. The rest of the investigation follows that clue into the node itself.