Debugging Kubernetes Chaos in a Pandemic

April 19, 2021

---

So here I am, staring at my laptop screen while the world is still dealing with the aftermath of a global pandemic. Today feels like any other day, except for the fact that I’m working remotely and my team has shrunk by two members. I’ve spent the last few days debugging a Kubernetes cluster issue that’s been bugging me, and it feels like I’m walking into a haunted house.

The Incident

A couple of weeks ago, we started noticing a series of 502 errors on our staging environment. It wasn’t immediately clear what was causing them, but the logs in the production environment were clean—no errors, no warnings. That’s always a good sign that your monitoring and logging are working as intended. But it also means you have to dig deeper.

The Investigation

I spent hours going through the Kubernetes cluster logs, trying to understand if there was any anomaly in the API server or controller manager. I hit all the usual suspects—DNS issues, network problems, resource constraints—but nothing seemed out of place. That’s when I remembered something from my SRE days: eBPF might be able to give us some insights.

I dug up a few eBPF snippets that can help trace network traffic and inspect what’s hitting the API server. Setting it up took more time than expected, but eventually, we had a working trace in place.

The Revelation

The eBPF trace revealed something unexpected: there was a periodic spike in request latency right before the 502 errors started appearing. I dug deeper and found that our internal developer portal, which is built on Kubernetes, was responsible for these spikes. Specifically, the Backstage application, which handles API calls from other microservices, seemed to be timing out.

I sat down with the Backstage team and we realized that their deployment had not been optimized for high concurrency. The requests were hitting a limit in the number of connections allowed by the service mesh, leading to backpressure and eventually 502 errors.

The Fix

We worked together to optimize the Backstage application’s connection settings and re-deployed it. It was a simple fix, but the process highlighted something important: as we formalize our platform engineering practices, ensuring that tools like Backstage are robust enough for high availability is crucial.

This experience also brought back memories of the ArgoCD and Flux GitOps days when we were first experimenting with continuous delivery to Kubernetes. The complexity of managing stateful applications in a CI/CD pipeline can be overwhelming, but it’s essential for maintaining reliability and consistency across our services.

Lessons Learned

Debugging this issue was like solving a puzzle piece by piece. While the tech stack and tools have evolved, the fundamental principles remain the same—monitoring, tracing, and optimizing are still the keys to successful platform engineering.

As we move forward, I believe that embracing DevOps practices more deeply will help us handle these kinds of issues more efficiently. The rise of SRE roles is a clear sign that we need to focus on resilience and reliability as much as on functionality.

And while the tech news might be filled with controversies and distractions (like the Cellebrite exploit or the University of Minnesota patching fiasco), in our world, it’s the small victories—like fixing a 502 error—that keep us going.

---

That’s today’s reflection. More work to do, more bugs to debug, but for now, I’ll enjoy this moment of relative calm before the next challenge comes knocking.