Title: The Year That Kubernetes Gave Me a Tummy Ache

---

December 30, 2019. Another year is coming to a close, and I find myself reflecting on the journey that has been platform engineering in this crazy tech world.

Kubernetes Complexity Fatigue, or My Love-Hate Relationship with K8s

As we enter 2020, one thing stands out: Kubernetes (K8s) has officially entered its second act. The initial hype is waning, and the reality of running a production-grade cluster on a tight budget is setting in. It’s like being in the final year of college when you realize that not every problem has an easy answer.

In my world, we’ve been trying to streamline our Kubernetes deployments and operations. We’re not using any fancy managed services—just vanilla K8s on some beefy bare metal servers. Every time I look at the cluster configuration, it feels like a monolithic application with 100+ microservices. Each service has its own deployment strategy, ingress rules, and monitoring hooks. It’s a perfect storm of complexity that makes me scratch my head every morning.

The Big Debug

A few weeks ago, we were seeing some unexpected behavior in one of our critical applications running on the cluster. The application would crash periodically without any obvious errors in the logs. After hours of digging through code and manifests, I finally stumbled upon a misconfigured resource quota that was causing pods to get evicted due to excessive memory usage.

The irony is that this issue wasn’t even directly related to the app itself but rather how it interacted with other services. It’s like trying to solve a jigsaw puzzle where all the pieces are the same color and shape—hard work, but no easy answers.

The Argument That Won’t Go Away

There’s been an ongoing debate within our team about whether we should adopt GitOps tools like ArgoCD or Flux for our deployments. On one hand, it would make our lives easier by automating a lot of the manual steps and reducing human error. On the other hand, there’s this nagging feeling that every tool comes with its own set of complexities.

We’ve been using Helm charts to manage our Kubernetes resources, but as the number of services grew, so did the complexity in managing the charts themselves. The argument boils down to whether the benefits of GitOps outweigh the learning curve and potential overhead. I’m still on the fence, but leaning towards adopting Flux because it seems more lightweight than ArgoCD.

Internal Developer Portals

One bright spot has been our internal developer portal built with Backstage.io. It’s a one-stop shop for all things related to our services—docs, APIs, and monitoring. We’ve been slowly adding content and integrating tools, but there are still gaps we need to fill. The challenge is balancing the technical details with usability so that developers can find what they need quickly.

Remote Work and Infrastructure

Speaking of Backstage.io, the past year has seen a significant shift towards remote work due to the pandemic. Our infrastructure had to adapt quickly, which meant scaling out our cloud resources while ensuring we maintained robust security practices. It’s a constant balancing act between flexibility and control, especially when managing multi-cloud environments.

eBPF: The New Kid on the Block

While Kubernetes is front and center in my daily work, there are whispers of excitement around eBPF (Extended Berkeley Packet Filter). I’ve started experimenting with it for some network monitoring tasks. It’s a powerful tool that can give us deep insights into what’s happening at the OS level without invasive changes to our applications.

Wrapping Up

2019 was a year of mixed emotions in tech—excitement and frustration, progress and setbacks. As we look ahead to 2020, I’m hopeful that Kubernetes will find its footing, GitOps will continue to mature, and eBPF will prove itself as the next big thing. But for now, I’m just glad to have a cup of coffee while my cluster runs smoothly.

Until next year, Brandon