Title: Kubernetes Complexity Fatigue and My Quest for Simplicity

---

February 24, 2020. Another day in the life of a platform engineer. Today was particularly grueling because I spent most of it dealing with the insidious complexity of Kubernetes (k8s). K8s has been an incredible journey from its early days to what we see now, but sometimes, just when you think you’ve got a handle on everything, something comes along and shakes your confidence.

The Setup

We had recently rolled out a new set of k8s cluster templates for our development environments. These were supposed to simplify the way developers spin up their own clusters, making life easier for everyone. But boy, was I wrong!

The Problem

One of the big changes was the move from Helm charts to a more opinionated setup using Kustomize. This shift aimed at streamlining deployments and reducing the number of customizations needed by individual teams. However, it quickly became clear that we had overengineered things.

Developers were complaining about the lack of flexibility. The new templates felt like they were forcing us into a box we couldn’t easily escape from. Some wanted to tweak resource limits; others needed different volumes or networking configurations. It was chaos in my inbox!

Debugging the Mess

I spent hours trying to understand why certain deployments were failing, and it boiled down to subtle differences between how k8s handles resources with Kustomize versus Helm. The error messages weren’t helpful at all, just generic “failed to apply” or “resource quota exceeded.” It was like trying to solve a puzzle without the pieces!

I finally broke down and wrote a script to diff the Kubernetes manifests generated by our old and new setups side-by-side. This helped me pinpoint where things were diverging, but it only solved part of the problem.

The Solution

After much debate with my team, we decided to take a step back. We realized that while Kustomize was great for simplifying common use cases, it wasn’t as flexible as Helm for our more complex needs. So, we rolled out a hybrid approach: keep the best parts of Kustomize and complement them with some Helm charts where necessary.

This didn’t solve everything overnight, but at least it provided a temporary fix while we figured out how to do things right. The key lesson was that simplicity can often be overrated. You need just enough complexity to meet your needs without adding unnecessary layers.

Moving On

As I sit here writing this, I’m reflecting on the journey. Kubernetes is a powerful tool, but it’s not a silver bullet. It requires constant attention and adaptation. The tech world moves fast, and we’re only as good as our latest iteration. Today was about learning from the past so that next time, things go smoother.

In the meantime, I’m going to hit up the team with some Go-to-Rust discussions (yes, I know, it’s a bit of a trend right now). Maybe we can find ways to improve our development workflows and reduce complexity without sacrificing too much flexibility. After all, in tech, simplicity is hard to achieve, but striving for it is a noble goal.

---

That’s the day. Complexity fatigue, hybrid solutions, and endless cycles of iteration. Stay tuned for more adventures in platform engineering!