July 1, 2019. I remember it like yesterday—the morning dew on my screen as I faced yet another Kubernetes cluster with a hairball of customizations and configurations. It’s been a while since I’ve felt that sense of dread before diving into an old cluster, but today is no exception.

The Problem

Every day, more engineers join our platform team. They bring in their ideas, their solutions, and sometimes, the baggage of years spent managing Kubernetes clusters. Each project starts with a fresh installation, but over time, it becomes clear that we’re all guilty of adding unnecessary complexity to these systems.

Take the cluster I’m working on now: it has multiple namespaces, custom resource definitions (CRDs), Helm charts, and a mix of manual and automated deployments. It’s like trying to find your way through a tangled spider web without a map.

The Solution

A couple of months ago, we started looking at ways to standardize our approach. One of the things that caught my eye was ArgoCD and Flux GitOps. These tools promised a cleaner, more manageable way to apply and maintain configurations across multiple clusters.

I decided to set up an internal Backstage portal to document all these clusters and their components. It wasn’t just about centralizing information; it was also about making sure everyone had the same baseline knowledge before diving into any new project.

The Argument

Not everyone was on board, of course. Some argued that this would slow down deployment cycles. “We know how to do it manually,” they said. “Why should we change?” I understand their point, but I see the value in consistency and predictability over time.

I drew an analogy: “Imagine you’re trying to build a house with 10 different blueprints. Sure, each blueprint might be unique, but building the same house repeatedly means less error and faster construction.”

The Implementation

So, I dove into setting up ArgoCD and Flux in our development environment. It wasn’t easy; there were lots of gotchas along the way. For instance, ensuring that all Kubernetes resources are properly namespaced and versioned can be a real headache. But as the weeks went by, the benefits became more apparent.

We started seeing less drift between environments, which meant fewer issues during deployments. And with Backstage, we had a centralized place to document everything from cluster architecture to deployment pipelines. It was like finally having that map I mentioned earlier.

The Reflection

Looking back at this time, it’s clear that we’re navigating the waters of Kubernetes complexity fatigue. It’s not just about deploying services; it’s about creating maintainable and scalable infrastructure that can support our growing platform team.

While the path wasn’t easy, the results have been worth it. We’ve moved from a cluster-by-cluster approach to one where consistency is key. And as we continue to evolve, I’m excited to see what new tools and best practices emerge.

Kubernetes has its quirks, but with careful planning and a commitment to simplicity, we can keep our clusters manageable and our deployments smooth. After all, in the world of distributed systems, the simpler you can make things, the better.

---

This isn’t just about Kubernetes; it’s about keeping technology simple enough for everyone involved. That’s what I’ve been thinking about today as I sit in front of my screen, surrounded by yet another complex cluster. But with ArgoCD and Flux, and a bit of Backstage, we’re making progress.