Title: September 16, 2019 - When Life Hacks Me into a Corner

---

Today’s date is September 16, 2019, and I can’t help but feel a mix of nostalgia and dread. Nostalgia for the simplicity that used to define my work; dread because as much as things have changed, they seem to be changing even more rapidly.

Over the past few months, platform engineering has become a formalized role in our organization. We’ve been exploring the concept of an internal developer portal using tools like Backstage, but it’s clear we still need to iron out some kinks. The idea is great—centralize all your infrastructure and development resources so everyone can get up and running quickly. But as with anything new, there are bumps along the way.

On the SRE front, things are getting interesting too. We’re seeing more roles being created, but it’s not always clear what the boundaries between dev and ops should be. At our company, we’ve been experimenting with using Kubernetes in a more complex environment, which has led to some healthy debates about when and where to apply the tooling.

And then there’s the ongoing Kubernetes complexity fatigue. We all know how Kubernetes can make life simpler by abstracting away a lot of low-level details, but it also adds another layer of complexity that we need to manage. Sometimes I wonder if our infrastructure is becoming a Frankenstein’s monster—a mishmash of tools and technologies that are just barely held together.

As for the tech world at large, there’s been an explosion in SRE roles and platform engineering formalization. It seems like every other day, someone is writing about how to better organize their teams or how to use Kubernetes more effectively. The articles and discussions can be overwhelming. But amidst all this noise, one thing stands out: the rise of eBPF.

eBPF (extended Berkeley Packet Filter) has been gaining traction over the past few years. It’s a powerful tool that allows us to inject custom code into the kernel without having to compile or install anything. This means we can do things like monitoring and tracing with minimal overhead, which is incredibly useful for our operations team.

But with great power comes great responsibility. We’ve been experimenting with eBPF in some of our performance-critical areas, but it’s not exactly a walk in the park. Debugging issues can be tricky because you’re dealing directly with kernel-level code, and every little thing matters. It’s like trying to find a needle in a haystack when that needle is invisible.

Speaking of experiments, we’ve also been looking at GitOps tools like ArgoCD and Flux. These tools aim to bring the same declarative approach to infrastructure management as they do to application development. The idea is compelling: you define your desired state once, and these tools ensure it’s always applied automatically. But in practice, there are still a lot of moving parts to get right.

One recent incident really brought this home. We were trying to set up an automated deployment using ArgoCD for one of our services. Everything seemed to be working fine until we hit the first real-world scenario where something went wrong. A simple change in configuration caused a cascading failure across multiple systems. It was like watching a carefully constructed Lego tower crumble before your eyes.

The incident led us into some intense troubleshooting sessions. We spent hours digging through logs and tracing dependencies, trying to understand what had gone wrong. And while we eventually figured it out, the experience left me questioning whether our GitOps setup was mature enough to handle real-world complexity.

As I sit here reflecting on all this, I can’t help but think about how much has changed in just a few years. When I started in this field, the idea of platform engineering and SRE roles were still nascent concepts. Now they’re not just buzzwords—they’re part of the daily reality for many engineers.

But amidst all the change, one thing remains constant: there’s always more to learn, more problems to solve. And while it can be frustrating at times, I wouldn’t have it any other way. This is why I love being an engineer—there’s a constant challenge and a sense of discovery that keeps things interesting.

So here’s to September 16, 2019—the day when life hacked me into a corner, but also gave me the tools to climb out.