$ cat post/the-pager-went-off-/-the-index-was-never-rebuilt-/-the-wire-holds-the-past.md

the pager went off / the index was never rebuilt / the wire holds the past


Title: September 14, 2020 - The Year of Remote Work and Kubernetes Complexity


September 14th, 2020. I woke up to the realization that this might be one of the last times I’d type this exact date before I was back in the office. But alas, it was just another remote day for me as a platform engineer at a tech company navigating through the pandemic.

The Remote Ramp-Up

Remote work has become the new norm here. We’re trying to scale up our infrastructure to support a more distributed workforce. Every time I look at my screen, there’s an internal developer portal—Backstage—that serves as a hub for all our microservices and applications. It’s like having a digital command center, but it feels odd when your co-workers are just names in Slack channels instead of the real-life humans you usually bump into.

Kubernetes Complexity Fatigue

Speaking of Backstage, one thing I’ve wrestled with is managing our complex Kubernetes setup. We’re hitting that point where the system has grown beyond simple cluster management. ArgoCD and Flux GitOps are slowly becoming part of our daily operations, but they come with their own set of challenges. How do you version control your Kubernetes manifests? What happens when a deployment fails, and how do you roll back to a previous state?

I’ve spent countless hours debugging deployments that went south because of minor configuration issues. The irony is not lost on me: in trying to make our infrastructure more robust with GitOps, I’ve ended up spending more time fixing GitOps-related problems.

eBPF: A New Frontier

Another tech trend catching my attention is eBPF (Extended Berkeley Packet Filter). It’s gaining traction as a powerful tool for kernel-level programming without the need for traditional module loading. We’re exploring its potential to improve our observability and performance tuning at runtime. It’s like having superpowers for your system, but you have to be careful not to misuse them.

The Great Remote Infrastructure Scaling

On a more mundane note, we’ve been working on scaling up our remote infrastructure to handle the surge in demand. With everyone working from home, network congestion and bandwidth issues have become real problems. We’re looking at ways to optimize our CDN setup and ensure that critical services remain accessible even during peak times.

A Day in the Life

A typical day starts with a Slack channel ping or two—most of my colleagues are still on Pacific time. I spend a few minutes catching up, trying to sync without any visual cues, and then jump into coding or debugging. The hardest part is staying focused when you can’t see your teammates. There’s an element of isolation that comes with remote work that I’ve never fully adapted to.

Learning and Growing

Despite the challenges, there’s also a lot of opportunity for learning. Working on Kubernetes and GitOps has pushed me to understand more about how these systems are built and why they work as they do. It’s given me a deeper appreciation for the complexity behind what we consider “simple” tools.

Reflections

As I type this, I can’t help but feel both frustrated and relieved. Frustrated because there’s so much to learn and do, and relieved that the world isn’t going back to normal any time soon. In a way, it’s given us more time to figure out what we really need from our tools and processes.

Looking Forward

In October 2020, I’ll be heading up a new project to integrate some of these learnings into our platform engineering practices. It’s exciting but also daunting—the road ahead is long, and the stakes are high. But that’s why I do this, right? To build things that make life better for everyone.


That’s where I was on September 14th, 2020—treading water in a sea of Kubernetes clusters, trying to navigate the complexities of remote work with the help of eBPF and GitOps. The future is uncertain, but one thing’s certain: we’ll figure it out together.