$ cat post/tail-minus-f-forever-/-a-webhook-fired-into-void-/-we-kept-the-old-flag.md

tail minus f forever / a webhook fired into void / we kept the old flag


Title: March 2, 2020 - A Day in the Life of Platform Engineering


March 2, 2020. I look out the window at the drizzle and think about how much things have changed just over a few months. Back in October, we were still trying to get our platform team up and running smoothly. Now, it’s March, and we’re already seeing the impacts of the pandemic starting to ripple through.

Today was a mix of debugging Kubernetes issues and arguing about tooling decisions. Let’s dive into some of that chaos.


The Kubernetes Headache

This morning started with a fire drill. We had a critical alert from our monitoring tools: one of our cluster nodes was running out of disk space, and it looked like the issue might be hitting multiple clusters at once. I knew immediately we needed to act fast.

I logged into my laptop and pulled up KubeStateMetrics to see if I could identify which pods were using the most resources. After a bit of digging, I found that a few of our microservices had been running for longer than expected without proper cleanup, leading to some unexpected state persistence. I quickly wrote down some notes about how we needed to improve our CI/CD pipeline to ensure better resource management.


The Backstage Debates

Lunch was spent in the usual back-and-forth with my team about our internal developer portal using Backstage. We’ve been playing around with it for a while, but there’s still so much to figure out. Our CTO is pushing hard for us to move to something more mature like Backstage, arguing that it will streamline our development process and make onboarding new developers easier.

I’ve been skeptical of the cost and complexity. “It’s great,” he says, “but are we sure this is worth all the effort?” I counter with concerns about over-engineering—Backstage has a lot of features, but do we really need them? We might be adding unnecessary bloat to our stack.

We end up agreeing to pilot Backstage further, but only for a subset of services. If it works out, great. If not, we can always go back to our homegrown solution.


The Pandemic’s Ripple Effects

As I head back to my desk after lunch, the office is quiet. Normally there are people walking around, but today feels more… sterile. Word has it that a few team members have started working from home due to health concerns, and everyone’s taking extra precautions.

The thought of how this might impact our infrastructure in the long term keeps bouncing around my head. We’ve been preparing for remote work for months now, but I wonder if we’re ready for a full-scale shift. How will this affect our ability to monitor and manage our clusters? Will there be more bugs due to fewer eyes on the code?

I try not to dwell too much on it, knowing that these are issues we’ll have to face soon enough.


The Tech News Reflection

Later in the day, I catch up on some Hacker News. The news stories this month really highlight how technology is intertwined with real-world events right now. Take Zoom, for example—privacy concerns and calls for regulation. It makes me think about our own use of video conferencing tools within our company. Are we doing enough to protect our data?

And then there’s the SETI@home shutdown—a project that felt like it was always going on in the background, now coming to a close after 21 years. It’s a reminder of how technology can have long lifespans and significant impacts.


Wrapping Up

As I close out my day, I find myself reflecting on all these things: the Kubernetes issues, the Backstage debates, the pandemic’s looming presence. It’s a mix of mundane problems and broader concerns about our industry and world. But it’s also a reminder that despite everything, we’re still here, tackling one challenge at a time.

In the tech world, 2020 is shaping up to be an interesting year. We’ll have our work cut out for us as we navigate through this new normal. I can’t wait to see what comes next.


That’s my day in review—real, messy, and full of challenges. Hope you found it useful!