$ cat post/tab-complete-recalled-/-a-midnight-pager-i-still-hear-/-no-rollback-existed.md

tab complete recalled / a midnight pager I still hear / no rollback existed


Title: February 1, 2016 - The Dawn of Platform Engineering


I’m sitting in a conference room at a small startup where we’re just starting to grapple with the complexity of our platform. We’ve got containers running on Kubernetes, some services are using Istio for service mesh, and most everyone is still trying to wrap their heads around what exactly serverless means. Our monitoring stack? A hodgepodge of Prometheus and Grafana. And then there’s Terraform—oh man, Terraform. The whole team knows we need it but no one wants the responsibility.

It’s a little surreal because just six months ago, we were using good old Nagios for everything. But that didn’t scale with our growing user base, especially as more and more microservices started popping up. We needed something better—something smarter—and that’s when Kubernetes came along like a savior. We started to migrate all our services over, but it wasn’t without its challenges.

One of the biggest issues we faced was how to manage the state of our infrastructure. With Kubernetes, everything is ephemeral; pods come and go in seconds. But our monitoring setup still relied on stable resources that didn’t match the dynamic nature of Kubernetes. We tried to force-fit things together, but it felt like fighting a losing battle.

That’s when I first heard about Terraform. It promised to be a game-changer—providing a declarative way to manage infrastructure across multiple cloud providers and orchestration tools. The idea was exciting, but the reality was harder than expected. We had to learn all these new concepts and figure out how to integrate it with our existing stack.

I remember spending countless hours wrestling with Terraform 0.x. It wasn’t user-friendly by any means; there were plenty of bugs and missing features. But we pushed through because we knew the benefits outweighed the pain points. Eventually, we got it working, but it was a learning curve that none of us had prepared for.

Speaking of curves—our platform engineering journey is full of them. One day, I’m dealing with Kubernetes pod crashes, and the next, I’m arguing about whether to adopt Helm or not. The serverless hype was everywhere; everyone wanted in on the action, but we were still figuring out what a serverless architecture even looked like for us.

Then there’s GitOps, which seems like the right way forward. But how do you apply it without breaking your current setup? The answer isn’t always clear-cut. Sometimes, it feels like we’re just making things more complex than they need to be. But at least we’re talking about it, and that’s progress.

As I look out the window on this February morning, I can’t help but think how much has changed in such a short time. It’s overwhelming, but also exciting. The technology landscape is shifting so rapidly, and we’re right in the middle of it. There are no easy answers, just a lot of questions to answer.

For now, we’ll keep moving forward one step at a time. Maybe tomorrow will be better than today, or maybe not. But either way, I’m glad I get to be part of this journey. After all, what’s the point of it if not to learn and grow?


This blog post captures the essence of working in an environment where new technologies are rapidly emerging and transforming how we build and operate systems. It reflects on the challenges and the excitement of being at a startup during the early days of platform engineering and Kubernetes adoption.