$ cat post/february-8,-2021---the-year-i-learned-the-hard-way-that-gitops-isn't-just-a-buzzword.md

February 8, 2021 - The Year I Learned the Hard Way That GitOps Isn't Just a Buzzword


February 8, 2021. A day that felt like the middle of winter, with a chill in the air and the faint hum of servers in my remote home office. Back then, I was deep into the weeds of Kubernetes management at our company. We had been running a small monolith for years, but as our team grew, so did the complexity of keeping everything up and running.

One weekend, things went south. The app started to slow down, and users were getting frustrated with laggy performance. I was on call, trying to diagnose what was going wrong. It turned out that a new version of our application had been deployed without proper testing, causing issues with database queries. In the heat of the moment, I rolled back the deployment and hoped for the best.

But as things settled down, I realized that something deeper was at play here. We needed better ways to manage deployments and ensure stability. That’s when my manager suggested we look into GitOps.

At first, it felt like another fad. But the more I dug in, the clearer it became: GitOps wasn’t just a buzzword; it could be our savior.

I started by setting up Backstage, the internal developer portal that our team was using to manage our application code and infrastructure documentation. We had been using Helm charts for Kubernetes deployments, but they were becoming unwieldy as we added more services. GitOps promised a way to keep our configurations in version control alongside our application code.

I spent days wrestling with ArgoCD, trying to get it to play nice with our existing setup. I hit every possible roadblock: misconfigured YAML files, missing annotations, and even some pesky RBAC permissions that kept getting in the way. But slowly, ArgoCD started to fit into our workflow like a puzzle piece.

We began by setting up a simple GitOps pipeline for one of our smaller microservices. It was a small victory, but it felt monumental at the time. As we watched the changes flow smoothly from our repository into Kubernetes, I realized that this was more than just automation; it was a cultural shift.

Over the next few months, we expanded our GitOps practices to cover multiple services and even our infrastructure as code (IaC). The transition wasn’t easy. We had to refactor existing Helm charts, clean up redundant configurations, and ensure that everyone on the team understood the new processes. But in the end, it was worth it.

One of the most significant challenges we faced was integrating SRE practices with GitOps. Our team had traditionally handled infrastructure management separately from application deployment. Now, we needed to merge these roles into a single streamlined process. This meant more collaboration and communication—sometimes painful but necessary for success.

As I reflect on that winter day in 2021, I realize that the journey wasn’t just about adopting GitOps; it was about transforming our entire approach to DevOps. It pushed us out of our comfort zones and forced us to confront some hard truths. But looking back, those struggles led us to a more stable, scalable, and maintainable system.

In the years since, I’ve continued to learn and grow as an engineer. The tech landscape is ever-changing, and GitOps has only matured further with tools like Flux and improvements in Kubernetes itself. But that winter day remains a defining moment for me, reminding me of the importance of embracing change when it leads to better outcomes.


That’s my take on how we rolled out GitOps at our company back then. It wasn’t always smooth sailing, but it was definitely worth it.