$ cat post/kubernetes-conundrums:-a-christmas-reflection.md

Kubernetes Conundrums: A Christmas Reflection


December 24th, 2018, and I find myself in a cozy office with the holiday lights twinkling from my computer screen. The snow outside mutes the city’s noise, but inside there’s chatter from team members finishing up their final tasks before the holidays. Kubernetes remains the star of the show; it’s been a year since I first dipped my toes into its deep waters.

Kubernetes was starting to win out in our container wars. We had a small but growing cluster, and Helm made installing and managing applications a breeze. But the journey wasn’t without its hiccups. Our initial setup was a patchwork quilt of bash scripts and hand-crafted manifests. Then Istio came along and promised to add service mesh goodness to our mix.

I remember the first time we tried to integrate Istio into our Kubernetes cluster. It felt like adding another layer of complexity on top of an already complex system. We spent hours wrestling with Pilot, Galley, and Mesh. Configuring our services was a nightmare; one misstep could send everything into chaos. The team was frustrated, but the potential benefits kept us going.

Around this time, I also started exploring serverless options. AWS Lambda was all the rage, and there were plenty of articles hyping the benefits. But after a few experiments, we decided that for our use case, Kubernetes was still the better fit. Serverless just didn’t offer the flexibility we needed, especially when it came to managing state.

Speaking of state, I had a long chat with one of my teammates about using Postgres instead of MongoDB. The argument went something like this: “Postgres is harder to set up and manage, but it gives us better data integrity.” The other side was that MongoDB was easier to work with for our team’s needs. In the end, we decided on a hybrid approach—use Postgres for the critical parts where reliability was key, and stick with MongoDB for the more flexible areas.

The serverless argument is one of those funny things in tech; everyone has an opinion. But I’ve learned that the best solution often lies somewhere between extremes. Overthinking can lead to overcomplicating things, but underthinking can leave holes in your architecture.

Security was another hot topic. The Quora User Data Compromised story really hit home for us. We had a long discussion about how to better protect our data and services. Open source tools like Prometheus and Grafana were becoming more popular for monitoring; they replaced our nagios-based system with something more modern and flexible.

As the year came to a close, I found myself reflecting on all the technologies that made 2018 so exciting but also challenging. Kubernetes was everywhere, but it wasn’t always clear which path was best. We were constantly evaluating new tools and architectures, trying to figure out what worked for our unique needs. There was no shortage of opinions, but making sense of them all was a challenge.

The day before Christmas, I sat down with my team to review the progress we made this year. It wasn’t glamorous; most of it involved debugging, fixing bugs, and arguing about which tool was best. But that’s how you make real, lasting improvements. We talked about what we could do better next year, about new technologies to explore, and about the importance of maintaining a culture of continuous learning.

As I look out at the quiet city outside my window, I’m reminded of the long journey ahead. There will be more Kubernetes updates, more serverless experiments, and probably even more security concerns. But with each challenge comes an opportunity for growth and improvement. This holiday season, let’s take a moment to appreciate the progress we’ve made and the hard work it took to get here.

Merry Christmas, everyone. Let’s make 2019 count.


This blog post reflects on my experiences as a platform engineer in 2018, focusing on Kubernetes, serverless, security, and team dynamics. It’s personal and honest, grounded in real work challenges and discussions.