$ cat post/march-4,-2013---a-day-in-the-life-of-a-devops-monkey.md

March 4, 2013 - A Day in the Life of a DevOps Monkey


March 4, 2013. Just another day for me to keep my head down and make sure our services don’t go into meltdown because someone accidentally deployed the wrong version of our app again.

Today, I started off with a bit of a scare. One of our production servers had a nasty crash at midnight, causing an outage that lasted a few hours. The stack trace was not very helpful—just some generic segfault in libstdc++. It turned out to be a race condition between two threads trying to access the same resource. The culprit? A badly written piece of code that hadn’t been caught by our automated tests.

I had just finished getting the server back online when I got an email from one of the junior developers asking about Docker. They heard it was going to be big, and wanted to know more about how we might start using it. I gave them a quick intro over IM, mentioning that CoreOS and etcd were starting to gain traction in the industry. At that point, containers seemed like the next logical step for us, but we needed to figure out how to integrate them with our current infrastructure.

Lunchtime rolled around, and I made my way down to the kitchen. The place was bustling with activity as usual. I grabbed a sandwich from the fridge and went over some of the recent discussions we had been having about microservices. Our team wasn’t convinced that they were worth the effort just yet, but given Google’s announcement of Kubernetes in 2014, it looked like our peers in Silicon Valley really thought they were onto something.

The kitchen was where I often found myself during these debates. It was a safe space, with everyone willing to share their thoughts and opinions. Today, we spent the whole lunchtime hashing out the pros and cons of microservices versus monolithic architectures. Some argued that microservices would make it easier for us to scale independently; others pointed out the complexity introduced by managing multiple services. I chimed in with a few personal experiences from projects where I had seen both approaches work (or not).

After lunch, I returned to my desk to tackle some urgent issues. One of our SaaS apps was misbehaving and causing slowdowns for a lot of customers. I started diving into the logs, trying to find out what was going wrong. It turned out that a recent update had caused a cascading failure in our data pipeline. The fix involved a quick patch to the code and a few manual steps to restart some services. Once that was done, everything seemed to go back to normal.

I spent the rest of the afternoon arguing with myself over whether we should have been using 12-factor principles all along. Our team had been pushing hard for more ops automation, but I wasn’t entirely convinced that the extra complexity was worth it. The argument felt a bit silly in hindsight—given what I know now about DevOps and CI/CD pipelines, it’s clear that those practices would have made our lives much easier.

As the day wound down, I couldn’t help but feel a mix of frustration and excitement. Frustration for all the issues we faced, and excitement for the technologies on the horizon. Docker was still in its early days, but CoreOS had already started to show how it could transform how we deploy and manage our services.


March 4, 2013, was a day filled with both challenges and opportunities. It’s days like these that remind me why I love this job—always something new to learn, and always more work than there is time for.