$ cat post/the-build-finally-passed-/-the-incident-taught-us-the-most-/-the-signal-was-nine.md
the build finally passed / the incident taught us the most / the signal was nine
Title: Kubernetes vs. Mesos: A Love Triangle
January 11, 2016 was a day much like any other at the office—emails to reply to, meetings to prep for, code to review. But I found myself thinking about something far more pressing: our container orchestration strategy.
At the time, we were heavily invested in Mesos with Marathon as our scheduler, running on top of an Apache Cassandra-based service registry called Tribulation. It was a beast, but it worked—and when something works, why fix it? Well, it turns out, when the tech industry changes faster than you can say “hello world,” you need to rethink your strategies.
Kubernetes had started to gain traction as the leading container orchestration tool. Google’s open-source project promised a more standardized and battle-tested approach compared to Mesos. And then there was the growing ecosystem around Kubernetes—Helm for package management, Istio for service mesh, Envoy as a sidecar proxy. It was hard not to get drawn into the hype.
But hey, who could blame me? I spent countless hours trying to make Tribulation work seamlessly with Marathon and our Cassandra cluster. The solution felt like it had all the complexity of running a small nation, but at least everything worked together (mostly). Now Kubernetes offered a simpler model: pods and services, no more worrying about a custom service registry.
So, we decided to dive in and see if we could move our apps from Mesos/Marathon onto Kubernetes. It seemed like a straightforward task—convert the YAML manifest files for our deployments, run some tests… right?
Wrong.
The first few attempts ended up with pods failing to start or services not being reachable. The logs were cryptic, and it was hard to track down what was going wrong. Our devops team spent hours trying to figure out why our applications behaved differently in Kubernetes compared to Marathon. It turned out that Marathon had some quirks for handling environment variables and health checks that we hadn’t anticipated.
We argued about whether this was a problem with the tool or if we were just doing it wrong. Some team members were skeptical, arguing that Mesos and Marathon worked fine for us, so why change? Others were enthusiastic, excited by the prospect of a more standardized approach and the potential benefits down the line.
In the end, we decided to stick with our current setup for now. It wasn’t a defeat; it was a recognition that change isn’t always easy, especially when you’ve built something that works, even if it’s a bit complex. But I couldn’t shake off the feeling that this was just the beginning of another tech cycle.
As I sat there staring at my computer screen late into the night, trying to get those pesky pods up and running, I realized how much the landscape had changed since we first started using Mesos and Marathon. Now, Kubernetes seemed like a natural progression, but it wasn’t clear what would come next. Was this just another passing fad, or was it here to stay?
For now, our platform remains a love triangle between Mesos, Marathon, and Kubernetes. But one thing is certain: the tech industry never stands still, and neither should we.
That’s how I thought about our container orchestration strategy back in 2016. The love triangle of Mesos, Marathon, and Kubernetes is just one example of the rapid changes happening in tech at that time. It was a reminder to stay adaptable and open to new tools and ideas, even when they seem intimidating or complex.