$ cat post/the-blinking-cursor-/-the-health-check-always-lied-/-the-patch-is-still-live.md

the blinking cursor / the health check always lied / the patch is still live


Title: When Mosh Was More Than a Bash


April 30th, 2012. It’s hard to believe this was more than ten years ago, but there it is. Back in those days, I had just joined a startup that was all about the bleeding edge of DevOps and cloud infrastructure. We were using OpenStack for our compute needs and Chef for our configuration management. The DevOps term was starting to take hold, and everyone was talking about how we could do things differently with automation.

One morning, I was trying to SSH into a server in my virtual private network (VPN) and it wasn’t working. It’s always frustrating when you can’t get onto your machine, especially if it’s an emergency. After restarting the machine a few times, I remembered reading about this cool new tool called Mosh.

For those of you who might not know, Mosh is a replacement for SSH that aims to be more robust and user-friendly. It claims to provide a better experience when dealing with unreliable network connections by automatically resuming your connection without dropping the terminal session—pretty nifty stuff if you ask me.

I installed it and tried connecting again. And it worked! The session was stable, even as I moved around my office. But here’s the kicker: it wasn’t just about the SSH stability; Mosh made debugging a nightmare.

You see, with regular old SSH, when a connection drops, your terminal session dies, and you get disconnected. With Mosh, while the connection issues were much less frequent, they still happened sometimes, and I was left staring at an unresponsive command line. It felt like trying to debug a program that was stuck in a loop—unpleasant and not very productive.

So there I was, trying to fix some configuration issue on one of our servers via Mosh, with no real way to make progress until the network settled down enough for me to re-establish a solid connection. It was a bit like debugging a live production environment while being unable to fully inspect or alter state.

But the problems didn’t end there. I started noticing that certain commands would take significantly longer to execute when using Mosh compared to SSH. The latency was frustrating, and it really hampered my productivity. Plus, I couldn’t use any of my usual keyboard shortcuts because they were subtly different in Mosh. It felt like a step back into the dark ages of command-line interfaces.

In the end, I decided that while Mosh had some great ideas, it wasn’t quite ready for prime time yet. The reliability issues and performance hits were too big of a trade-off given our current needs.

Reflecting on this experience, I realized that sometimes the best solution isn’t necessarily the most cutting-edge one. We needed a tool that was rock-solid and didn’t introduce new problems just to solve old ones. This experience taught me that every time we adopt something new, we need to carefully weigh its benefits against potential downsides.

That’s not to say I never tried Mosh again—far from it! Over the years, as network technology improved and tools matured, I came back to it multiple times. Each time, though, there were still trade-offs that made me pause before fully committing.

In the grand scheme of things, this little episode with Mosh helped shape my approach to adopting new technologies. I now try to look beyond just the hype and consider how a tool or practice will impact our specific environment and workflow. It’s a reminder that sometimes the most straightforward solutions are the best ones.

And if nothing else, it made for some entertaining debugging sessions. Now whenever someone mentions Mosh, I can’t help but think of my battle with unreliable network connections and long command execution times—fun times!