$ cat post/ps-aux-at-midnight-/-what-the-stack-trace-never-showed-/-we-were-on-call-then.md

ps aux at midnight / what the stack trace never showed / we were on call then


Reflections on a WebAssembly World: A Day in the Life of an Engineering Manager

November 25, 2024. Another day on the web, another codebase to debug and a platform to optimize. Today, I find myself wrestling with the latest twist in my ops world—WebAssembly (Wasm) on the server side.

The Era of WebAssembly

Just last month, Wasm took a leap forward with its integration into backend services. As someone who’s seen firsthand how containerization simplified deployments, this move felt like another step toward seamless, cross-platform development. But as with any new technology, there are growing pains.

One morning, I’m sifting through logs from our latest Wasm deployment on Kubernetes. It’s a typical Monday—nothing out of the ordinary. Then, my eyes catch an error that stands out: Wasm module failed to initialize. A quick glance at our monitoring dashboard shows a few containers going down left and right.

The Debugging Session

I hop into my debugging session, armed with wasm-gdb and a stack trace. It’s frustratingly verbose—too much information for my taste, but it gets the job done. I start tracing through the initialization process, looking for where things might be going wrong. After an hour of stepping through code, I finally find the culprit: a mismatch in the WASM version.

This isn’t a bug in our application; rather, it’s a deployment issue. The version we built and deployed wasn’t compatible with the runtime environment. A small mistake that could have been avoided if I had double-checked the build process before the rush to deploy.

Platform Engineering and DevEx

As platform engineering continues to mainstream, I can’t help but think about how much this affects my day-to-day work. Our team is now more focused on creating robust platforms that developers can easily use without getting mired in low-level details. The idea of developer experience (DevEx) as a discipline has really taken hold. Every piece of infrastructure we build needs to be not just functional but also intuitive and easy to integrate.

Today, I spent some time working with the DevOps team on a new CI/CD pipeline for our Wasm projects. They’ve got some cool stuff going on with GitHub Actions and Kubernetes clusters. We’re also looking into using Helm charts to standardize our deployments across different environments. It’s nice to see everyone collaborating more closely, but it can be a bit overwhelming with all the new tools and practices.

FinOps and Cost Optimization

And then there’s FinOps. The pressure is on to optimize costs as we scale. Every penny counts in cloud infrastructure these days. I’ve been spending some time analyzing our AWS bill and looking for ways to reduce costs without compromising performance. It’s a bit of a balancing act, but with DORA metrics widely adopted, we’re constantly monitoring and adjusting.

One thing that has helped is the introduction of cost allocation tags on resources. We now have more granular control over how costs are billed, which makes it easier to track spending. However, it also means we need to be extra mindful about tagging our resources correctly to avoid unnecessary charges.

The Human Side

In between all this tech talk, I’ve been thinking a lot about the human side of engineering. Our team has grown significantly over the past few years, and with that growth comes new challenges. We’re exploring different management styles and trying to find what works best for everyone. It’s not always easy, but it’s rewarding.

Today, my colleague sent around an article on “How I ship projects at big tech companies.” It reminded me of some of the struggles we face in balancing perfectionism with pragmatism. There are times when we need to ship something quickly and iteratively rather than waiting for the perfect solution. This mindset shift has been key as we navigate the complex landscape of platform engineering.

Conclusion

As I wrap up my day, looking at the screen filled with code and logs, I reflect on how far we’ve come since those early days when Wasm was just a concept. Now it’s a real player in our tech stack, bringing new possibilities but also new challenges. It’s a mix of excitement and frustration, but ultimately, that’s what makes this job so interesting.

Tomorrow, the cycle starts again—more code to debug, more platforms to optimize, and more ways to make developer lives easier. Here’s to another day in the world of platform engineering!


This blog post is a blend of personal reflection, technical challenges, and industry trends—a true snapshot of what it’s like to be an engineering manager in 2024.