A Month of Twists and Turns: October 2022

October 10, 2022 was just another day in the life of a platform engineer. The days blended together with the endless stream of meetings, code reviews, and infrastructure debugging sessions. Yet, as I reflect on that month, it feels like a whirlwind of change and learning.

The AI/LLM Infrastructure Explosion

One of the most notable shifts was the rapid growth in AI/LLM (large language model) infrastructure. ChatGPT had sent shockwaves through the tech industry, and everyone was scrambling to adapt. At our company, we were evaluating different options for hosting and scaling these models. The sheer scale required us to rearchitect parts of our platform, considering everything from GPU allocation strategies to cold start times.

Platform Engineering as a Mainstream Discipline

Platform engineering had become a buzzword in the tech world, but it was no longer just hype. We were actively building out shared services and tooling to support distributed teams more effectively. This meant investing time in DevOps practices like CI/CD, monitoring, and logging. The goal? To make life easier for our engineers, so they could focus on writing robust code rather than firefighting.

The CNCF Landscape

The Cloud Native Computing Foundation (CNCF) landscape was overwhelming. Every other day, a new service or project would emerge, promising to revolutionize container orchestration, edge computing, and more. It felt like we needed a new strategy for managing all these moving parts. We held an internal workshop to map out which CNCF projects made sense for our stack and how they could integrate seamlessly.

WebAssembly on the Server Side

WebAssembly was gaining traction in server-side applications too. While it’s still early days, I experimented with using WASM to offload some of the heavy lifting from our Node.js backend. The performance benefits were promising, but integrating WASM into our existing infrastructure was a challenge. There are still many kinks to work out, like ensuring compatibility across different environments and figuring out how to manage dependencies.

Developer Experience as a Discipline

Developer experience (DX) had become an official discipline at our company. We were setting up more robust tooling for front-end developers, such as pre-built templates and automated testing frameworks. It was refreshing to see DX take center stage; it’s not just about making life easier but also about fostering creativity and innovation.

FinOps and Cloud Cost Pressure

Cloud cost pressure continued to be a constant concern. DORA (DevOps Research and Assessment) metrics were widely adopted, with an emphasis on reducing lead time and improving deployment frequency. We spent significant time optimizing our cloud spend by automating cost controls and implementing stricter budget policies.

Personal Reflections and Learnings

On October 10th, I spent the morning debugging a particularly pesky issue in one of our microservices. It turned out to be a race condition that was only triggered under certain conditions, making it incredibly difficult to reproduce. The root cause was ultimately traced back to an incorrect assumption we had made during code review, which highlighted the importance of thorough testing and documentation.

In the afternoon, I attended a meeting where we discussed adopting GitHub Copilot for our development teams. The discussion was intense; some were skeptical about the impact on productivity, while others saw it as a valuable tool. It’s clear that these types of conversations will continue to shape how we work in the future.

Closing Thoughts

As October 2022 drew to a close, I found myself reflecting on the rapid changes and the constant learning curve. The tech landscape is dynamic and full of challenges, but it’s also incredibly rewarding. Each problem solved feels like a small victory, and each new technology adopted opens up endless possibilities.

In the next blog post, I’ll dive deeper into one of these technologies or tools that caught my eye during this month—maybe even 3D scanning the Great Pyramid, as mentioned in some Hacker News stories! Stay tuned.

Until then, keep pushing boundaries and exploring new frontiers. After all, every day is an adventure in the world of tech!

---

Brandon Camenisch
Engineering Manager & Platform Engineer