November 13, 2023. As I sit down to write this, the tech world feels like it’s been tossed into an emotional and technical whirlwind. From the boardroom dramas at OpenAI to the unexpected twists in the saga of Sam Altman, it’s hard not to feel a mix of amusement and bewilderment.

AI and the Great Reorganization

The past month has seen significant changes within the AI landscape. With OpenAI’s board making waves by firing Sam Altman, it’s clear that the company is undergoing a major realignment. While I don’t work directly with AI models, I can’t help but reflect on how these organizational shifts affect our tech stack and operational challenges.

In my role as an engineering manager, one of the biggest impacts has been in the realm of infrastructure management for large language models (LLMs). Post-ChatGPT, there’s a surge in demand for more robust LLM infrastructures. This means dealing with increased load on our servers, managing cold starts, and optimizing training cycles—tasks that have become even more complex as we scale.

Platform Engineering and FinOps

Platform engineering has continued to gain traction this month, with FinOps becoming an increasingly crucial discipline. As cloud costs continue to pressure budgets, teams like mine are forced to scrutinize every line of code and every service deployed on our platforms. DORA (DevOps Research and Assessment) metrics have become a standard way to measure how well we’re delivering value.

One recent project that really brought this into focus was optimizing our CI/CD pipelines for cost efficiency. We had several services running 24/7, which were not only unnecessary but also eating up our budget. After some intense discussions and experiments, we managed to reduce our cloud bill by over 30%. It wasn’t easy; it required a lot of debugging and code refactoring, but the results have been worth it.

WebAssembly on Servers

WebAssembly (Wasm) is still in its early stages for server-side applications, but I’ve started experimenting with using it to offload some of our compute-heavy tasks. For example, we’re exploring ways to use Wasm to process large data sets more efficiently without the overhead of traditional VMs or containers.

One major challenge has been finding a balance between performance gains and ease of deployment. Wasm is still niche compared to more established technologies like Node.js or Python, but as it matures, I believe it will play an increasingly important role in our tech stack.

Developer Experience and Platform Engineering

The developer experience (DX) team has become a crucial part of platform engineering. We’re not just about writing code anymore; we’re also focused on making sure that the entire development lifecycle is smooth and efficient for our engineers. Tools like GitHub Copilot have made coding faster, but it’s still up to us to set up everything in a way that encourages productivity.

Recently, we’ve been working on integrating some new tools to improve DX. One of them is an auto-configuration tool for our Kubernetes clusters. It significantly reduced the time developers spend setting up their environments from scratch, allowing them to focus more on coding and less on configuration. This has led to a noticeable increase in developer happiness.

Conclusion

Reflecting on this month, it’s clear that change is happening fast in tech. The events at OpenAI and the various boardroom dramas have been fascinating but also somewhat distracting. As for me, I’m focused on making our systems more efficient, more scalable, and easier to maintain—tasks that require constant learning and adaptation.

Stay tuned for what’s next in this ever-evolving landscape!

---