Navigating the LLM Tsunami

August 5, 2024

Today feels like a whirlwind. The past few months have been dominated by AI/LLM infrastructure explosions, platform engineering’s rise to mainstream status, and a constant barrage of new tools and technologies that make me feel like I’m trying to navigate an uncharted sea.

The LLM Tsunami

Since the ChatGPT fiasco, it seems every company is scrambling to either build their own AI wonder or leverage existing frameworks. We’re seeing everything from bare-bones APIs for generating text snippets to full-blown chatbots that promise to revolutionize customer service and beyond. Our infrastructure team has been swamped with requests to handle new model deployments, storage requirements, and compute power. It’s like we’re in the eye of a hurricane, with every team clamoring for more resources.

Platform Engineering Mainstream

Platform engineering is no longer just a buzzword; it’s real. I find myself spending more time working on abstractions that make it easier for other teams to build applications without worrying about underlying infrastructure. This involves a lot of work in Kubernetes, Helm charts, and Operator patterns. The goal is simple: reduce friction so developers can focus on their core business logic instead of plumbing. It’s a bit like setting up a series of conveyor belts to move raw materials through a factory; the key is making sure everything flows smoothly.

CNCF Landscape Overwhelming

The Cloud Native Computing Foundation (CNCF) landscape has become overwhelmingly complex. With every month, new projects are announced, and existing ones seem to evolve at breakneck speed. It’s hard to keep up with all the changes, but it’s essential for staying relevant. We’ve been investing in newer technologies like WebAssembly running on servers—a bit of a niche area right now, but one that could become mainstream as more organizations start looking for ways to optimize their compute environments.

Developer Experience Discipline

Developer experience (DX) is now its own discipline, and I’m seeing it permeate every aspect of our work. It’s about making sure developers have the tools they need, a consistent environment, and clear documentation. We’re using CI/CD pipelines more than ever, with a focus on DORA metrics to ensure we’re delivering software faster and with higher quality. One recent project involved setting up a continuous delivery pipeline for an AI model deployment, which was both exciting and frustrating due to the complexity of integrating various components.

FinOps and Cloud Cost Pressure

FinOps is finally becoming a reality in our organization. We’re no longer just developers; we’re also financial stewards. The pressure to optimize cloud costs is real, and it’s pushing us to rethink how we use resources. This involves not only monitoring usage but also making strategic decisions about which services to use based on cost-benefit analysis. It’s a constant battle between saving money and ensuring the performance of our applications.

Debugging Real-World Problems

One recent challenge involved debugging an issue with our AI model serving endpoint. The application was running fine in development, but as soon as it went live, we started seeing anomalies in the response times. After hours of profiling and analyzing logs, I realized that the bottleneck was due to excessive garbage collection in our JVM—something that didn’t show up in local testing environments. We fixed this by tweaking the garbage collector settings and adding some monitoring around resource usage.

Learning from Real Projects

On a side note, one project we worked on involved building a web application using Mono (a .NET framework) for a customer. The idea was to offer an open-source alternative to proprietary software. It was a bit like starting a marathon with a full backpack—exciting and challenging all at once. We had to deal with some unexpected issues, but the experience taught us valuable lessons about cross-platform development and community engagement.

Conclusion

Looking back on this month, it’s clear that tech is moving faster than ever before. The challenges are real, but so are the opportunities. As I write this, I’m already thinking about the next big thing—whether it’s a new AI framework or an innovative platform tool—that will define the landscape in the coming months.

Stay tuned for more updates!

---

That wraps up my thoughts for August 5, 2024. The tech world is always changing, and being part of that change is both exhilarating and humbling.