Title: Cloud Costs Are a Pain in the Wallet

---

January 24, 2022. It’s been another year since I started my role as an Engineering Manager at a mid-sized tech company, and with it came the usual mix of challenges and triumphs. But one thing that’s become a constant source of frustration is cloud cost management.

This month, I hit rock bottom when a random spike in our AWS bill showed up in my inbox, nearly doubling our monthly spend without any clear explanation. “I got pwned by my cloud costs,” I wrote to the team Slack channel, trying to keep it light. The response was immediate and mostly supportive, but also an overwhelming mix of sympathy and confusion.

For context, around this time, FinOps was starting to become a buzzword, and while our finance team had been tracking our spending for some time, they were not involved in the day-to-day engineering decisions that drive costs. In fact, it felt like the cloud provider’s pricing models were as complex as the algorithms driving ChatGPT.

The spike turned out to be due to an improperly configured AWS Lambda function running unoptimized code—a classic case of “serverless gone wrong.” But it also highlighted a larger issue: our infrastructure was becoming harder to manage and optimize. As we scaled our platform engineering efforts, the complexity grew exponentially, and with it, the risk of overspending.

I decided to dive into some old logs and configurations to try to figure out what went wrong. After an hour of digging through cloudwatch and Lambda metrics, I found a clue: there was an increase in traffic that I hadn’t seen before. A quick scan of our load balancer revealed a surge in requests from a single IP address—clearly an anomaly.

Upon further investigation, it turned out that we had accidentally set up an API Gateway endpoint to serve as a direct proxy for a third-party service. This was supposed to be a temporary measure while we developed and deployed a custom solution. However, the developer who set this up didn’t document the configuration properly, leading to the unoptimized Lambda function running behind it.

Fixing the issue involved several steps:

1. Deleting the API Gateway endpoint: Simple enough in theory but not always straightforward with AWS.
2. Scaling down the Lambda functions: We had to manually decrease the concurrency limits and scale them back to their baseline state.
3. Optimizing the code: A quick code review showed that some of our Lambda functions were doing unnecessary work, so I refactored them to run more efficiently.

While these steps reduced our costs significantly, they also highlighted a deeper issue with our development process. We needed better documentation and governance around resource management, especially for infrastructure as code (IaC) templates.

This experience reinforced the importance of FinOps practices in engineering teams. As DORA metrics became widely adopted, I realized that we were not just shipping features; we were also managing a growing expense. The shift towards platform engineering had given us more tools and responsibilities, but it hadn’t yet fully addressed the financial implications of our work.

Looking ahead, I plan to push for more integration between development, operations, and finance teams. We need better visibility into costs and automated alerts when spending exceeds certain thresholds. And perhaps most importantly, we need to foster a culture where every engineer is responsible for not just writing code but also understanding its impact on the bottom line.

As I reflect on this episode, I’m reminded of the blog post “I got pwned by my cloud costs,” which resonated with many others who have faced similar issues. It’s a reminder that even in an era where technology is advancing at breakneck speed, basic principles of good practice and transparency remain crucial.

---

This isn’t about avoiding complexity or underutilizing modern tools; it’s about finding the right balance between innovation and responsibility. After all, as we continue to build more sophisticated systems, the costs and challenges will only grow. But with a little more mindfulness, we can ensure that our tech advancements benefit both our users and our wallets.

---