Debugging the Realms of Apollo

June 5th, 2023

---

Title: Debugging the Realms of Apollo

It’s been a wild ride so far this year in tech. AI and LLM infrastructure have exploded post-ChatGPT, platform engineering has gone mainstream, and FinOps with cloud cost pressures are everywhere. The CNCF landscape is overwhelming, WebAssembly on the server side is taking shape, and DORA metrics are being adopted widely. Developer experience as a discipline continues to gain traction, and staff+ engineering tracks have become normalized.

Today, I want to share some thoughts on something near and dear to my heart: Apollo, which will close down on June 30th. This has hit me like a tidal wave, especially since I spent so many hours (and nights) in its forums. It’s a stark reminder of the ever-changing nature of tech.

The Dawn of Debugging

A few weeks ago, we started noticing some weird behavior with Apollo’s GraphQL infrastructure. We were seeing requests timing out and errors that didn’t seem to make sense. I rolled up my sleeves and dove into the logs, hoping to find something I hadn’t noticed before.

After a couple of hours of digging, I found an interesting pattern: there was a sudden surge in requests coming from a single IP address every night around 2 AM. This was strange because Apollo had a robust rate limiter that should have caught this traffic. But somehow, it was slipping through.

I took a closer look at the logs and realized that the request payloads were being truncated, which would explain why the rate limiter wasn’t catching them. I dug into the code and found that there was an issue with how the request body was being read in the Apollo server setup. Specifically, the bodyParser middleware for Express was configured incorrectly.

Code Snippet: Incorrect Middleware Configuration

const express = require('express');
const bodyParser = require('body-parser');

const app = express();

// This is incorrect and leads to issues with request bodies
app.use(bodyParser.json({ limit: '1mb' }));

// The correct configuration should be:
app.use(bodyParser.json({ type: ['application/json'], limit: '1mb', verify(req, res, buf, encoding) {
  if (buf.length > this.limit && !req.is('application/x-www-form-urlencoded')) {
    return false;
  }
} }));

Once I fixed the middleware configuration and restarted the server, the issues were resolved. The truncation of request bodies was no longer happening, and our rate limiter started catching the offending traffic.

Lessons Learned

This experience reminded me that even in mature frameworks like Apollo, subtle issues can still cause big problems. It’s important to stay vigilant and not assume that everything is working as expected just because it has been for a long time.

It also highlighted the importance of proper logging and monitoring. If we hadn’t had comprehensive logs and real-time monitoring set up, this issue might have gone unnoticed much longer.

The Closure of Apollo

As I write this, the end date for Apollo looms. It’s a bit bittersweet knowing that so many developers who relied on it are about to lose a platform they grew to love. But with every change comes new opportunities and challenges. We should take this as an opportunity to reflect on what worked well and what didn’t in our projects.

For those of us moving away from Apollo, there’s no shortage of alternatives. GraphQL is still the future, and there are plenty of robust implementations out there like AWS AppSync or even custom solutions using Apollo Server itself.

The Future of Developer Tools

The tech landscape continues to evolve rapidly. As we move forward, it will be interesting to see how developer tools adapt to these changes. Whether it’s WebAssembly on the server side or new FinOps practices, staying ahead requires constant learning and adaptation.

For now, I’m focusing on making sure my team is well-prepared for whatever comes next. After all, as a platform engineer, part of my job is not just debugging issues but also planning for the future.

---

In closing, Apollo has been more than just a tool; it’s been a community and a home. As we say goodbye, let’s take this experience to heart and continue to build better tools and practices together.