$ cat post/ps-aux-at-midnight-/-i-traced-it-to-the-library-/-a-ghost-in-the-pipe.md

ps aux at midnight / I traced it to the library / a ghost in the pipe


Title: When Perl Was Still Sexy and I Spent an Entire Day on a Bug


July 3, 2006. The air was thick with the scent of open-source and the faint, almost nostalgic whiff of what would soon be considered old-school tech. We were in the midst of Web 2.0’s early days—Digg was gaining ground, and Reddit was just a twinkle in some Perl hacker’s eye. I remember the smell of freshly printed papers from our local Linux user group meetings, the hum of servers in the basement, and the ever-present buzz of conversation around Google’s aggressive hiring spree.

The Bug That Bit Me

It started like any other day. Our team was working on a small feature for one of our applications—a simple form submission that needed to handle some heavy lifting behind the scenes. We were using Perl as our main language, with MySQL for storage and Apache for web serving. A typical stack back then.

We’d been working late again, trying to squeeze in just enough time to make sure everything was ship-shape before launch. The codebase had grown organically over months—lots of helper scripts, a bit of Perl magic here and there, and some Python thrown in for good measure. As the clock ticked closer to midnight, I decided to do what any sane person would: dive into the latest commit logs just to double-check our work.

The Oh-So-Simple Bug

One line stood out:

if ($request->param('action') eq 'submit') {
    # Process form data
}

It seemed so innocuous. The eq operator was there, comparing the value of action with a string literal. I double-checked and triple-checked, but everything looked fine. So why was it breaking?

After hours of digging, I found myself staring at a stack trace that made no sense. Something about how the data from the form submission wasn’t being processed correctly. I tried everything: re-wrote parts of the code, added logging everywhere—nothing worked.

The Perl Witch Hunt

The more I thought about it, the more convinced I became that there must be some sort of subtle issue with our custom Apache module that handled form data. We had a few modules in place to handle different types of requests, and they all interacted through a shared context object. Could something be going wrong there? I spent another hour or so tracking down the context object, making sure it wasn’t being mangled.

Finally, as the clock hit 3 AM, I decided to take a step back and look at the entire request flow again from start to finish. That’s when I noticed something glaringly obvious but initially overlooked: the param method of the CGI module returns a reference to an array if there are multiple values for the same parameter.

The Fix

So simple, yet so frustrating. I had been treating it as a scalar value throughout my code, assuming that each form submission would only have one action. Changing this line:

if ($request->param('action') eq 'submit') {

To:

if (grep { $_ eq 'submit' } @{$request->param('action')}) {

Fixed the issue in a matter of minutes.

Lessons Learned

That night taught me a valuable lesson about assumptions. Sometimes, even with all our experience and knowledge, we can miss things that are right in front of us. It’s easy to get tunnel vision when you’re under pressure, but stepping back can often reveal what you’ve been overlooking.

Perl might not be the most glamorous language these days, but it still has its moments. And sometimes, even after all these years, those moments can still teach us something new.

A Parting Thought

As I lay down to sleep that night, exhausted yet oddly content, I couldn’t help but think about how much our tools and technologies have changed since 2006. But one thing remains the same: the frustration of a bug that just won’t go away no matter how many times you check your code.

And so, back then and now, I’ll keep working through those bugs, pushing forward with whatever tools are at my disposal.