$ cat post/sudo-bang-bang-run-/-we-documented-nothing-then-/-a-ghost-in-the-pipe.md

sudo bang bang run / we documented nothing then / a ghost in the pipe


Title: July 7, 2008 - A Year of Turbulence in Tech


July 7, 2008

This time last year, I was part of a small team at a startup that was trying to make waves with a cloud-based solution for managing server infrastructure. It was a challenging but exciting period full of twists and turns.

First off, let me take a moment to acknowledge the broader tech landscape at the time. GitHub’s launch in 2008 was already shaking things up, making version control more accessible than ever. The iPhone SDK was just around the corner, which would dramatically change how people interacted with technology on their mobile devices. Hadoop was gaining traction as a way to handle massive amounts of data, and cloud services like AWS EC2 and S3 were becoming the new norm for scalable computing. Colocation data centers were still hanging on but seemed less sexy compared to the promise of cloud.

In my day-to-day, I was grappling with a mix of exhilarating projects and frustrating bugs. One particular issue that haunted me for weeks involved our custom build process for an application running on AWS EC2 instances. The app needed to be deployed across multiple regions, and we were using S3 buckets for storing the deployment artifacts. The problem? Our script would sometimes fail to upload files due to transient network errors, leaving us with half-deployed instances that just wouldn’t start properly.

I remember spending countless hours trying to figure out a solution. I tried implementing exponential backoff in our scripts, but even that didn’t solve all the issues. It was frustrating because we knew the basic approach worked in other contexts, but there seemed to be some edge case with AWS’s API that we just couldn’t nail down.

One evening, as I was walking home from work and pondering this problem, a colleague of mine suggested using S3’s multipart upload functionality. This allowed us to break up large files into smaller chunks, upload them individually, and then stitch them back together on the instance side. It sounded like overkill for our needs, but desperation drove me to try it.

To my relief, it worked! The script started failing much less frequently, and even when it did fail, we had a much better chance of recovering from transient network issues by simply retrying the failed parts of the upload process.

This experience taught me the importance of breaking down complex tasks into simpler, more manageable pieces. It also highlighted how often we take basic building blocks for granted—like S3’s multipart upload feature—and might miss out on powerful solutions if we don’t explore them fully.

On a related note, I remember reading about the AWS S3 outage in July 2008, which made me appreciate the stability of our implementation even more. It was a stark reminder that no matter how robust your system is, you can’t control everything, and there’s always room for improvement.

The startup world was also bustling with activity back then. Startup Ideas We’d Like to Fund got me thinking about where we were in the pipeline—were we cutting it close or did we have a good chance of making it? The idea of selling our software for $20,000 hit home; I wondered if that was an option if things didn’t pan out as expected. But at the same time, Randy Pausch’s passing made me think about the legacy and impact you want to leave behind.

As we entered July 7, 2008, it felt like a critical juncture for our company—would the bugs be fixed in time for the next release? Would our idea finally take off or fade into obscurity? The tech world was full of excitement and uncertainty, much like our own little startup journey.


This piece is just a slice of life from July 2008. It’s raw, it’s real, and it reflects the challenges and lessons we faced during that time. Tech moves fast, but some things remain constant—like the need to debug, the importance of breaking down problems, and the impact one can have through perseverance.