$ cat post/dial-up-tones-at-night-/-the-endpoint-broke-on-staging-/-the-stack-still-traces.md
dial-up tones at night / the endpoint broke on staging / the stack still traces
Title: On the Cusp of Chaos: December 2008
December 2008 was a month where everything felt like it was about to change. I remember it well because that’s when we were deciding whether our startup should move entirely into the cloud or stick with colocation. The world of tech was in flux, and everyone seemed to be trying to figure out what would stick.
Our team had been working on a SaaS product for almost two years, but things were starting to get hairy. We had some issues with uptime that were driving our customers crazy. Our ops guy, let’s call him Dave (who is actually a super smart guy), was banging his head against the wall trying to figure out how we could scale without spending an arm and a leg.
GitHub just launched, and it was all the rage among developers. The idea of version control being accessible on the web seemed revolutionary at the time. But I couldn’t help but feel that GitHub was like the shiny new toy that everyone wanted to play with. In our world, we were still grappling with SVN. Git adoption was spreading fast, but not yet mainstream enough for us to fully embrace it.
Meanwhile, Amazon Web Services (AWS) EC2 and S3 had become must-haves in any developer’s toolkit. I remember the excitement when a colleague got up and said, “Why don’t we try running our app on EC2? We can save money and scale better.” But the reality was that AWS was still quite new and unreliable at times. Our IT department wouldn’t even consider it because of the potential risk.
Dave had set up an initial test environment on AWS, but things were going sideways. One day, we woke up to find our application down with no logs or error messages. It was like a puzzle where every piece needed to be in just the right spot for everything to work smoothly. After hours of digging through Amazon’s forums and Stack Overflow, we finally tracked it down to a misconfigured security group rule. Once that was fixed, things started humming along much better.
But as much as AWS had shown promise, our internal debate over cloud vs. colocation continued. The economic crash was hitting us hard, and budgets were being slashed across the board. We had to weigh the cost of moving entirely to AWS against the potential savings and flexibility it offered versus staying in a colo where we already had infrastructure set up.
One night, I found myself arguing with our CTO about this decision. He wanted to go all-in on cloud, but I was still skeptical. “What if something goes wrong again?” I asked, pointing out that AWS wasn’t as stable as we needed it to be for a mission-critical application. But he argued back that the potential benefits were too good to pass up, especially given our tight budget constraints.
The next morning, I woke up and read about So You Want to Be a Consultant? on Hacker News. It felt like a stark reminder of how much the tech landscape was changing. Suddenly, everyone was a consultant because they had “ideas” or wanted to start their own startups. The world seemed to be filled with opportunities, but also with people who were taking the easy way out.
Looking back, I realized that our decision back then was influenced by all these factors. We decided to stay in colocation for now, leveraging AWS only as needed. It wasn’t an easy choice, and we knew it was a temporary fix. But sometimes, you have to make do with what’s available until something better comes along.
That’s the kind of thinking that goes into making decisions during such turbulent times. You weigh the pros and cons, take risks where necessary, and hope for the best. And in December 2008, we were just trying our best to keep everything running while navigating through this rapidly changing landscape.