Building a Kubernetes Cluster on Bare Metal: A 4th of July Hackathon

---

It was July 4, 2016, and I found myself sitting in front of my laptop, staring at the cluster I had just set up on our bare metal servers. The day started with a bit of a firework display outside (literally), but what I was really celebrating was this little Kubernetes cluster I’d been working on for weeks.

Kubernetes had already won the container wars—Docker Swarm and Mesos were both losing ground to it. But setting up a production-grade, bare metal cluster wasn’t exactly straightforward. We were still running CentOS 6, which made things even more challenging.

I had spent most of the morning debugging the latest version of kubeadm, trying to get everything to work perfectly. Kubernetes 1.3 was out and with it came some much-needed improvements, but also a few gotchas that I needed to iron out.

The Setup

We decided to go with a three-node cluster: one master node and two worker nodes. Setting up the master node was relatively simple—once you get past all the yum updates and the fact that kubeadm only works on CentOS 7, it’s fairly straightforward. But setting up the workers took some extra effort.

Each worker node needed to have specific packages installed and configured correctly. I spent a good part of my morning getting cni-plugins-linux-amd64-0.2.0.tar.gz to unpack properly into /opt/cni/bin. It was a bit of a headache, but eventually, it worked.

Debugging and Troubleshooting

Once everything was set up on the master node, I ran kubeadm init, which went off without any issues. However, when I tried to add the first worker node with kubeadm join, things started to fall apart. The logs were cryptic, and it took a few hours of digging through error messages before I realized that I had missed adding the --apiserver-advertise-address flag during the initial configuration.

Fixing this was just the tip of the iceberg. After getting past that issue, I encountered another one: network connectivity between nodes. Our bare metal servers were all on different subnets, and setting up the CNI plugin to handle the networking was a bit tricky. In the end, it required some manual configuration in ifcfg files.

The Joy of GitOps

As we got closer to having a fully functioning cluster, I started thinking about how to manage our infrastructure going forward. We had been using Puppet for most of our configuration management, but with Kubernetes becoming more mainstream, it seemed like the perfect time to explore GitOps principles.

We decided to start small by storing our Kubeconfig files and manifests in GitHub. This way, we could use kubectl apply directly from our CI/CD pipeline to ensure that everything stayed up-to-date. It was a bit of a learning curve for us, but it felt like the right direction to go.

A Thoughtful Reflection

As the sun began to set over my hometown, I sat back and took stock of what we had accomplished. We now had a functional Kubernetes cluster running on bare metal servers, which was a significant achievement in itself. But more importantly, we were starting to think about how we could leverage modern DevOps practices to improve our infrastructure management.

Looking at the Hacker News stories from that month, I felt a bit of déjà vu. The web development and security articles reminded me of the challenges we faced with maintaining our old applications. Meanwhile, the discussions around new tools like Terraform 0.x made me wonder how we could use these technologies to build more robust infrastructure.

In many ways, July 4th, 2016, was a day of progress, but also a reminder that there’s always more work to be done. As I watched the fireworks, I knew that this was just the beginning—Kubernetes would continue to evolve, and so would we.

---

This post is a reflection of my personal experience setting up a Kubernetes cluster in 2016. The journey wasn’t easy, but it felt like the right move for our organization at the time.