It’s October 26th, 2020. As I sit down to write this post, Kubernetes has been the go-to container orchestration platform for a few years now. But lately, it feels like everyone is starting to question its complexity—especially when it comes to managing state in a scalable and maintainable way.

The Complexity of State Management

Let’s dive into a recent experience that exemplifies this fatigue. We had been using Kubernetes for our application deployment needs for quite some time now, but as the team grew, so did the complexities. Specifically, we were dealing with persistent storage—managing stateful services like databases and caches. This required an intricate dance of PVCs (PersistentVolumeClaims), PVs (PersistentVolumes), and the ever-daunting StatefulSets.

One day, I found myself in a meeting where someone asked, “Why do we need so many different objects to manage our database’s state?” It was a fair question, but it also made me realize that managing this complexity wasn’t getting any easier. The solution involved YAML files, annotations, and complex CRDs (Custom Resource Definitions) that were hard to read and maintain.

Enter eBPF

Just as I was feeling this frustration, eBPF began to emerge from the shadows of its niche beginnings. I remember following some blog posts about how it could be used for tracing and performance optimization without modifying any code. The idea of being able to write tiny programs that run in the kernel space, which could inspect and manipulate network packets or even filesystem operations, seemed like a game changer.

So, I started looking into eBPF more seriously. It felt like a breath of fresh air compared to Kubernetes’ complexity. There was something liberating about not having to deploy custom controllers or configure complex YAML files just to add logging or trace requests. The simplicity and power of eBPF appealed to me, and it seemed like the perfect tool for some low-level network tasks we were working on.

Learning eBPF

I decided to take a deep dive into eBPF by setting up a small project to instrument our network traffic using BCC (BPF Compiler Collection). The first step was to get K8s nodes running BCC. This involved modifying the init scripts and ensuring that the necessary kernel modules were loaded. Once I had everything set up, I started writing some simple eBPF programs to trace API calls and log relevant data.

The experience was enlightening. It’s one thing to read about eBPF in a blog post; it’s another to actually write your first BPF program. The code was surprisingly straightforward once you got the hang of it. For example, writing a simple bpf_tracepoint program to trace HTTP requests looked something like this:

#include <linux/bpf.h>
#include <linux/in.h>

int http_request_handler(struct pt_regs *ctx) {
    struct iphdr *iph;
    
    // Extract the IP header from the context
    ph = (struct iphdr *)ctx->ax;

    // Print out some basic information about the request
    bpf_trace_printk("Received HTTP request from %pI4\\n", &iph->saddr);

    return 0;
}

The Future of Infrastructure

As I look back on this month, it’s clear that there is a growing sentiment around Kubernetes complexity. While Kubernetes has been a game-changer for container orchestration, the sheer number of moving parts and the need for deep understanding to manage stateful services can be overwhelming.

On the other hand, technologies like eBPF offer a more lightweight approach to solving some common infrastructure challenges. They provide a way to add functionality without introducing additional complexity in the form of new abstractions or complex YAML files.

For now, I’m still working with Kubernetes for most of our needs, but eBPF has definitely piqued my interest. It’s a reminder that sometimes, simpler isn’t always worse; it just requires a different mindset and tools. In an era where tech is constantly evolving, keeping an open mind to new solutions can help us find better ways to do things.

Until next time, Brandon