Back to Projects

Distributed Edge Cluster

Production-grade MLOps infrastructure running on a 20-node Raspberry Pi cluster with custom thermal management and distributed storage.

Raspberry Pi CM4KubernetesTalos LinuxRustNVMePoE+MLOpsDistributed Systems

Architectural Overview

This project originated from a requirement to simulate edge computing constraints (network partitions, hardware failures, and strict resource limits) within a physical environment. While cloud infrastructure abstracts these complexities, building resilient distributed systems requires confronting them directly.

The solution is a high-density compute cluster comprising 20 Raspberry Pi Compute Module 4 nodes, orchestrated by Talos Linux. Each node is equipped with NVMe storage and powered via PoE+, eliminating common bottlenecks associated with single-board computers. The entire infrastructure, including networking and management, occupies just 4.33U of rack space while operating silently in an office environment.

The cluster serves as the production backbone for my personal infrastructure, hosting RAG pipelines, vector databases, and distributed caching layers. It demonstrates that enterprise-grade architecture principles (immutable infrastructure, GitOps, and observability) can be effectively scaled down to edge hardware.

High-Level System Context

At a high level, the system bridges public cloud services with private home infrastructure. Cloudflare Zero Trust secures the entry point, while Vercel handles the frontend edge caching. The Kubernetes cluster itself sits safely behind the home firewall, connected only via secure outbound tunnels and GitOps synchronization loops.

High-Level System Context

C4 Level 1 Diagram

👤
End User
HTTPS Traffic
👨‍💻
Developer
GitOps & Maintenance
Vercel
Frontend Hosting
HTTPS (Frontend)
☁️
Cloudflare Edge
DNS, Security, Tunnel Endpoint
API CallsHTTPS (Backend)
🐙
GitHub
Source Code & Actions
Git Push / Kubeseal
Home Lab Infrastructure
🏠
Home Router
192.168.0.x
L2 Traffic (ARP)
Kubernetes Cluster
Talos OS • GitOps Managed
Cloudflared TunnelArgoCD Sync

Logical Cluster Topology

Internally, the cluster is strictly segmented by namespaces. Traffic enters via Cloudflared tunnels into the ingress namespace, where Traefik routes requests to application services.ArgoCD acts as the control plane, continuously reconciling the cluster state with the GitHub repository, while Actions Runner Controller (ARC) spins up ephemeral build agents on-demand for CI/CD workflows.

Detailed Cluster Architecture

C4 Level 2/3 Container View

☁️ Cloudflare Edge
Ingress Traffic
🐙 GitHub Repo
Manifests & Secrets
🏠 Home LAN
ARP Requests
⎈ Kubernetes Cluster
ns: networking & ingress
Cloudflared Pod
Tunnel Daemon
Traefik Proxy
DaemonSet
svc/traefik
LoadBalancer
ns: argocd
🐙
ArgoCD Controller
GitOps Operator
ns: metallb-system
MetalLB Controller
Layer 2 Mode (ARP)
ns: actions-runner-system
ARC Controller
Self-Hosted Runners
Build & Deploy
ns: apps (Application Namespace)
SQL API Pod
Application Logic
svc/sql-api
Data Tier
PostgreSQL
Redis Cache
Persistent Volume Claim (Local)

Physical Implementation

The physical layer is designed for density and efficiency. Twenty Raspberry Pi CM4 nodes are mounted on blade carrier boards, each with dedicated NVMe storage and PoE+ delivery. This 4.33U rack setup provides a completely self-contained compute environment with its own networking and power management context.

Physical Rack Layout (4.33U Total)

Office environment • ~400W compute power • Silent operation

Network Layer - 2U

UniFi
Dream Machine Pro (1U)
Router • Gateway • Controller • IDS/IPS
USW Pro 24 PoE (1U)
24-port PoE+ switch • Powers all 20 Pi nodes

Management Layer - 1.33U

Racknex Mount
Intel NUC #1
Proxmox Host
Intel NUC #2
Proxmox Host
Intel NUC #3
Proxmox Host
Ubuntu VM Jumpbox
kubectl • talosctl • GitOps • CI/CD

Compute Layer - 1U

Compute Blade
Pi 1
Pi 2
Pi 3
Pi 4
Pi 5
Pi 6
Pi 7
Pi 8
Pi 9
Pi 10
Pi 11
Pi 12
Pi 13
Pi 14
Pi 15
Pi 16
Pi 17
Pi 18
Pi 19
Pi 20
CM4 Modules
ARM64 • 8GB RAM
Storage
1TB NVMe per node
Power
PoE+ per node
Cooling
Noctua + Rust Control

Software Stack

GitOps-managed infrastructure via ArgoCD

Control Plane

  • • ArgoCD (GitOps)
  • • Sealed Secrets
  • • Kustomize
  • • GitHub Actions (CI/CD)
  • • Actions Runner Controller

Networking & Ingress

  • • Traefik (Ingress Controller)
  • • Cloudflared (Zero Trust Tunnel)
  • • MetalLB (L2 Load Balancer)
  • • IngressRoutes & Middleware
  • • Service Mesh Ready

Applications

  • • SQL API (Go)
  • • PostgreSQL
  • • Redis
  • • GHCR Registry Access
  • • Database Migrations

Observability

  • • Prometheus (Metrics)
  • • Grafana (Dashboards)
  • • AlertManager (Notifications)
  • • Custom Thermal Monitor (Rust)

Infrastructure

  • • Talos Linux (Immutable OS)
  • • Kubernetes 1.28+
  • • NVMe Local Storage (20TB)
  • • Distributed Block Storage

Network Architecture

  • • UniFi Dream Machine Pro (routing/gateway)
  • • USW Pro 24 PoE (24-port switch)
  • • Single cable per Pi (PoE+ power + data)
  • • Kubernetes CNI for pod networking
  • • Service mesh & network policies

Power & Thermal

  • • ~400W total power consumption
  • • PoE+ budget management (~25W/port)
  • • Custom Rust thermal controller
  • • Noctua fan curves (0-100% PWM)
  • • Silent operation in office environment

Engineering Decisions

01

GitOps-First Architecture

The entire infrastructure is declaratively defined in Git. ArgoCD continuously reconciles cluster state, while Sealed Secrets manages sensitive data encryption. Every change flows through version control. No manual kubectl apply commands in production.

02

Zero-Trust Networking

Rather than exposing services via port forwarding, Cloudflare Tunnels establish outbound-only connections. Combined with Traefik for internal routing and MetalLB for L2 load balancing, the cluster remains completely shielded from direct internet exposure.

03

Immutable Infrastructure

Talos Linux provides an API-driven, immutable OS with no SSH access. All configuration happens through declarative machine configs. This eliminates configuration drift and ensures consistent, reproducible node states across the entire cluster.

04

Self-Hosted CI/CD

Actions Runner Controller spins up ephemeral GitHub Actions runners directly on the cluster. This enables ARM64-native builds, secrets access via Sealed Secrets, and the ability to test deployments in the actual target environment.

Technical Challenges

Secret Management

Storing secrets in Git required Sealed Secrets with asymmetric encryption. The cluster holds the private key, while CI/CD pipelines encrypt using the public cert. Database credentials, API tokens, and tunnel configs are version-controlled safely without exposing plaintext values.

ARM64 Container Images

Many upstream projects lack ARM64 images. GitHub Actions builds multi-arch containers using buildx, pushing to GHCR. The cluster authenticates via sealed ImagePullSecrets. For custom apps, cross-compilation from x86 build agents targets ARM64 natively.

No-SSH Operations

Talos eliminates SSH entirely. Debugging requires talosctl and kubectl exclusively. Logs flow through aggregators, metrics to Prometheus, and configuration via machine configs. This forced adoption of cloud-native observability patterns from day one.

Stateful Workload Persistence

PostgreSQL and Redis require persistent storage across pod restarts. Each node provides local NVMe via PersistentVolumes. Database migrations run as Kubernetes Jobs before deployment, ensuring schema consistency while maintaining zero-downtime rolling updates.

Operational Metrics

4.33U
Total rack space
20TB
Distributed NVMe storage
~400W
Compute power draw
2U UniFi networking + 1.33U management (3 NUCs) + 1U Pi cluster (20 nodes)