Kubernetes Homelab

Overview

I run this eight-node bare-metal Kubernetes cluster for personal projects and services used by family and friends. It hosts this website, media and photo libraries, productivity tools, and local AI inference.

The cluster has three groups of machines: control and general compute, storage and I/O, and GPU workers. Labels, taints, and scheduling rules keep ordinary applications on the compute nodes. Workloads need explicit placement rules to use the storage or GPU nodes.

Since taking on clients, I use separate dashboards rather than a single view of everything. The image above shows infrastructure services and node status.

Hardware

• Control and compute: three Ryzen AI 9 HX 370 systems, each with 96 GiB of physical memory, Radeon 890M integrated graphics, and 2.5 GbE. These run the Kubernetes control plane, application services, and AI frontends and gateways. The intended memory allocation is 48 GiB each for the system and integrated GPU.
• Storage and I/O: three Ryzen 7 7745HX systems, each with 64 GiB of memory, four dedicated NVMe Ceph OSDs, and 10 GbE. Together, the twelve OSDs provide approximately 36 TB of raw capacity. Databases and other storage-heavy services also run here.
• GPU workers: two NVIDIA GB10 Grace Blackwell systems, each with a 20-core ARM64 CPU, 128 GB of unified memory, local NVMe, and 10 GbE. A private 200 Gb interconnect links the pair for distributed inference.

The compute nodes run Arch Linux, the storage nodes run NixOS, and the GPU workers run DGX OS on Ubuntu. Kubernetes is bootstrapped with kubeadm; kube-vip provides the shared API endpoint.

Local AI

OpenWebUI provides the chat interface, LiteLLM handles model routing, and vLLM runs a GLM model across the two GB10 workers. Authentik supplies login through OIDC. Model serving runs on the GPU workers while the frontend and gateway run on general compute.

The NVIDIA GPU Operator manages device access. Its time-sliced resource claims allow workloads to share a GPU; they do not provide isolated GPU memory. Placement and memory allocation still need to account for the model loaded on each worker.

Edge and access

OPNsense forwards public HTTP and HTTPS traffic to HAProxy Ingress, which has a LAN address assigned by MetalLB. The Kubernetes API and storage services have no WAN forwarding rules. Split DNS serves internal clients, and Headscale provides private remote access.

Public ingresses declare whether traffic arrives directly or through Cloudflare, along with an approval label and TLS configuration. An admission policy checks these declarations. Restricted ingresses accept LAN and Headscale sources; Cloudflare origins restrict traffic to Cloudflare and private networks.

Cilium provides pod networking, replaces kube-proxy, and exposes network flows through Hubble. Linkerd supplies mTLS and access policy for meshed workloads. Authentik handles OIDC and forward authentication for integrated applications, with separate administrative recovery access where supported.

Storage and state

Rook/Ceph offers several storage profiles: three-way replicated block storage for critical data, two-way block storage for applications that replicate their own data, three-way replicated CephFS for shared files, and 2+1 erasure-coded CephFS for bulk data such as media and model caches. Usable capacity depends on the profile; the 36 TB figure is raw storage.

Five three-instance CloudNativePG clusters separate AI, media, platform, productivity, and other databases. Valkey has separate cache and persistent queue tiers. An internal three-peer Qdrant cluster supports vector database testing.

CSI snapshots are configured, but off-cluster backup work is unfinished. VolSync and PostgreSQL backup definitions remain disabled pending a replacement object store. Replication and local snapshots do not cover loss of the whole cluster.

Operations

Argo CD reconciles platform and application configuration from my self-hosted Gitea repository. GitHub is a downstream mirror. Node setup, kubeadm upgrades, and the initial networking and GitOps bootstrap remain separate operating tasks.

Harbor stores application images and mirrored dependencies. Renovate proposes updates, and an artifact promotion job copies the referenced images and charts into Harbor. Vault holds runtime secrets; External Secrets Operator projects them into the namespaces that need them.

Prometheus, Alertmanager, and Grafana cover metrics and alerts. Fluent Bit, Data Prepper, and OpenSearch collect and index logs. I maintain the machines, upgrades, storage, networking, and deployment configuration, including separate staging and production deployments for this site.