I’ve spent the last 6 months building two production-grade Kubernetes platforms from scratch — not labs, not tutorials, but environments that run real workloads 24/7, handle real traffic, and solve real operational problems.
Here’s what that looks like under the hood.
PLATFORM 1: GKE Terraform Full GitOps
I designed a complete GCP environment around Google Kubernetes Engine, provisioned entirely with modular Terraform. The network is a custom dual-stack VPC (IPv4/IPv6) with private subnets, strict NetworkPolicies, and zero public ingress for internal services.
The cluster runs three distinct node pools:
• Standard e2-standard-2 nodes for general workloads
• SPOT L4 GPUs for low-latency inference
• SPOT RTX 6000 Pro nodes for heavy reasoning workloads
Everything ships through Git. FluxCD bootstraps the entire stack — infrastructure controllers, application manifests, secrets, and Helm releases. Kustomize manages overlays. SOPS encrypts every secret with Age, and Flux decrypts them during deployment automatically.
The application layer is a multi-tenant AI platform (AgentForge) with dual-tier vLLM inference:
• Tier 1 — L4 SPOT 24/7 running a Qwen coder model for fast tool-calling triage
• Tier 2 — RTX 6000 Pro SPOT, KEDA scale-to-zero, running a larger Qwen model with fp8 KV cache for multi-file reasoning
Each tenant gets isolated Hermes agent profiles, dedicated PostgreSQL databases, and independent observability. I also run a FastAPI waitlist API, an autonomous news quant pipeline that scrapes 371 feeds, N8N automation, and automated local-business web deployments.
The data layer is a CloudNative PG fleet: four HA PostgreSQL clusters across seven databases, automated GCS backups, 7–30 day retention, and failover. Strimzi Kafka handles event-driven workloads.
Security isn’t afterthought — it’s architecture:
• Cilium CNI with eBPF datapath Hubble L7 observability
• Strict NetworkPolicies namespace isolation
• Tailscale-only access for monitoring and databases
• Automated security pipeline: a sec-ops agent scans for CVEs and misconfigurations, confirmed findings auto-dispatch a backend-dev agent to patch them — audit to fix in one GitOps cycle
• Every container runs non-root with enforced resource limits
Cost discipline is built in: 60–90% GPU savings through SPOT instances, KEDA scale-to-zero, scheduled CronJobs, and strict CPU/memory caps.
PLATFORM 2: Kubeadm Multi-Cloud Advanced Networking
My second cluster is completely different self-hosted with kubeadm across 3 nodes (1 control plane, 2 workers), giving me direct control over the entire stack.
The networking story here is the highlight. Cilium 1.16.6 is configured with:
• BGP control plane for dynamic route advertisement
• L2 announcements for cross-node LoadBalancer failover
• A shared LoadBalancer IP pool for high availability
This cluster runs diverse, real workloads:
• Zcash blockchain node (zebrad) lightwallet with StatefulSets and gRPC/P2P
• Prosody XMPP server handling C2S, S2S, and HTTPS
• Linkding bookmark manager with persistent storage
• Multipaper Minecraft server cluster: master Velocity proxy MariaDB (50Gi) plugins
The data layer includes a 3-node HA CloudNative PG cluster, dedicated MariaDB for Minecraft, and Local Path Provisioner for dynamic PVs.
GitOps is identical in philosophy to the GKE lab: Flux 2.7.2, Kustomize overlays, Renovate for automated dependency updates, SOPS-encrypted secrets, Hubble observability, and AlertManager for routing.
What I’ve Learned
These two platforms taught me something most tutorials can’t: the difference between “I followed a guide” and “I own this system.”
The GKE lab taught me platform engineering at scale — how to optimize inference costs, design multi-tenant isolation, and build self-healing security pipelines.
The kubeadm lab taught me what’s under the hood — control plane components, CNI internals, BGP in Kubernetes, and how to make bare-metal-style networking