Kubernetes Flavor Decision

Decision: k3s (confirmed)

k3s is the best fit for Lab. OpenShift and most other flavors conflict with the puppet-managed, multi-arch, lightweight approach.

Evaluation

Flavor	Puppet-Friendly	ARM	Multi-arch	Enterprise	License	Verdict
k3s	✓ binary + config files	✓	✓	Rancher/SUSE	Apache 2.0	CHOSEN
k0s	✓ single binary, config-driven	✓	✓	Mirantis	Apache 2.0	Good alternative
kubeadm	✓ well-understood bootstrap	✓	✓	Upstream K8s	Apache 2.0	Viable but heavier
RKE2	✓ config files	✓	✓	Rancher/SUSE	Apache 2.0	Heavier k3s
OpenShift	✗ operator-driven, fights puppet	✗ limited	✗ limited	Red Hat	Proprietary	REJECTED
MicroK8s	⚠ snap-based, puppet+snaps awkward	✓	✓	Canonical	Apache 2.0	Not great
Talos	✗ immutable OS, no SSH, no puppet	✓	✓	Sidero Labs	MPL 2.0	Incompatible

Why NOT OpenShift — Deep Analysis

OpenShift Does Overlap With Lab

OpenShift is the closest existing thing to what Lab does. The overlap is real:

Capability	OpenShift	Lab
Manages nodes end-to-end	Yes (RHCOS + MCO)	Yes (OpenVox + labels)
Immutable infrastructure	Yes (rpm-ostree, operator-driven)	Yes (puppet convergence)
Fights config drift	Yes (operators reconcile)	Yes (puppet + sync pillar)
Built-in monitoring	Yes (Prometheus + Alertmanager bundled)	Yes (health aggregator)
Built-in secrets	Yes (etcd-encrypted secrets)	Yes (secret store + local cache)
Certificate management	Yes (internal CA, auto-rotation)	Yes (identity layer)
Node lifecycle	Yes (MachineSet, MachinePools)	Yes (onboard, labels, providers)
Self-managing	Yes (operators update themselves)	Yes (lab manages itself)

Why OpenShift Still Doesn't Fit

1. Single OS — OpenShift control plane = RHCOS only. Can't run on Apple Silicon, Asahi Linux, or any non-RHCOS system. Lab needs Ubuntu, Debian, Fedora, AlmaLinux, XCP-ng, VyOS across x86 and ARM.

2. K8s only — OpenShift manages k8s nodes. Lab manages everything: k8s nodes, standalone VMs, bare metal hypervisors, network appliances, physical servers that will never run k8s. Not everything is a container.

3. Single cluster scope — OpenShift manages one cluster. Lab manages homelab k3s + enterprise AWS EKS + XCP-ng hypervisors + bare metal + OVH vRack. Cross-provider, cross-cluster.

4. Fights puppet — OpenShift has ~30+ operators that each own a piece of the system. If puppet changes kubelet config, the Machine Config Operator detects "drift" and reverts it. Two reconciliation loops fighting each other, possibly rebooting nodes in a loop. You're supposed to change everything via CRDs, not external tools.

5. No XCP-ng/hypervisor management — Can't provision VMs on XCP-ng, manage Xen hosts, or understand hypervisors that aren't VMware/OpenStack.

6. Throws away puppet modules — Company has existing puppet modules. OpenShift's model is operators, not puppet. Complete rewrite of config management.

7. Heavyweight — Minimum 6 nodes, 88GB RAM just for the platform. k3s uses 512MB. Our entire homelab is 5 nodes, 320GB RAM.

8. ARM limited — RHCOS on Apple Silicon doesn't exist. ARM support is limited to AWS Graviton and some server ARM platforms.

The Scope Difference

OpenShift:  "I am your platform. Everything runs in me. I control the OS."
            Scope: Kubernetes cluster + its nodes

Lab:        "I manage your infrastructure. K8s is one thing I deploy."
            Scope: Everything — VMs, bare metal, hypervisors, k8s,
                   network gear, containers, across any provider

Lab is closer to what OpenShift + Satellite + RHCOS + ACM (Advanced Cluster Management) do together — but unified, lighter, open source, and not locked to Red Hat's ecosystem.

Why k3s

Puppet-friendly — it's just a binary and config files in /etc/rancher/k3s/
Ultra-light — runs on Mac Studio, ARM boxes, small VMs
Multi-arch — native x86 and ARM
Same K8s API as EKS/GKE — portable to cloud
Single binary — trivial to manage with puppet
Proven — CNCF certified, widely used in edge/IoT/homelab

k3s via Puppet (OpenVox)

# Label: k8s-server → puppet class
class kubernetes::server {
  class { 'k3s::server':
    token        => lab::secret('k8s/cluster-token'),
    cluster_init => true,
    tls_san      => [$facts['fqdn'], 'k8s.lab.internal'],
  }
}

# Label: k8s-worker → puppet class
class kubernetes::worker {
  class { 'k3s::worker':
    server_url => 'https://k8s.lab.internal:6443',
    token      => lab::secret('k8s/cluster-token'),
  }
}

Same puppet classes work on bare metal, XCP-ng VM, EC2 instance, any architecture.

k0s as Backup Option

If k3s ever becomes problematic, k0s is the closest alternative:

Also single binary, config-driven, multi-arch
k0sctl adds cluster management (bootstrap, upgrade, reset)
Mirantis backing (Lens, Docker EE)
Worth monitoring but no reason to switch from k3s today

5.2 KiB Raw Blame History