# Kubernetes Flavor Decision

## Decision: k3s (confirmed)

k3s is the best fit for Lab. OpenShift and most other flavors conflict with
the puppet-managed, multi-arch, lightweight approach.

## Evaluation

| Flavor | Puppet-Friendly | ARM | Multi-arch | Enterprise | License | Verdict |
|--------|:-:|:-:|:-:|:-:|---------|---------|
| **k3s** | ✓ binary + config files | ✓ | ✓ | Rancher/SUSE | Apache 2.0 | **CHOSEN** |
| **k0s** | ✓ single binary, config-driven | ✓ | ✓ | Mirantis | Apache 2.0 | Good alternative |
| **kubeadm** | ✓ well-understood bootstrap | ✓ | ✓ | Upstream K8s | Apache 2.0 | Viable but heavier |
| **RKE2** | ✓ config files | ✓ | ✓ | Rancher/SUSE | Apache 2.0 | Heavier k3s |
| **OpenShift** | ✗ operator-driven, fights puppet | ✗ limited | ✗ limited | Red Hat | Proprietary | REJECTED |
| **MicroK8s** | ⚠ snap-based, puppet+snaps awkward | ✓ | ✓ | Canonical | Apache 2.0 | Not great |
| **Talos** | ✗ immutable OS, no SSH, no puppet | ✓ | ✓ | Sidero Labs | MPL 2.0 | Incompatible |

## Why NOT OpenShift — Deep Analysis

### OpenShift Does Overlap With Lab

OpenShift is the closest existing thing to what Lab does. The overlap is real:

| Capability | OpenShift | Lab |
|-----------|-----------|-----|
| Manages nodes end-to-end | Yes (RHCOS + MCO) | Yes (OpenVox + labels) |
| Immutable infrastructure | Yes (rpm-ostree, operator-driven) | Yes (puppet convergence) |
| Fights config drift | Yes (operators reconcile) | Yes (puppet + sync pillar) |
| Built-in monitoring | Yes (Prometheus + Alertmanager bundled) | Yes (health aggregator) |
| Built-in secrets | Yes (etcd-encrypted secrets) | Yes (secret store + local cache) |
| Certificate management | Yes (internal CA, auto-rotation) | Yes (identity layer) |
| Node lifecycle | Yes (MachineSet, MachinePools) | Yes (onboard, labels, providers) |
| Self-managing | Yes (operators update themselves) | Yes (lab manages itself) |

### Why OpenShift Still Doesn't Fit

**1. Single OS** — OpenShift control plane = RHCOS only. Can't run on Apple Silicon,
   Asahi Linux, or any non-RHCOS system. Lab needs Ubuntu, Debian, Fedora, AlmaLinux,
   XCP-ng, VyOS across x86 and ARM.

**2. K8s only** — OpenShift manages k8s nodes. Lab manages everything: k8s nodes,
   standalone VMs, bare metal hypervisors, network appliances, physical servers that
   will never run k8s. Not everything is a container.

**3. Single cluster scope** — OpenShift manages one cluster. Lab manages homelab k3s +
   enterprise AWS EKS + XCP-ng hypervisors + bare metal + OVH vRack. Cross-provider,
   cross-cluster.

**4. Fights puppet** — OpenShift has ~30+ operators that each own a piece of the system.
   If puppet changes kubelet config, the Machine Config Operator detects "drift" and
   reverts it. Two reconciliation loops fighting each other, possibly rebooting nodes
   in a loop. You're supposed to change everything via CRDs, not external tools.

**5. No XCP-ng/hypervisor management** — Can't provision VMs on XCP-ng, manage Xen
   hosts, or understand hypervisors that aren't VMware/OpenStack.

**6. Throws away puppet modules** — Company has existing puppet modules. OpenShift's
   model is operators, not puppet. Complete rewrite of config management.

**7. Heavyweight** — Minimum 6 nodes, 88GB RAM just for the platform. k3s uses 512MB.
   Our entire homelab is 5 nodes, 320GB RAM.

**8. ARM limited** — RHCOS on Apple Silicon doesn't exist. ARM support is limited to
   AWS Graviton and some server ARM platforms.

### The Scope Difference

```
OpenShift:  "I am your platform. Everything runs in me. I control the OS."
            Scope: Kubernetes cluster + its nodes

Lab:        "I manage your infrastructure. K8s is one thing I deploy."
            Scope: Everything — VMs, bare metal, hypervisors, k8s,
                   network gear, containers, across any provider
```

Lab is closer to what OpenShift + Satellite + RHCOS + ACM (Advanced Cluster Management)
do **together** — but unified, lighter, open source, and not locked to Red Hat's ecosystem.

## Why k3s

- **Puppet-friendly** — it's just a binary and config files in `/etc/rancher/k3s/`
- **Ultra-light** — runs on Mac Studio, ARM boxes, small VMs
- **Multi-arch** — native x86 and ARM
- **Same K8s API** as EKS/GKE — portable to cloud
- **Single binary** — trivial to manage with puppet
- **Proven** — CNCF certified, widely used in edge/IoT/homelab

## k3s via Puppet (OpenVox)

```puppet
# Label: k8s-server → puppet class
class kubernetes::server {
  class { 'k3s::server':
    token        => lab::secret('k8s/cluster-token'),
    cluster_init => true,
    tls_san      => [$facts['fqdn'], 'k8s.lab.internal'],
  }
}

# Label: k8s-worker → puppet class
class kubernetes::worker {
  class { 'k3s::worker':
    server_url => 'https://k8s.lab.internal:6443',
    token      => lab::secret('k8s/cluster-token'),
  }
}
```

Same puppet classes work on bare metal, XCP-ng VM, EC2 instance, any architecture.

## k0s as Backup Option

If k3s ever becomes problematic, k0s is the closest alternative:
- Also single binary, config-driven, multi-arch
- `k0sctl` adds cluster management (bootstrap, upgrade, reset)
- Mirantis backing (Lens, Docker EE)
- Worth monitoring but no reason to switch from k3s today