121 lines
5.2 KiB
Markdown
121 lines
5.2 KiB
Markdown
|
# Kubernetes Flavor Decision
|
||
|
|
|
||
|
|
## Decision: k3s (confirmed)
|
||
|
|
|
||
|
|
k3s is the best fit for Lab. OpenShift and most other flavors conflict with
|
||
|
|
the puppet-managed, multi-arch, lightweight approach.
|
||
|
|
|
||
|
|
## Evaluation
|
||
|
|
|
||
|
|
| Flavor | Puppet-Friendly | ARM | Multi-arch | Enterprise | License | Verdict |
|
||
|
|
|--------|:-:|:-:|:-:|:-:|---------|---------|
|
||
|
|
| **k3s** | ✓ binary + config files | ✓ | ✓ | Rancher/SUSE | Apache 2.0 | **CHOSEN** |
|
||
|
|
| **k0s** | ✓ single binary, config-driven | ✓ | ✓ | Mirantis | Apache 2.0 | Good alternative |
|
||
|
|
| **kubeadm** | ✓ well-understood bootstrap | ✓ | ✓ | Upstream K8s | Apache 2.0 | Viable but heavier |
|
||
|
|
| **RKE2** | ✓ config files | ✓ | ✓ | Rancher/SUSE | Apache 2.0 | Heavier k3s |
|
||
|
|
| **OpenShift** | ✗ operator-driven, fights puppet | ✗ limited | ✗ limited | Red Hat | Proprietary | REJECTED |
|
||
|
|
| **MicroK8s** | ⚠ snap-based, puppet+snaps awkward | ✓ | ✓ | Canonical | Apache 2.0 | Not great |
|
||
|
|
| **Talos** | ✗ immutable OS, no SSH, no puppet | ✓ | ✓ | Sidero Labs | MPL 2.0 | Incompatible |
|
||
|
|
|
||
|
|
## Why NOT OpenShift — Deep Analysis
|
||
|
|
|
||
|
|
### OpenShift Does Overlap With Lab
|
||
|
|
|
||
|
|
OpenShift is the closest existing thing to what Lab does. The overlap is real:
|
||
|
|
|
||
|
|
| Capability | OpenShift | Lab |
|
||
|
|
|-----------|-----------|-----|
|
||
|
|
| Manages nodes end-to-end | Yes (RHCOS + MCO) | Yes (OpenVox + labels) |
|
||
|
|
| Immutable infrastructure | Yes (rpm-ostree, operator-driven) | Yes (puppet convergence) |
|
||
|
|
| Fights config drift | Yes (operators reconcile) | Yes (puppet + sync pillar) |
|
||
|
|
| Built-in monitoring | Yes (Prometheus + Alertmanager bundled) | Yes (health aggregator) |
|
||
|
|
| Built-in secrets | Yes (etcd-encrypted secrets) | Yes (secret store + local cache) |
|
||
|
|
| Certificate management | Yes (internal CA, auto-rotation) | Yes (identity layer) |
|
||
|
|
| Node lifecycle | Yes (MachineSet, MachinePools) | Yes (onboard, labels, providers) |
|
||
|
|
| Self-managing | Yes (operators update themselves) | Yes (lab manages itself) |
|
||
|
|
|
||
|
|
### Why OpenShift Still Doesn't Fit
|
||
|
|
|
||
|
|
**1. Single OS** — OpenShift control plane = RHCOS only. Can't run on Apple Silicon,
|
||
|
|
Asahi Linux, or any non-RHCOS system. Lab needs Ubuntu, Debian, Fedora, AlmaLinux,
|
||
|
|
XCP-ng, VyOS across x86 and ARM.
|
||
|
|
|
||
|
|
**2. K8s only** — OpenShift manages k8s nodes. Lab manages everything: k8s nodes,
|
||
|
|
standalone VMs, bare metal hypervisors, network appliances, physical servers that
|
||
|
|
will never run k8s. Not everything is a container.
|
||
|
|
|
||
|
|
**3. Single cluster scope** — OpenShift manages one cluster. Lab manages homelab k3s +
|
||
|
|
enterprise AWS EKS + XCP-ng hypervisors + bare metal + OVH vRack. Cross-provider,
|
||
|
|
cross-cluster.
|
||
|
|
|
||
|
|
**4. Fights puppet** — OpenShift has ~30+ operators that each own a piece of the system.
|
||
|
|
If puppet changes kubelet config, the Machine Config Operator detects "drift" and
|
||
|
|
reverts it. Two reconciliation loops fighting each other, possibly rebooting nodes
|
||
|
|
in a loop. You're supposed to change everything via CRDs, not external tools.
|
||
|
|
|
||
|
|
**5. No XCP-ng/hypervisor management** — Can't provision VMs on XCP-ng, manage Xen
|
||
|
|
hosts, or understand hypervisors that aren't VMware/OpenStack.
|
||
|
|
|
||
|
|
**6. Throws away puppet modules** — Company has existing puppet modules. OpenShift's
|
||
|
|
model is operators, not puppet. Complete rewrite of config management.
|
||
|
|
|
||
|
|
**7. Heavyweight** — Minimum 6 nodes, 88GB RAM just for the platform. k3s uses 512MB.
|
||
|
|
Our entire homelab is 5 nodes, 320GB RAM.
|
||
|
|
|
||
|
|
**8. ARM limited** — RHCOS on Apple Silicon doesn't exist. ARM support is limited to
|
||
|
|
AWS Graviton and some server ARM platforms.
|
||
|
|
|
||
|
|
### The Scope Difference
|
||
|
|
|
||
|
|
```
|
||
|
|
OpenShift: "I am your platform. Everything runs in me. I control the OS."
|
||
|
|
Scope: Kubernetes cluster + its nodes
|
||
|
|
|
||
|
|
Lab: "I manage your infrastructure. K8s is one thing I deploy."
|
||
|
|
Scope: Everything — VMs, bare metal, hypervisors, k8s,
|
||
|
|
network gear, containers, across any provider
|
||
|
|
```
|
||
|
|
|
||
|
|
Lab is closer to what OpenShift + Satellite + RHCOS + ACM (Advanced Cluster Management)
|
||
|
|
do **together** — but unified, lighter, open source, and not locked to Red Hat's ecosystem.
|
||
|
|
|
||
|
|
## Why k3s
|
||
|
|
|
||
|
|
- **Puppet-friendly** — it's just a binary and config files in `/etc/rancher/k3s/`
|
||
|
|
- **Ultra-light** — runs on Mac Studio, ARM boxes, small VMs
|
||
|
|
- **Multi-arch** — native x86 and ARM
|
||
|
|
- **Same K8s API** as EKS/GKE — portable to cloud
|
||
|
|
- **Single binary** — trivial to manage with puppet
|
||
|
|
- **Proven** — CNCF certified, widely used in edge/IoT/homelab
|
||
|
|
|
||
|
|
## k3s via Puppet (OpenVox)
|
||
|
|
|
||
|
|
```puppet
|
||
|
|
# Label: k8s-server → puppet class
|
||
|
|
class kubernetes::server {
|
||
|
|
class { 'k3s::server':
|
||
|
|
token => lab::secret('k8s/cluster-token'),
|
||
|
|
cluster_init => true,
|
||
|
|
tls_san => [$facts['fqdn'], 'k8s.lab.internal'],
|
||
|
|
}
|
||
|
|
}
|
||
|
|
|
||
|
|
# Label: k8s-worker → puppet class
|
||
|
|
class kubernetes::worker {
|
||
|
|
class { 'k3s::worker':
|
||
|
|
server_url => 'https://k8s.lab.internal:6443',
|
||
|
|
token => lab::secret('k8s/cluster-token'),
|
||
|
|
}
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
Same puppet classes work on bare metal, XCP-ng VM, EC2 instance, any architecture.
|
||
|
|
|
||
|
|
## k0s as Backup Option
|
||
|
|
|
||
|
|
If k3s ever becomes problematic, k0s is the closest alternative:
|
||
|
|
- Also single binary, config-driven, multi-arch
|
||
|
|
- `k0sctl` adds cluster management (bootstrap, upgrade, reset)
|
||
|
|
- Mirantis backing (Lens, Docker EE)
|
||
|
|
- Worth monitoring but no reason to switch from k3s today
|