first commit
This commit is contained in:
Michal Rydlikowski
120
kubernetes-flavors.md
Normal file
120
kubernetes-flavors.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# Kubernetes Flavor Decision
|
||||
|
||||
## Decision: k3s (confirmed)
|
||||
|
||||
k3s is the best fit for Lab. OpenShift and most other flavors conflict with
|
||||
the puppet-managed, multi-arch, lightweight approach.
|
||||
|
||||
## Evaluation
|
||||
|
||||
| Flavor | Puppet-Friendly | ARM | Multi-arch | Enterprise | License | Verdict |
|
||||
|--------|:-:|:-:|:-:|:-:|---------|---------|
|
||||
| **k3s** | ✓ binary + config files | ✓ | ✓ | Rancher/SUSE | Apache 2.0 | **CHOSEN** |
|
||||
| **k0s** | ✓ single binary, config-driven | ✓ | ✓ | Mirantis | Apache 2.0 | Good alternative |
|
||||
| **kubeadm** | ✓ well-understood bootstrap | ✓ | ✓ | Upstream K8s | Apache 2.0 | Viable but heavier |
|
||||
| **RKE2** | ✓ config files | ✓ | ✓ | Rancher/SUSE | Apache 2.0 | Heavier k3s |
|
||||
| **OpenShift** | ✗ operator-driven, fights puppet | ✗ limited | ✗ limited | Red Hat | Proprietary | REJECTED |
|
||||
| **MicroK8s** | ⚠ snap-based, puppet+snaps awkward | ✓ | ✓ | Canonical | Apache 2.0 | Not great |
|
||||
| **Talos** | ✗ immutable OS, no SSH, no puppet | ✓ | ✓ | Sidero Labs | MPL 2.0 | Incompatible |
|
||||
|
||||
## Why NOT OpenShift — Deep Analysis
|
||||
|
||||
### OpenShift Does Overlap With Lab
|
||||
|
||||
OpenShift is the closest existing thing to what Lab does. The overlap is real:
|
||||
|
||||
| Capability | OpenShift | Lab |
|
||||
|-----------|-----------|-----|
|
||||
| Manages nodes end-to-end | Yes (RHCOS + MCO) | Yes (OpenVox + labels) |
|
||||
| Immutable infrastructure | Yes (rpm-ostree, operator-driven) | Yes (puppet convergence) |
|
||||
| Fights config drift | Yes (operators reconcile) | Yes (puppet + sync pillar) |
|
||||
| Built-in monitoring | Yes (Prometheus + Alertmanager bundled) | Yes (health aggregator) |
|
||||
| Built-in secrets | Yes (etcd-encrypted secrets) | Yes (secret store + local cache) |
|
||||
| Certificate management | Yes (internal CA, auto-rotation) | Yes (identity layer) |
|
||||
| Node lifecycle | Yes (MachineSet, MachinePools) | Yes (onboard, labels, providers) |
|
||||
| Self-managing | Yes (operators update themselves) | Yes (lab manages itself) |
|
||||
|
||||
### Why OpenShift Still Doesn't Fit
|
||||
|
||||
**1. Single OS** — OpenShift control plane = RHCOS only. Can't run on Apple Silicon,
|
||||
Asahi Linux, or any non-RHCOS system. Lab needs Ubuntu, Debian, Fedora, AlmaLinux,
|
||||
XCP-ng, VyOS across x86 and ARM.
|
||||
|
||||
**2. K8s only** — OpenShift manages k8s nodes. Lab manages everything: k8s nodes,
|
||||
standalone VMs, bare metal hypervisors, network appliances, physical servers that
|
||||
will never run k8s. Not everything is a container.
|
||||
|
||||
**3. Single cluster scope** — OpenShift manages one cluster. Lab manages homelab k3s +
|
||||
enterprise AWS EKS + XCP-ng hypervisors + bare metal + OVH vRack. Cross-provider,
|
||||
cross-cluster.
|
||||
|
||||
**4. Fights puppet** — OpenShift has ~30+ operators that each own a piece of the system.
|
||||
If puppet changes kubelet config, the Machine Config Operator detects "drift" and
|
||||
reverts it. Two reconciliation loops fighting each other, possibly rebooting nodes
|
||||
in a loop. You're supposed to change everything via CRDs, not external tools.
|
||||
|
||||
**5. No XCP-ng/hypervisor management** — Can't provision VMs on XCP-ng, manage Xen
|
||||
hosts, or understand hypervisors that aren't VMware/OpenStack.
|
||||
|
||||
**6. Throws away puppet modules** — Company has existing puppet modules. OpenShift's
|
||||
model is operators, not puppet. Complete rewrite of config management.
|
||||
|
||||
**7. Heavyweight** — Minimum 6 nodes, 88GB RAM just for the platform. k3s uses 512MB.
|
||||
Our entire homelab is 5 nodes, 320GB RAM.
|
||||
|
||||
**8. ARM limited** — RHCOS on Apple Silicon doesn't exist. ARM support is limited to
|
||||
AWS Graviton and some server ARM platforms.
|
||||
|
||||
### The Scope Difference
|
||||
|
||||
```
|
||||
OpenShift: "I am your platform. Everything runs in me. I control the OS."
|
||||
Scope: Kubernetes cluster + its nodes
|
||||
|
||||
Lab: "I manage your infrastructure. K8s is one thing I deploy."
|
||||
Scope: Everything — VMs, bare metal, hypervisors, k8s,
|
||||
network gear, containers, across any provider
|
||||
```
|
||||
|
||||
Lab is closer to what OpenShift + Satellite + RHCOS + ACM (Advanced Cluster Management)
|
||||
do **together** — but unified, lighter, open source, and not locked to Red Hat's ecosystem.
|
||||
|
||||
## Why k3s
|
||||
|
||||
- **Puppet-friendly** — it's just a binary and config files in `/etc/rancher/k3s/`
|
||||
- **Ultra-light** — runs on Mac Studio, ARM boxes, small VMs
|
||||
- **Multi-arch** — native x86 and ARM
|
||||
- **Same K8s API** as EKS/GKE — portable to cloud
|
||||
- **Single binary** — trivial to manage with puppet
|
||||
- **Proven** — CNCF certified, widely used in edge/IoT/homelab
|
||||
|
||||
## k3s via Puppet (OpenVox)
|
||||
|
||||
```puppet
|
||||
# Label: k8s-server → puppet class
|
||||
class kubernetes::server {
|
||||
class { 'k3s::server':
|
||||
token => lab::secret('k8s/cluster-token'),
|
||||
cluster_init => true,
|
||||
tls_san => [$facts['fqdn'], 'k8s.lab.internal'],
|
||||
}
|
||||
}
|
||||
|
||||
# Label: k8s-worker → puppet class
|
||||
class kubernetes::worker {
|
||||
class { 'k3s::worker':
|
||||
server_url => 'https://k8s.lab.internal:6443',
|
||||
token => lab::secret('k8s/cluster-token'),
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Same puppet classes work on bare metal, XCP-ng VM, EC2 instance, any architecture.
|
||||
|
||||
## k0s as Backup Option
|
||||
|
||||
If k3s ever becomes problematic, k0s is the closest alternative:
|
||||
- Also single binary, config-driven, multi-arch
|
||||
- `k0sctl` adds cluster management (bootstrap, upgrade, reset)
|
||||
- Mirantis backing (Lens, Docker EE)
|
||||
- Worth monitoring but no reason to switch from k3s today
|
||||
Reference in New Issue
Block a user