first commit

2026-03-15 23:50:43 +00:00
commit ac695f506f
9 changed files with 3003 additions and 0 deletions
--- a/os-install-research.md
+++ b/os-install-research.md
@@ -0,0 +1,356 @@
+# OS Installation Research
+
+## Target Operating Systems
+
+All must support unattended network installation and automated OpenVox enrollment.
+All must work across multiple CPU architectures where the OS supports it.
+
+| OS | Install System | Answer Format | Architectures | PXE Difficulty |
+|-----|---------------|--------------|---------------|---------------|
+| Ubuntu 24.04 | autoinstall (cloud-init) | YAML | x86_64, aarch64, RISC-V | Easy |
+| Debian 12 | preseed | preseed.cfg | x86_64, aarch64, many others | Medium |
+| Fedora 41+ | Anaconda/kickstart | .ks file | x86_64, aarch64 | Easy |
+| AlmaLinux 9 | Anaconda/kickstart | .ks file | x86_64, aarch64 | Easy |
+| XCP-ng 8.3 | Custom Python TUI | XML answer file | x86_64 only | HARD |
+| VyOS 1.4 | Custom installer | config.boot | x86_64, aarch64 | Medium |
+
+## XCP-ng Network Install — Known Hard
+
+### Why it's difficult
+- iPXE UEFI is fundamentally broken (open bug, multiboot module corruption)
+- Serial/headless install hangs after detecting storage — no fix
+- No VNC installer mode (unlike RHEL/Debian)
+- TFTP agonizingly slow for large install.img
+- Custom Python TUI designed for VGA console, not automation
+- No major provisioning tool has first-class XCP-ng support
+
+### What works
+- **BIOS PXE** more reliable than UEFI
+- **IPMI virtual media** with remastered ISO is most reliable
+- Answer file XML with `<post-install-script>` and `<script stage="filesystem-populated">`
+- Post-install puppet enrollment via `/etc/firstboot.d/` scripts
+- XCP-ng enables SSH by default after install
+
+### Answer file format (XML, custom to XenServer/XCP-ng)
+```xml
+<?xml version="1.0"?>
+<installation mode="fresh" srtype="ext">
+    <primary-disk>sda</primary-disk>
+    <keymap>us</keymap>
+    <root-password type="hash">$6$...</root-password>
+    <source type="url">http://server/xcp-ng/</source>
+    <admin-interface name="eth0" proto="dhcp" />
+    <hostname>xcphost01</hostname>
+    <timezone>Europe/London</timezone>
+    <ntp-server>pool.ntp.org</ntp-server>
+    <network-backend>openvswitch</network-backend>
+    <post-install-script type="url">http://server/scripts/post-install.sh</post-install-script>
+    <script stage="filesystem-populated" type="url">http://server/scripts/fs-setup.sh</script>
+</installation>
+```
+
+### Post-install puppet enrollment
+The `filesystem-populated` stage script drops a firstboot script:
+```bash
+#!/bin/bash
+MOUNT=$1
+cat > "$MOUNT/etc/firstboot.d/99-lab-enroll" << 'SCRIPT'
+#!/bin/bash
+# Install puppet agent (XCP-ng is CentOS-based, yum works)
+yum install -y puppet-agent
+# Configure and start
+puppet config set server puppet.lab.internal
+systemctl enable --now puppet
+SCRIPT
+chmod +x "$MOUNT/etc/firstboot.d/99-lab-enroll"
+```
+
+## Lab Install Profile Abstraction
+
+Lab needs an `InstallerPlugin` interface so the same `lab onboard` command works
+for all OS types. Each plugin handles answer file generation, PXE chain setup,
+and post-install enrollment for its OS type.
+
+```go
+type InstallerPlugin interface {
+    Name() string
+    SupportedArchitectures() []string
+
+    // Generate the answer/config file for unattended install
+    GenerateAnswerFile(config InstallConfig) ([]byte, error)
+
+    // Set up PXE boot artifacts (kernel, initrd, bootloader configs)
+    PreparePXE(config PXEConfig) error
+
+    // Generate post-install enrollment script
+    GenerateEnrollmentScript(token string, labels []string) ([]byte, error)
+}
+```
+
+Built-in installer plugins:
+- `installer-autoinstall` — Ubuntu (cloud-init based autoinstall YAML)
+- `installer-kickstart` — Fedora, AlmaLinux, RHEL (kickstart .ks files)
+- `installer-preseed` — Debian (preseed.cfg)
+- `installer-xcpng` — XCP-ng (custom XML + firstboot.d scripts)
+- `installer-vyos` — VyOS (config.boot)
+
+## Auto-Onboard Rules
+
+Automatic onboarding based on detected hardware characteristics:
+
+```yaml
+auto-onboard:
+  rules:
+    - name: large-compute-to-xcpng
+      conditions:
+        cores: ">= 40"
+        memory: ">= 500GB"
+        provider: ovh
+      action:
+        image: xcpng-8.3
+        labels: [xen-host, production]
+
+    - name: arm-to-ubuntu
+      conditions:
+        arch: aarch64
+      action:
+        image: ubuntu-24.04
+        labels: [arm, k8s-worker]
+```
+
+Must support:
+- Preview: show which existing servers match/don't match rules
+- Dry-run: show what would happen for pending servers
+- Apply: actually onboard matching servers
+
+## Deployment Approach: Universal PXE Agent + Rootfs Images
+
+### Decision: NOT using native installers
+
+Instead of dealing with 6 different installer formats (autoinstall, kickstart, preseed,
+XCP-ng XML, VyOS config), Lab uses a universal approach:
+
+1. PXE boot ONE agent OS (same for all target distros)
+2. Agent contacts Lab server, gets instructions
+3. Agent partitions disk, deploys rootfs tarball, injects config, reboots
+4. Target OS boots with lab-agent, enrolls with OpenVox
+
+This avoids the nightmare of maintaining 6 installer plugins × 3 architectures.
+
+### Tool Evaluation
+
+| Tool | What It Does | For Lab? |
+|------|-------------|----------|
+| **Tinkerbell (CNCF)** | PXE → HookOS agent → workflow actions (partition, deploy, inject) | **Best candidate to wrap** |
+| **LinuxKit** | Build minimal agent OS (used by Tinkerbell's HookOS) | Build our PXE agent |
+| **mkosi** | Build rootfs tarballs for any distro (Fedora, Ubuntu, Debian, etc.) | **Image production** |
+| **iPXE** | Universal PXE bootloader with scripting | PXE foundation |
+| **Pixiecore** | Simple Go PXE server with per-MAC API mode | PXE building block |
+| **bootc** | Bootable OCI containers → install to disk (RHEL-family) | Image format option |
+| **cloud-init** | First-boot config injection | Post-deploy config |
+| **Packer** | Build VM/machine images | Golden image building |
+| **MAAS/Curtin** | Production-grade, same pattern, but Ubuntu-centric + heavy | Too opinionated |
+| **Warewulf** | Stateless/diskless boot from container images | Wrong model (RAM-only) |
+| **Kairos** | Immutable k8s-focused OS from containers | Too opinionated |
+| **FOG/Clonezilla** | Block-level disk cloning | Too rigid |
+| **FAI** | Debian-centric installer framework | Too narrow |
+| **Razor (Puppet)** | Dead (archived 2019) | Dead |
+| **netboot.xyz** | PXE boot menu into native installers | Opposite of what we want |
+
+### Tinkerbell — Closest Match
+
+Tinkerbell already implements this pattern:
+- **HookOS**: minimal agent OS built with LinuxKit, boots via PXE, multi-arch (x86 + ARM)
+- **Tink Worker**: runs inside HookOS, contacts server via gRPC, executes workflows
+- **Workflow Actions**:
+  - `rootio` — partition disks, create filesystems
+  - `archive2disk` — stream compressed rootfs tarball to mounted filesystem
+  - `image2disk` — write raw disk image (dd-style)
+  - `oci2disk` — pull OCI container image, write to disk
+  - `writefile` — write individual files (puppet certs, config, enrollment token)
+  - `cexec` — chroot and run commands (install bootloader, etc.)
+  - `kexec` — kexec into new kernel (avoids reboot)
+
+**Tinkerbell's limitation:** requires Kubernetes to run (Tink Server is k8s-native).
+Options:
+- Run on bootstrap node's k3s (works but adds k3s dependency before we have k3s)
+- Extract just HookOS + actions, replace Tink Server with Lab's own API
+- Use Tinkerbell after initial bootstrap
+
+### Option A: Wrap Tinkerbell
+Use Tinkerbell's HookOS and actions, Lab translates `lab onboard` into Tinkerbell
+workflows. Proven, multi-arch, battle-tested by Equinix Metal.
+
+### Option B: Build our own lightweight agent
+If Tinkerbell's k8s dependency is too heavy:
+- Build agent OS with LinuxKit (like HookOS but simpler)
+- Small Go binary as the agent: contacts lab-server, gets instructions, partitions,
+  deploys rootfs, injects files, installs bootloader, reboots
+- Embedded in Lab binary — no k8s dependency
+- Essentially "Tinkerbell actions without Tinkerbell's workflow engine"
+
+### Decision: TBD — needs hands-on evaluation of Tinkerbell
+
+### VyOS Inspiration
+
+VyOS proves this pattern works:
+- Image-based install (rootfs deployed to partition)
+- Also runs as Docker container (same config system)
+- Same concept as Lab: one definition → VM image, bare metal, or container
+
+### Image Production Pipeline
+
+Lab needs to produce rootfs tarballs for each OS × architecture:
+
+```
+$ lab image build ubuntu-24.04 --arch x86_64,aarch64
+  → Uses mkosi or debootstrap to build rootfs
+  → Injects lab-agent, cloud-init datasource
+  → Produces: ubuntu-24.04-x86_64.tar.gz, ubuntu-24.04-aarch64.tar.gz
+
+$ lab image build xcpng-8.3 --arch x86_64
+  → Extract/capture rootfs from XCP-ng installer/installed system
+  → Produces: xcpng-8.3-x86_64.tar.gz
+
+$ lab image list
+IMAGE              ARCH              SIZE      BUILT
+ubuntu-24.04       x86_64, aarch64   850MB     2026-03-15
+debian-12          x86_64, aarch64   620MB     2026-03-14
+fedora-41          x86_64, aarch64   920MB     2026-03-14
+almalinux-9        x86_64, aarch64   780MB     2026-03-13
+xcpng-8.3          x86_64            1.2GB     2026-03-10
+vyos-1.4           x86_64, aarch64   450MB     2026-03-12
+```
+
+Image build tools per OS:
+- Ubuntu/Debian: debootstrap or mkosi
+- Fedora/AlmaLinux: dnf --installroot or mkosi
+- XCP-ng: install in QEMU + Packer, capture rootfs (only viable method)
+- VyOS: extract squashfs from ISO (`unsquashfs /mnt/live/filesystem.squashfs`)
+- Asahi Linux: NOT BUILDABLE — SSH onboard only, OS already installed by user
+
+## XCP-ng Rootfs Production — Detailed
+
+### Why package-based build doesn't work
+- `install.img` is the installer ramdisk, NOT the target system
+- The installer (`host-installer/backend.py`) does post-install XAPI setup that
+  can't be replicated with just yum --installroot
+- Nobody has successfully built XCP-ng from packages alone
+- `create-install-image` scripts only produce ISOs
+
+### Viable approach: Packer + QEMU capture
+```
+1. Boot XCP-ng ISO in QEMU with answerfile (unattended)
+2. Installer runs normally, does all XAPI/Xen setup
+3. Mount resulting disk image
+4. Tar up root partition
+5. Generalize: remove SSH keys, XAPI state.db, hostname, UUIDs, persistent net rules
+6. Output: xcpng-8.3-x86_64.tar.gz
+```
+
+### XCP-ng partition layout (PXE agent must recreate this)
+```
+sda1: 18GB  ext3  /           (dom0 root)
+sda2: 18GB  ext3  (backup)    (upgrade slot)
+sda3: rest  LVM   (SR)        (VM storage repository)
+sda4: 512MB vfat  /boot/efi   (UEFI ESP)
+sda5: 4GB   ext3  /var/log
+sda6: 1GB   swap
+```
+
+## Asahi Linux — Special Case
+
+### Why it can't follow the standard path
+- No PXE boot — Apple Silicon only boots from internal NVMe or USB (iBoot)
+- Firmware partition — m1n1 must be in Apple's APFS container, coexists with macOS
+- Device tree — generated per-chip at install time
+- GPU drivers — Asahi's reverse-engineered drivers are kernel-specific
+- Boot chain: iBoot → m1n1 → U-Boot/GRUB → Linux (completely non-standard)
+
+### How Lab handles it
+- SSH onboard only: `lab onboard mac-studio --provider ssh --host <ip>`
+- Asahi is already installed (user did this manually or via Asahi installer)
+- Lab manages the userspace (Fedora-based) via puppet normally
+- Kernel updates from Asahi repos, managed by puppet/dnf
+- m1n1/U-Boot/firmware layer is untouched by Lab
+
+### Lesson
+Not everything is PXE-bootable. Lab needs two onboard paths:
+- **PXE onboard**: bare metal with no OS (Beelinks, OVH servers, XCP-ng hosts)
+- **SSH onboard**: OS already installed (Mac Studio, DGX Spark, cloud VMs)
+
+## Image Deployment Matrix
+
+```
+                    PXE Deploy    SSH Onboard    Container    VM Image
+Ubuntu 24.04        ✓ rootfs      ✓              ✓            ✓ qcow2
+Debian 12           ✓ rootfs      ✓              ✓            ✓ qcow2
+Fedora 41           ✓ rootfs      ✓              ✓            ✓ qcow2
+AlmaLinux 9         ✓ rootfs      ✓              ✓            ✓ qcow2
+XCP-ng 8.3          ✓ rootfs      ✓ (existing)   ✗            ✗
+VyOS 1.4            ✓ rootfs      ✓ (existing)   ✓ docker     ✓ qcow2
+Asahi Linux         ✗ impossible  ✓ (only way)   ✗            ✗
+```
+
+## Automated Image Pipeline
+
+Images must be rebuilt regularly to include security updates and new lab-agent versions.
+
+### Pipeline Configuration
+```yaml
+image-pipelines:
+  ubuntu-24.04:
+    method: debootstrap
+    schedule: weekly
+    architectures: [x86_64, aarch64]
+    outputs: [rootfs-tarball, container-base, qcow2]
+    retention: 4 builds
+
+  xcpng-8.3:
+    method: packer-qemu          # install in QEMU, capture
+    schedule: monthly
+    architectures: [x86_64]
+    outputs: [rootfs-tarball]
+    retention: 3 builds
+
+  vyos-1.4:
+    method: squashfs-extract     # extract from ISO
+    schedule: monthly
+    architectures: [x86_64, aarch64]
+    outputs: [rootfs-tarball, container-base]
+    retention: 3 builds
+```
+
+### Build runs on Lab itself (dogfooding)
+- x86 images build on x86 machines (Beelink SER9 MAX)
+- ARM images build on ARM machines (DGX Spark, Minisforum)
+- XCP-ng builds on any x86 with QEMU/KVM
+- Lab picks the right builder based on architecture
+
+### Upgrade flow
+- New image built → Lab knows which servers run old version
+- `lab image diff` shows package changes
+- `lab image promote` makes new image the default for new deploys
+- Existing servers: puppet manages package updates (not re-imaged unless requested)
+
+### Connection to Puppet → Container Artifact Builder
+
+Same pipeline, different output targets:
+
+```
+Label "mailserver" + base image "ubuntu-24.04":
+  → rootfs + puppet classes = bare metal image (tar.gz for PXE deploy)
+  → rootfs + puppet classes = container image (OCI for k8s/docker)
+  → rootfs + puppet classes = VM image (qcow2/vmdk for XCP-ng/AWS)
+
+One label, one set of puppet modules, three deployment formats.
+```
+
+## Multi-Architecture Considerations
+
+- PXE boot chain differs between x86 (BIOS/UEFI) and ARM (UEFI only)
+- Need separate kernel/initrd per architecture for the agent OS
+- Rootfs tarballs are architecture-specific
+- Some OS images don't exist for all architectures (XCP-ng = x86 only)
+- Lab must track architecture per image and refuse mismatches
+- Tinkerbell's HookOS already builds for x86_64 and aarch64