# Kickstart Reference — Lessons Learned

This documents pitfalls discovered during PXE boot testing. Read before modifying
the kickstart template (`src/bastion/src/templates/install.ks.ts`).

## Package requirements

### `kernel-modules` is mandatory

`@core` only installs `kernel-modules-core`, which lacks common modules like `vfat`,
`zram`, and many network/filesystem drivers. Without `kernel-modules`:

- `/boot/efi` (FAT32) cannot mount → `systemd-remount-fs` fails → **root stays
  read-only** → sshd-keygen can't write host keys → SSH unreachable
- `zram-generator` fails → can trigger emergency mode

**Always include `kernel-modules` in %packages.** This matches what the real
labmaster (192.168.8.11) has installed.

Regression introduced in commit `fac14b6` which removed `@server-product`
(that group pulled in `kernel-modules` via `fedora-release-server`).

### `dosfstools` is needed

Provides `mkfs.vfat` and ensures FAT filesystem support is available. The real
labmaster has it installed.

### Verify against the real machine

Before changing the package list, SSH to the labmaster and compare:
```bash
ssh 192.168.8.11 "rpm -q <package>"
```

## Anaconda %post execution order

This is critical and not well documented:

1. `%pre` scripts run
2. Disk partitioning and formatting
3. Package installation
4. **Anaconda writes system config (fstab, hostname, etc.)**
5. `%post` scripts run (in chroot of installed system)
6. `%post --nochroot` scripts run
7. **Anaconda MAY overwrite fstab again after %post scripts**

**Consequence:** You cannot reliably modify `/etc/fstab` from `%post` or
`%post --nochroot`. Anaconda overwrites it. Tested and confirmed — both
`sed` in %post and %post --nochroot had no effect on the final fstab.

What DOES work from %post:
- Writing files to `/etc/` (systemd units, config files, SSH keys)
- Enabling/disabling systemd services
- Installing additional packages
- Running `systemctl enable/mask`

What does NOT work from %post:
- Modifying `/etc/fstab` (Anaconda overwrites it)
- `--fsoptions` on `part /boot/efi` (Anaconda ignores it for EFI partitions)

## UEFI / EFI partition

- Anaconda always creates an EFI System Partition for UEFI installs
- The EFI partition is FAT32 — requires `vfat` kernel module to mount
- If `/boot/efi` fails to mount, `systemd-remount-fs` fails, which leaves
  root as read-only. This cascades to break ALL services that need to write
- The EFI partition is used by firmware directly for bootloader — the OS
  doesn't strictly need it mounted, but Anaconda adds it to fstab

## VM-specific issues (libvirt/QEMU/OVMF)

### iPXE exit behavior
- `exit` (no args) returns EFI_SUCCESS → OVMF retries PXE, never reaches disk
- `exit 1` returns EFI_ABORTED → OVMF moves to next boot device (disk)
- VM boot order needs both `network` and `hd`: `--boot=uefi,network,hd`

### nftables
- libvirt creates reject rules for NAT networks in table `ip libvirt_network`
  (NOT `inet libvirt` — this wrong table name cost hours of debugging)
- These rules block new host→VM connections (SSH)
- Rules are recreated on every `virsh start` — must delete after each VM restart
- Chains: `guest_input` and `guest_output`

### Serial console
- VM serial port: `--serial=tcp,host=127.0.0.1:4555,mode=bind,protocol=telnet`
- Use `virsh console <vm-name>` for interactive access (handles telnet protocol)
- Raw `socat` works for reading but pagers/readline break interactive use
- Add `console=ttyS0,115200n8` to kernel args for boot output on serial

### SELinux on labmaster
- Set to **permissive** — this is for k3s/kubernetes, NOT because SSH needs it
- SSH works fine with SELinux enforcing on a properly installed Fedora system
- The `ld.so.cache` AVC denials seen during debugging were caused by the
  read-only root filesystem, not by SELinux policy

## Testing checklist

Before merging kickstart changes:
1. Check the real labmaster has the same packages: `ssh 192.168.8.11 "rpm -q <pkg>"`
2. Run the PXE integration test: `sudo pnpm run test:integration:pxe`
3. Verify via serial console (root / `lab-root-pw`) if SSH fails
4. Check `mount | grep " / "` — must show `rw`, not `ro`
5. Check `systemctl --failed` — no critical failures