feat: PXE debug boot mode for rescue/diagnostics #4

Merged

michal merged 16 commits from wip/ks-debugging into main 2026-03-30 02:59:35 +00:00

Conversation 0 Commits 16 Files Changed 33 +1074 -103

michal commented 2026-03-29 21:26:26 +00:00

Owner

Copy Link

Summary

• New labctl provision debug <target> command that PXE boots into Fedora rescue mode
• Auto-clears debug state after one boot (next reboot = normal)
• Full stack: shared types, bastion dispatch/API, labd routing, CLI with rescue workflow guide
• Prints LVM mount + systemd-nspawn workflow after queuing

Test plan

• All 202 tests pass
• TypeScript type-check clean
• Manual: labctl provision debug worker0-k8s0 → PXE boot into rescue → mount + nspawn

🤖 Generated with Claude Code

## Summary - New `labctl provision debug <target>` command that PXE boots into Fedora rescue mode - Auto-clears debug state after one boot (next reboot = normal) - Full stack: shared types, bastion dispatch/API, labd routing, CLI with rescue workflow guide - Prints LVM mount + systemd-nspawn workflow after queuing ## Test plan - [x] All 202 tests pass - [x] TypeScript type-check clean - [ ] Manual: `labctl provision debug worker0-k8s0` → PXE boot into rescue → mount + nspawn 🤖 Generated with [Claude Code](https://claude.com/claude-code)

michal added 8 commits 2026-03-29 21:26:27 +00:00

fix: syslog parser TS strict null check, deploy script ... aae03d9877

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat: persist machine state to CockroachDB on bastion-state-sync ... 0c1e18cee1

When bastion syncs state, labd now upserts discovered and installed
machines into the Server table. /api/machines merges live bastion
state with DB records, so machines survive pod restarts.

Discovered machines get status=discovered with hardware labels.
Installed machines get status=online with hostname, role, IP.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: add nomodeset to iPXE kernel args (amdgpu hangs on SER9MAX) ... a14fd04947

Radeon 780M GPU driver initialization hangs during Anaconda boot
on SER9MAX. nomodeset disables kernel modesetting so the installer
doesn't try to initialize the GPU.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: disable logging --host (UDP not exposed), add nomodeset + JetKVM helper ... 5b04d3162b

- logging --host blocks Anaconda when syslog UDP port not reachable
- nomodeset prevents amdgpu hang on SER9MAX (Radeon 780M)
- JetKVM helper script for device control (status, reboot, power)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: remove serial console from iPXE kernel args (may hang on SER9MAX) ... e3523d642c

ttyS0 console output on iPXE kernel line may cause kernel hang on
hardware without physical serial port. Removed from both discover
and install iPXE scripts. Serial console stays in bootloader config
for the installed system only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: skip removable/USB disks in %pre, wait for NVMe init ... a7a6ad8098

JetKVM virtual media appears as /dev/sda before NVMe initializes.
Now: wait up to 10s for disks, skip removable disks and anything
under 20GB. Fixes "ignoredisk: sda does not exist" on SER9MAX.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: skip USB-attached disks in %pre (JetKVM virtual media is SCSI-over-USB) ... 6c6d5763c4

Check sysfs device path for 'usb' to skip JetKVM virtual media which
appears as /dev/sda but is not a real install target.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat: PXE debug boot mode for rescue/diagnostics ... e87edfcfbd

New `labctl provision debug <target>` command that PXE boots a machine
into Fedora rescue mode (inst.rescue) for live debugging. Auto-clears
after one boot so next reboot returns to normal.

Adds debug state to BastionState, dispatch routing, API endpoints,
labd command routing, and CLI with rescue workflow guide.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

michal added 1 commit 2026-03-29 21:42:53 +00:00

fix: add command-debug to LabdBastionMessage protocol types ... 52150fd955

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

michal added 1 commit 2026-03-29 21:54:05 +00:00

fix: add debug field to inline BastionState in labd server ... 82ca93f4d7

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

michal added 1 commit 2026-03-29 22:01:18 +00:00

fix: route command-debug through bastion WebSocket handler ... d7a59665ad

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

michal added 1 commit 2026-03-29 22:53:22 +00:00

feat: debug --sshd flag, auto SSH + nc listener + IP callback ... 816736793d

When using `labctl provision debug <target> --sshd`, the rescue
kickstart generates host keys, starts sshd (pw: debug) and nc
listener (port 2323), and reports the IP back to bastion via
/api/progress callback. Fully self-contained, no mounted FS needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

michal force-pushed wip/ks-debugging from 816736793d to 3835fefba1 2026-03-29 22:54:24 +00:00 Compare

michal added 1 commit 2026-03-29 22:59:41 +00:00

fix: generic rescue instructions in debug command output ... 92c65b4672

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

michal added 1 commit 2026-03-29 23:25:21 +00:00

fix: use %pre instead of %post for debug --sshd (rescue mode skips %post) ... 8da947a1c3

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

michal added 1 commit 2026-03-29 23:49:46 +00:00

feat: debug --pxe-boot flag, boot installed system via PXE ... a4a4840930

Loads kernel+initrd from bastion HTTP server, mounts root from local
NVMe. Workaround for UEFI firmware bugs that make local disk boot
100x slower. One-time use, auto-clears after boot.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

michal added 1 commit 2026-03-30 02:59:00 +00:00

fix: remove serial console (root cause of 30s boot delay), enable syslog logging, disk auto-detect ... 0a4916d3c9

Root cause found: console=ttyS0,115200n8 causes 30-second timeout at every
systemd boot phase on hardware without a physical serial UART. Each phase
transition blocks waiting for the non-existent UART.

Changes:
- Remove console=ttyS0 from kickstart bootloader args and %post setup
- Enable Anaconda syslog forwarding (logging --host --port) for install visibility
- Improve syslog IP→MAC resolution (register from kickstart fetch + progress)
- Fix disk auto-detect: default to empty string (not /dev/sda) for NVMe support
- Enable SysRq magic keys (kernel.sysrq=1) for emergency reboot via JetKVM
- Simplify debug command: remove --sshd flag (inst.sshd always available),
  add /debug-setup.sh HTTP endpoint for nc listener setup
- Add labctl provision logs -f (follow mode with polling)
- Add syslog listener unit tests
- Enable syslog log capture test in integration suite

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

michal merged commit 84afe7d5e4 into main 2026-03-30 02:59:35 +00:00

michal referenced this issue from a commit 2026-03-30 02:59:35 +00:00

Merge pull request 'feat: PXE debug boot mode for rescue/diagnostics' (#4) from wip/ks-debugging into main

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

michal

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: michal/lab#4