Compare commits
77 Commits
feat/asahi
...
7bf3f42e19
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7bf3f42e19 | ||
|
|
09ede73b67 | ||
|
|
a973b51b9c | ||
|
|
d727a50ca0 | ||
|
|
0481c38e09 | ||
|
|
23783b8486 | ||
|
|
11344eab92 | ||
|
|
e8679f45b5 | ||
|
|
527e0798ae | ||
|
|
842408c0d9 | ||
|
|
ad6eb7a9a6 | ||
|
|
7f551081ad | ||
|
|
f41ffdd039 | ||
|
|
a187703a3a | ||
|
|
86c2a36f00 | ||
|
|
27a343bc75 | ||
|
|
672b89ce38 | ||
|
|
f4984e3962 | ||
|
|
7b5331ddcd | ||
|
|
ce6911c196 | ||
|
|
54b21fa9ff | ||
|
|
ff86a421f4 | ||
|
|
ee070371a8 | ||
|
|
41b5448f56 | ||
|
|
63061e6e7e | ||
|
|
ccdd1e7e49 | ||
|
|
64e748ea94 | ||
|
|
bb654d83f8 | ||
|
|
952f5c66e3 | ||
|
|
f81c94af43 | ||
|
|
febe4b72bc | ||
|
|
3768657b91 | ||
|
|
2a8fcb3bd3 | ||
|
|
fc31013ceb | ||
|
|
b37cd79432 | ||
|
|
7e464a2828 | ||
|
|
01a923352f | ||
|
|
7f5d3517a3 | ||
|
|
d56bbf6db0 | ||
|
|
6c4318d3ae | ||
|
|
f36ff4c6e3 | ||
|
|
44dbd5188c | ||
|
|
b0b68f2edd | ||
|
|
72c54edce2 | ||
|
|
33be713d0c | ||
|
|
a5b36678ed | ||
|
|
c91e44f796 | ||
|
|
df2dfc5d71 | ||
|
|
e36a7a193c | ||
|
|
5d00c42f5a | ||
|
|
cb9d99dd69 | ||
|
|
4f9a6f64e4 | ||
|
|
12fa954a05 | ||
| 7181a61cec | |||
|
|
cdf3b5c045 | ||
| f3c50f71ef | |||
|
|
98b0ccc6c9 | ||
|
|
37a3b51e57 | ||
|
|
d6e1f3c74d | ||
|
|
52e831b8c1 | ||
| f5af24699a | |||
|
|
dd92147341 | ||
|
|
04faa079e2 | ||
| 95c99cb4d5 | |||
|
|
2eda926d4c | ||
|
|
70258a0cc3 | ||
|
|
e9944c5413 | ||
| 22e2946e95 | |||
|
|
9ddab24931 | ||
|
|
ae91f2895e | ||
|
|
06fc40a857 | ||
|
|
a68d6d617e | ||
|
|
c49a650888 | ||
|
|
87e09af941 | ||
|
|
6f13e284fd | ||
|
|
6c963a15bd | ||
| 8c737d163d |
6
.gitignore
vendored
6
.gitignore
vendored
@@ -31,3 +31,9 @@ node_modules/
|
||||
# Asahi build artifacts (large)
|
||||
bastion/.asahi-cache/
|
||||
bastion/asahi-repo/*.zip
|
||||
|
||||
# Regenerated by labsim/dualstack-lab.sh; derived state, not source.
|
||||
labsim/dualstack-evidence/
|
||||
|
||||
# Runtime snapshots from labsim/cilium-ipam-switch.sh
|
||||
labsim/.ipam-switch-state/
|
||||
|
||||
19
CLAUDE.md
Normal file
19
CLAUDE.md
Normal file
@@ -0,0 +1,19 @@
|
||||
## Skill routing
|
||||
|
||||
When the user's request matches an available skill, ALWAYS invoke it using the Skill
|
||||
tool as your FIRST action. Do NOT answer directly, do NOT use other tools first.
|
||||
The skill has specialized workflows that produce better results than ad-hoc answers.
|
||||
|
||||
Key routing rules:
|
||||
- Product ideas, "is this worth building", brainstorming → invoke gstack-office-hours
|
||||
- Bugs, errors, "why is this broken", 500 errors → invoke gstack-investigate
|
||||
- Ship, deploy, push, create PR → invoke gstack-ship
|
||||
- QA, test the site, find bugs → invoke gstack-qa
|
||||
- Code review, check my diff → invoke gstack-review
|
||||
- Update docs after shipping → invoke gstack-document-release
|
||||
- Weekly retro → invoke gstack-retro
|
||||
- Design system, brand → invoke gstack-design-consultation
|
||||
- Visual audit, design polish → invoke gstack-design-review
|
||||
- Architecture review → invoke gstack-plan-eng-review
|
||||
- Save progress, checkpoint, resume → invoke gstack-checkpoint
|
||||
- Code quality, health check → invoke gstack-health
|
||||
47
TODOS.md
Normal file
47
TODOS.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# TODOS
|
||||
|
||||
## P1 — Ship with Phase 1
|
||||
|
||||
### v2.0 Architecture Document Update
|
||||
Update `bastion/docs/ARCHITECTURE.md` to cover v2.0: driver model, fleet system,
|
||||
Pulumi integration, Vault secrets, Deno evaluator, new CLI grammar. The existing
|
||||
doc covers v1.0 comprehensively (432 lines). v2.0 adds 5+ major subsystems.
|
||||
**Effort:** M (human: 1 week / CC: 1-2 days)
|
||||
**Depends on:** Phase 1 complete
|
||||
**Source:** CEO review 2026-04-01
|
||||
|
||||
## P2 — Post-v2.0 Core
|
||||
|
||||
### SSH Emergency Mode (scoped)
|
||||
SSH-based operations limited to: (1) earliest necessary box provisioning before agent
|
||||
is installed, and (2) emergency debugging/fixing operations that can't be done via agent.
|
||||
NOT a general-purpose DeploymentTarget alternative. The v1.0 `recheck` and `fix-ssh-root.sh`
|
||||
patterns are the model. Agent stays the primary management path.
|
||||
**Effort:** S (human: 1 week / CC: 1 day)
|
||||
**Depends on:** Phase 2 complete (DeploymentTarget interface exists)
|
||||
**Source:** CEO review 2026-04-01
|
||||
|
||||
### Prometheus Metrics Endpoint
|
||||
Add `/metrics` endpoint to labd: resource counts by status, apply duration histograms,
|
||||
driver operation latency, fleet pipeline completion rates. Standard Prometheus scraping
|
||||
for Grafana dashboards and alerting.
|
||||
**Effort:** S (human: 2-3 days / CC: 2-3 hours)
|
||||
**Depends on:** Phase 1 (labd exists with resource store)
|
||||
**Source:** CEO review 2026-04-01 (observability gap)
|
||||
|
||||
## P3 — Future Enhancements
|
||||
|
||||
### Infrastructure Graph Visualization
|
||||
Visual representation of resource dependencies, environment topology, fleet status.
|
||||
Could be a web UI or terminal-based (like `kubectl tree`).
|
||||
**Source:** CEO review 2026-04-01
|
||||
|
||||
### `labctl import` for Existing Cloud Resources
|
||||
Discover and import existing AWS/GCP resources into the state store.
|
||||
Pulumi's import functionality could be leveraged.
|
||||
**Source:** CEO review 2026-04-01
|
||||
|
||||
### Built-in Secrets Rotation
|
||||
Automatic rotation of managed secrets (database passwords, API keys).
|
||||
Vault handles rotation but a labctl-native workflow could simplify.
|
||||
**Source:** CEO review 2026-04-01
|
||||
@@ -11,6 +11,7 @@ WORKDIR /app
|
||||
# Copy workspace config and package manifests first (layer cache)
|
||||
COPY pnpm-workspace.yaml pnpm-lock.yaml package.json tsconfig.base.json tsconfig.json ./
|
||||
COPY src/shared/package.json src/shared/tsconfig.json src/shared/
|
||||
COPY src/core/package.json src/core/tsconfig.json src/core/
|
||||
COPY src/labd/package.json src/labd/tsconfig.json src/labd/
|
||||
|
||||
# Install all dependencies (dev included -- needed for build)
|
||||
@@ -22,10 +23,13 @@ RUN pnpm --filter @lab/labd exec prisma generate
|
||||
|
||||
# Copy source code
|
||||
COPY src/shared/src/ src/shared/src/
|
||||
COPY src/core/src/ src/core/src/
|
||||
COPY src/labd/src/ src/labd/src/
|
||||
|
||||
# Build TypeScript (shared first via project references)
|
||||
RUN pnpm --filter @lab/shared build && pnpm --filter @lab/labd build
|
||||
# Build TypeScript (shared + core before labd via project references)
|
||||
RUN pnpm --filter @lab/shared build \
|
||||
&& pnpm --filter @lab/core build \
|
||||
&& pnpm --filter @lab/labd build
|
||||
|
||||
# Hoist the generated Prisma client so stage 2 can COPY it from a stable path
|
||||
RUN mkdir -p /app/_prisma && \
|
||||
@@ -41,6 +45,7 @@ WORKDIR /app
|
||||
# Copy workspace config and package manifests
|
||||
COPY pnpm-workspace.yaml pnpm-lock.yaml package.json ./
|
||||
COPY src/shared/package.json src/shared/
|
||||
COPY src/core/package.json src/core/
|
||||
COPY src/labd/package.json src/labd/
|
||||
|
||||
# Install production dependencies only
|
||||
@@ -48,6 +53,7 @@ RUN pnpm install --frozen-lockfile --prod 2>/dev/null || pnpm install --prod
|
||||
|
||||
# Copy built output from builder
|
||||
COPY --from=builder /app/src/shared/dist/ src/shared/dist/
|
||||
COPY --from=builder /app/src/core/dist/ src/core/dist/
|
||||
COPY --from=builder /app/src/labd/dist/ src/labd/dist/
|
||||
|
||||
# Copy Prisma schema + generated client into pnpm store location
|
||||
|
||||
4
bastion/bastion/.gitignore
vendored
Normal file
4
bastion/bastion/.gitignore
vendored
Normal file
@@ -0,0 +1,4 @@
|
||||
|
||||
# Asahi build artifacts (large)
|
||||
.asahi-cache/
|
||||
asahi-repo/*.zip
|
||||
@@ -59,10 +59,10 @@ _labctl() {
|
||||
COMPREPLY=($(compgen -W "-h --help" -- "$cur"))
|
||||
return ;;
|
||||
"provision install")
|
||||
COMPREPLY=($(compgen -W "--role --os --disk -h --help" -- "$cur"))
|
||||
COMPREPLY=($(compgen -W "--role --os --disk --vyos-mgmt --vyos-mgmt-address --vyos-bond --vyos-bond-address --vyos-bond-vrrp --vlan-vip --vyos-vrrp-priority --vyos-mgmt-vlan --vlan --vyos-password --vyos-hwid --vyos-fresh-config -h --help" -- "$cur"))
|
||||
return ;;
|
||||
"provision reprovision")
|
||||
COMPREPLY=($(compgen -W "--role --os --disk -h --help" -- "$cur"))
|
||||
COMPREPLY=($(compgen -W "--role --os --disk --user -h --help" -- "$cur"))
|
||||
return ;;
|
||||
"provision debug")
|
||||
COMPREPLY=($(compgen -W "--pxe-boot -h --help" -- "$cur"))
|
||||
@@ -82,6 +82,9 @@ _labctl() {
|
||||
"provision makeiso")
|
||||
COMPREPLY=($(compgen -W "--arch --local --out -h --help" -- "$cur"))
|
||||
return ;;
|
||||
"provision recheck")
|
||||
COMPREPLY=($(compgen -W "--user --target -h --help" -- "$cur"))
|
||||
return ;;
|
||||
"config list")
|
||||
COMPREPLY=($(compgen -W "-h --help" -- "$cur"))
|
||||
return ;;
|
||||
@@ -107,7 +110,7 @@ _labctl() {
|
||||
COMPREPLY=($(compgen -W "bastion -h --help" -- "$cur"))
|
||||
return ;;
|
||||
"provision")
|
||||
COMPREPLY=($(compgen -W "list install reprovision debug forget register asahi logs makeiso -h --help" -- "$cur"))
|
||||
COMPREPLY=($(compgen -W "list install reprovision debug forget register asahi logs makeiso recheck -h --help" -- "$cur"))
|
||||
return ;;
|
||||
"config")
|
||||
COMPREPLY=($(compgen -W "list get set path -h --help" -- "$cur"))
|
||||
|
||||
@@ -128,16 +128,30 @@ complete -c labctl -n "__labctl_using_cmd provision" -a register -d 'Register an
|
||||
complete -c labctl -n "__labctl_using_cmd provision" -a asahi -d 'Show instructions to provision an Apple Silicon Mac with Asahi Linux'
|
||||
complete -c labctl -n "__labctl_using_cmd provision" -a logs -d 'Show provisioning logs for a machine (hostname, MAC, or IP)'
|
||||
complete -c labctl -n "__labctl_using_cmd provision" -a makeiso -d 'Generate a UEFI-bootable iPXE ISO for network provisioning'
|
||||
complete -c labctl -n "__labctl_using_cmd provision" -a recheck -d 'Refresh hardware info for all installed machines via SSH'
|
||||
|
||||
# provision install options
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller'
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04'
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04 vyos-rolling'
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l disk -d 'Target disk device (auto-detect if omitted)' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt -d 'VyOS: untagged interface the machine PXE boots from (default eth0)' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt-address -d 'VyOS: CIDR for the management interface, or \'dhcp\' (default dhcp)' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond -d 'VyOS: comma-separated LACP bond members (must exclude the PXE NIC)' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond-address -d 'VyOS: address on the untagged bond (trunk native VLAN)' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond-vrrp -d 'VyOS: VRRP VIP floated on the untagged bond' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l vlan-vip -d 'VyOS: VRRP VIP for a --vlan entry (repeatable)' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-vrrp-priority -d 'VyOS: VRRP priority for all groups on this box (higher = master)' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt-vlan -d 'VyOS: tagged management VLAN on the PXE port' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l vlan -d 'VyOS: tagged VLAN sub-interface on the bond (repeatable)' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-password -d 'VyOS: password for the \'vyos\' user' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-hwid -d 'VyOS: pin an interface name to a MAC via hw-id (repeatable)' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-fresh-config -d 'VyOS: on reinstall, overwrite the preserved config with the generated one'
|
||||
|
||||
# provision reprovision options
|
||||
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller'
|
||||
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04'
|
||||
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04 vyos-rolling'
|
||||
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l disk -d 'Target disk device (auto-detect if omitted)' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l user -d 'SSH user for the reboot (default: vyos for VyOS machines, else current user)' -x
|
||||
|
||||
# provision debug options
|
||||
complete -c labctl -n "__labctl_in_cmd provision debug" -l pxe-boot -d 'Boot installed system via PXE (kernel+initrd from network, root from NVMe)'
|
||||
@@ -154,6 +168,10 @@ complete -c labctl -n "__labctl_in_cmd provision makeiso" -l arch -d 'Target arc
|
||||
complete -c labctl -n "__labctl_in_cmd provision makeiso" -l local -d 'Build ISO locally instead of using bastion-hosted URL'
|
||||
complete -c labctl -n "__labctl_in_cmd provision makeiso" -l out -d 'Output path for local ISO build' -x
|
||||
|
||||
# provision recheck options
|
||||
complete -c labctl -n "__labctl_in_cmd provision recheck" -l user -d 'SSH user' -x
|
||||
complete -c labctl -n "__labctl_in_cmd provision recheck" -l target -d 'Only recheck a specific machine (by hostname or MAC)' -x
|
||||
|
||||
# config subcommands
|
||||
complete -c labctl -n "__labctl_using_cmd config" -a list -d 'Show all configuration values'
|
||||
complete -c labctl -n "__labctl_using_cmd config" -a get -d 'Get a configuration value'
|
||||
|
||||
@@ -21,6 +21,8 @@
|
||||
"test:integration:pxe:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'PXE boot'",
|
||||
"test:integration:iso": "vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
|
||||
"test:integration:iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
|
||||
"test:integration:vyos": "vitest run -c tests/integration/vitest.config.ts -t 'VyOS provisioning'",
|
||||
"test:integration:vyos:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'VyOS provisioning'",
|
||||
"test:integration:arm-iso": "vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
|
||||
"test:integration:arm-iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
|
||||
"test:integration:asahi": "vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
|
||||
|
||||
1847
bastion/pnpm-lock.yaml
generated
1847
bastion/pnpm-lock.yaml
generated
File diff suppressed because it is too large
Load Diff
@@ -99,16 +99,22 @@ if [ "$PUSH" = true ]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
# Use --tls-verify=false for plain HTTP registries (e.g. 10.0.0.194:3012)
|
||||
TLS_FLAG=""
|
||||
if [[ "$REGISTRY" =~ ^[0-9] ]] || [[ "$REGISTRY" =~ ^localhost ]]; then
|
||||
TLS_FLAG="--tls-verify=false"
|
||||
fi
|
||||
|
||||
echo "==> Logging in to $REGISTRY..."
|
||||
podman login -u michal -p "$GITEA_TOKEN" "$REGISTRY"
|
||||
podman login $TLS_FLAG -u michal -p "$GITEA_TOKEN" "$REGISTRY"
|
||||
|
||||
echo "==> Pushing $FULL_IMAGE:$TAG..."
|
||||
podman manifest push --all "$MANIFEST" "docker://$FULL_IMAGE:$TAG"
|
||||
podman manifest push --all $TLS_FLAG "$MANIFEST" "docker://$FULL_IMAGE:$TAG"
|
||||
|
||||
# Also tag as :latest if not already
|
||||
if [ "$TAG" != "latest" ]; then
|
||||
echo "==> Also pushing as :latest..."
|
||||
podman manifest push --all "$MANIFEST" "docker://$FULL_IMAGE:latest"
|
||||
podman manifest push --all $TLS_FLAG "$MANIFEST" "docker://$FULL_IMAGE:latest"
|
||||
fi
|
||||
|
||||
# Link package to repository if script exists
|
||||
|
||||
@@ -92,15 +92,21 @@ if [ "$PUSH" = true ]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
# Use --tls-verify=false for plain HTTP registries (e.g. 10.0.0.194:3012)
|
||||
TLS_FLAG=""
|
||||
if [[ "$REGISTRY" =~ ^[0-9] ]] || [[ "$REGISTRY" =~ ^localhost ]]; then
|
||||
TLS_FLAG="--tls-verify=false"
|
||||
fi
|
||||
|
||||
echo "==> Logging in to $REGISTRY..."
|
||||
podman login -u michal -p "$GITEA_TOKEN" "$REGISTRY"
|
||||
podman login $TLS_FLAG -u michal -p "$GITEA_TOKEN" "$REGISTRY"
|
||||
|
||||
echo "==> Pushing $FULL_IMAGE:$TAG..."
|
||||
podman manifest push --all "$MANIFEST" "docker://$FULL_IMAGE:$TAG"
|
||||
podman manifest push --all $TLS_FLAG "$MANIFEST" "docker://$FULL_IMAGE:$TAG"
|
||||
|
||||
if [ "$TAG" != "latest" ]; then
|
||||
echo "==> Also pushing as :latest..."
|
||||
podman manifest push --all "$MANIFEST" "docker://$FULL_IMAGE:latest"
|
||||
podman manifest push --all $TLS_FLAG "$MANIFEST" "docker://$FULL_IMAGE:latest"
|
||||
fi
|
||||
|
||||
if [ -f "$SCRIPT_DIR/link-package.sh" ]; then
|
||||
|
||||
131
bastion/scripts/fix-ssh-root.sh
Normal file
131
bastion/scripts/fix-ssh-root.sh
Normal file
@@ -0,0 +1,131 @@
|
||||
#!/bin/bash
|
||||
# Fix root SSH access on all provisioned machines.
|
||||
# Tries root, lab, michal users to find one that works,
|
||||
# then ensures root has the SSH key and PermitRootLogin is enabled.
|
||||
set -euo pipefail
|
||||
|
||||
SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMJ3FkUGbG174eoO5RjZd2eNV680FM5pgp0AgpW/QwlJExK3qxMk0DJSr4ICmzGUx4yujAXcrqU1otcOMPzzFzwc5heWpSmlNHU3TIW6NHEt0sF9ZTAbGLw2zSw3si5UouqFkCcENA40mePFJqY+Q9R8N1uvLgu4m/do+Zrn/mk5Ewc1V7OCRE5Acrnaec4T7LTB0BuVXcjPUfAmZ0q5fI+bKPR1q2Kc3+IeGhVkBuZ9OJVeXXhnpedm0uEbLeriK/jUYKYw/1QhsNDM8Tyty+UIGr9QVnWwzCMHB+wuQcDYC9mPGTqg0fYwX8Mp8xMi1PPxdsh1G7bj/cpWMAF43KswWORF2ul8ICGbaE1zEgIYXO790SuBjpBHhaC6Iegqi58hmCuP+a9893q/EU9HyrWTJHCZXC5E4kP1MsM57KrhEpszM6I3sW9f9zMTPd5QsCXFi4si4OMwX4kYNVu3fQGQPpseDPlTTSrT6uUdqj4Irm0c1m9cYTmK0vYgsM3ss= michal@fedora"
|
||||
|
||||
SSH_OPTS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=5"
|
||||
USERS_TO_TRY=(root lab michal)
|
||||
|
||||
# Machines: hostname ip
|
||||
MACHINES=(
|
||||
"labmaster 192.168.8.11"
|
||||
"worker0-k8s0 192.168.8.23"
|
||||
"worker1-k8s0 192.168.8.13"
|
||||
"worker2-k8s0 192.168.8.25"
|
||||
"spark-2935 192.168.8.12"
|
||||
)
|
||||
|
||||
BOLD="\033[1m"
|
||||
GREEN="\033[0;32m"
|
||||
RED="\033[0;31m"
|
||||
DIM="\033[2m"
|
||||
RESET="\033[0m"
|
||||
|
||||
# Script to run on each machine (via sudo if needed)
|
||||
read -r -d '' FIX_SCRIPT << 'FIXEOF' || true
|
||||
#!/bin/bash
|
||||
set -e
|
||||
KEY="$1"
|
||||
|
||||
# 1. Ensure root .ssh dir exists
|
||||
mkdir -p /root/.ssh
|
||||
chmod 700 /root/.ssh
|
||||
touch /root/.ssh/authorized_keys
|
||||
chmod 600 /root/.ssh/authorized_keys
|
||||
|
||||
# 2. Add key if not present
|
||||
if ! grep -qF "$KEY" /root/.ssh/authorized_keys 2>/dev/null; then
|
||||
echo "$KEY" >> /root/.ssh/authorized_keys
|
||||
echo "KEY_ADDED"
|
||||
else
|
||||
echo "KEY_EXISTS"
|
||||
fi
|
||||
|
||||
# 3. Fix sshd_config for root login with keys
|
||||
SSHD_CONF="/etc/ssh/sshd_config"
|
||||
CHANGED=0
|
||||
|
||||
# Ensure PermitRootLogin allows key auth
|
||||
CURRENT=$(grep -E "^PermitRootLogin" "$SSHD_CONF" 2>/dev/null | tail -1 || true)
|
||||
if [ "$CURRENT" = "PermitRootLogin prohibit-password" ] || [ "$CURRENT" = "PermitRootLogin without-password" ]; then
|
||||
echo "SSHD_OK"
|
||||
elif [ "$CURRENT" = "PermitRootLogin yes" ]; then
|
||||
echo "SSHD_OK"
|
||||
else
|
||||
# Remove any existing PermitRootLogin lines
|
||||
sed -i '/^#*PermitRootLogin/d' "$SSHD_CONF"
|
||||
echo "PermitRootLogin prohibit-password" >> "$SSHD_CONF"
|
||||
CHANGED=1
|
||||
echo "SSHD_FIXED"
|
||||
fi
|
||||
|
||||
# Ensure PubkeyAuthentication is enabled
|
||||
if grep -qE "^PubkeyAuthentication no" "$SSHD_CONF" 2>/dev/null; then
|
||||
sed -i 's/^PubkeyAuthentication no/PubkeyAuthentication yes/' "$SSHD_CONF"
|
||||
CHANGED=1
|
||||
echo "PUBKEY_FIXED"
|
||||
else
|
||||
echo "PUBKEY_OK"
|
||||
fi
|
||||
|
||||
# Restart sshd if changed
|
||||
if [ "$CHANGED" -eq 1 ]; then
|
||||
systemctl restart sshd 2>/dev/null || systemctl restart ssh 2>/dev/null || true
|
||||
echo "SSHD_RESTARTED"
|
||||
fi
|
||||
|
||||
# 4. Verify root can be reached
|
||||
echo "DONE"
|
||||
FIXEOF
|
||||
|
||||
echo ""
|
||||
echo -e "${BOLD}Fixing root SSH access on all machines...${RESET}"
|
||||
echo ""
|
||||
|
||||
for entry in "${MACHINES[@]}"; do
|
||||
read -r hostname ip <<< "$entry"
|
||||
printf " %-24s ${DIM}(%s)${RESET} " "$hostname" "$ip"
|
||||
|
||||
# Try each user until one works
|
||||
WORKING_USER=""
|
||||
for user in "${USERS_TO_TRY[@]}"; do
|
||||
if ssh $SSH_OPTS "$user@$ip" "true" 2>/dev/null; then
|
||||
WORKING_USER="$user"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -z "$WORKING_USER" ]; then
|
||||
echo -e "${RED}UNREACHABLE${RESET} (tried: ${USERS_TO_TRY[*]})"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Run fix script (with sudo if not root)
|
||||
if [ "$WORKING_USER" = "root" ]; then
|
||||
RESULT=$(ssh $SSH_OPTS "root@$ip" "bash -s -- '$SSH_KEY'" <<< "$FIX_SCRIPT" 2>&1)
|
||||
else
|
||||
RESULT=$(ssh $SSH_OPTS "$WORKING_USER@$ip" "sudo bash -s -- '$SSH_KEY'" <<< "$FIX_SCRIPT" 2>&1)
|
||||
fi
|
||||
|
||||
# Parse result
|
||||
DETAILS=""
|
||||
if echo "$RESULT" | grep -q "KEY_ADDED"; then DETAILS="key added"; fi
|
||||
if echo "$RESULT" | grep -q "KEY_EXISTS"; then DETAILS="key ok"; fi
|
||||
if echo "$RESULT" | grep -q "SSHD_FIXED"; then DETAILS="$DETAILS, sshd fixed"; fi
|
||||
if echo "$RESULT" | grep -q "SSHD_OK"; then DETAILS="$DETAILS, sshd ok"; fi
|
||||
if echo "$RESULT" | grep -q "SSHD_RESTARTED"; then DETAILS="$DETAILS, restarted"; fi
|
||||
|
||||
# Verify root works now
|
||||
if ssh $SSH_OPTS "root@$ip" "true" 2>/dev/null; then
|
||||
echo -e "${GREEN}OK${RESET} ${DIM}(via $WORKING_USER: $DETAILS)${RESET}"
|
||||
else
|
||||
echo -e "${RED}PARTIAL${RESET} ${DIM}(via $WORKING_USER: $DETAILS -- root still blocked)${RESET}"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo -e "${BOLD}Done.${RESET} Verify: labctl provision recheck --user root"
|
||||
echo ""
|
||||
@@ -20,6 +20,15 @@ export function loadConfig(overrides: Partial<BastionConfig> = {}): BastionConfi
|
||||
const ubuntuMirror = overrides.ubuntuMirror ?? process.env["UBUNTU_MIRROR"]
|
||||
?? `https://releases.ubuntu.com/${ubuntuVersion}`;
|
||||
|
||||
// "latest" resolves the newest nightly ISO from the vyos-nightly-build GitHub
|
||||
// releases at startup. downloads.vyos.io no longer serves direct rolling ISOs
|
||||
// (it returns the vyos.io site, and nightly builds sit behind a signup form);
|
||||
// GitHub releases are the remaining free, unauthenticated direct source.
|
||||
// LTS ISOs are subscription-only. Set VYOS_ISO_URL to pin a specific build.
|
||||
const vyosIsoUrl = overrides.vyosIsoUrl ?? process.env["VYOS_ISO_URL"] ?? "latest";
|
||||
const vyosDefaultPassword = overrides.vyosDefaultPassword
|
||||
?? process.env["VYOS_DEFAULT_PASSWORD"] ?? "vyos";
|
||||
|
||||
const fedoraMirror = `https://download.fedoraproject.org/pub/fedora/linux/releases/${fedoraVersion}/Everything/${arch}/os`;
|
||||
const tftpDir = `${bastionDir}/tftp`;
|
||||
const httpDir = `${bastionDir}/http`;
|
||||
@@ -38,6 +47,8 @@ export function loadConfig(overrides: Partial<BastionConfig> = {}): BastionConfi
|
||||
dhcpRangeEnd,
|
||||
ubuntuVersion,
|
||||
ubuntuMirror,
|
||||
vyosIsoUrl,
|
||||
vyosDefaultPassword,
|
||||
// These are populated at runtime by the network service
|
||||
iface: overrides.iface ?? "",
|
||||
serverIp: overrides.serverIp ?? "",
|
||||
|
||||
@@ -40,6 +40,125 @@ function download(url: string, dest: string, label: string): void {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Pick the largest regular-file initrd from an `xorriso -lsl` listing.
|
||||
*
|
||||
* /live carries decoys: a 0-byte initrd.img placeholder on some images, or an
|
||||
* initrd.img SYMLINK to the real version-suffixed file on others. Parsing is
|
||||
* field-based (ls -l layout: perms links uid gid size month day time 'name')
|
||||
* and considers only lines whose mode string marks a regular file — symlinks
|
||||
* report their link size, not the target's, and must not win.
|
||||
*/
|
||||
export function pickLargestInitrd(
|
||||
listing: string,
|
||||
): { name: string; size: number } | undefined {
|
||||
let best: { name: string; size: number } | undefined;
|
||||
for (const line of listing.split("\n")) {
|
||||
if (!line.startsWith("-")) continue; // regular files only
|
||||
const quoted = /'([^']+)'/.exec(line);
|
||||
const fields = line.trim().split(/\s+/);
|
||||
const size = parseInt(fields[4] ?? "", 10);
|
||||
const name = quoted?.[1] ?? "";
|
||||
if (!name.startsWith("initrd")) continue;
|
||||
if (!Number.isFinite(size) || size <= 0) continue;
|
||||
if (best === undefined || size > best.size) {
|
||||
best = { name, size };
|
||||
}
|
||||
}
|
||||
return best;
|
||||
}
|
||||
|
||||
const VYOS_NIGHTLY_RELEASES =
|
||||
"https://api.github.com/repos/vyos/vyos-nightly-build/releases/latest";
|
||||
|
||||
/**
|
||||
* Resolve the configured VyOS ISO URL, expanding the "latest" sentinel.
|
||||
*
|
||||
* The nightly asset filename embeds a build date, so there is no stable
|
||||
* "latest.iso" path to hardcode — the newest release has to be looked up.
|
||||
* Any other value is used verbatim, which is how VYOS_ISO_URL pins a build
|
||||
* or points at a locally mirrored copy.
|
||||
*/
|
||||
function resolveVyosIsoUrl(configured: string): string {
|
||||
if (configured !== "latest") return configured;
|
||||
|
||||
const body = execSync(`curl -sSfL "${VYOS_NIGHTLY_RELEASES}"`, {
|
||||
encoding: "utf-8",
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
});
|
||||
const release = JSON.parse(body) as {
|
||||
tag_name?: string;
|
||||
assets?: Array<{ name: string; browser_download_url: string }>;
|
||||
};
|
||||
|
||||
const asset = (release.assets ?? []).find((a) =>
|
||||
/generic-amd64\.iso$/.test(a.name),
|
||||
);
|
||||
if (!asset) {
|
||||
throw new Error(
|
||||
`No generic-amd64 ISO asset in VyOS nightly release ${release.tag_name ?? "?"}`,
|
||||
);
|
||||
}
|
||||
|
||||
logger.info(` VyOS ISO resolved to ${asset.name} (${release.tag_name ?? "?"})`);
|
||||
return asset.browser_download_url;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract VyOS netboot artifacts from the release ISO.
|
||||
*
|
||||
* VyOS publishes no netboot bundle, so kernel/initrd/squashfs have to come out
|
||||
* of the ISO. xorriso is already in the bastion image (used for boot.iso) and
|
||||
* extracts without root or a loop mount.
|
||||
*
|
||||
* The initrd needs care: /live contains an empty initrd.img placeholder
|
||||
* alongside the real one, which carries a version-suffixed name. Booting the
|
||||
* 0-byte file fails with no useful diagnostic, so pick the largest initrd*.
|
||||
*/
|
||||
export function prepareVyosArtifacts(config: BastionConfig): void {
|
||||
const kernel = `${config.httpDir}/vyos-vmlinuz`;
|
||||
const initrd = `${config.httpDir}/vyos-initrd`;
|
||||
const squashfs = `${config.httpDir}/vyos-filesystem.squashfs`;
|
||||
|
||||
if (existsSync(kernel) && existsSync(initrd) && existsSync(squashfs)) {
|
||||
logger.info(" VyOS netboot artifacts -- cached");
|
||||
return;
|
||||
}
|
||||
|
||||
const iso = `${config.bastionDir}/vyos.iso`;
|
||||
download(resolveVyosIsoUrl(config.vyosIsoUrl), iso, "VyOS ISO");
|
||||
|
||||
const extract = (isoPath: string, dest: string, label: string): void => {
|
||||
execSync(
|
||||
`xorriso -osirrox on -indev "${iso}" -extract "${isoPath}" "${dest}"`,
|
||||
{ stdio: "pipe" },
|
||||
);
|
||||
logger.info(` ${label} -- extracted from ${isoPath}`);
|
||||
};
|
||||
|
||||
extract("/live/vmlinuz", kernel, "VyOS kernel");
|
||||
extract("/live/filesystem.squashfs", squashfs, "VyOS squashfs");
|
||||
|
||||
// Pick the real initrd by size from the ISO's own directory listing.
|
||||
const listing = execSync(`xorriso -indev "${iso}" -lsl /live/ --`, {
|
||||
encoding: "utf-8",
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
});
|
||||
|
||||
const best = pickLargestInitrd(listing);
|
||||
if (best === undefined) {
|
||||
throw new Error("No non-empty initrd found in /live on the VyOS ISO");
|
||||
}
|
||||
extract(`/live/${best.name}`, initrd, `VyOS initrd (${best.name}, ${best.size} bytes)`);
|
||||
|
||||
// The ISO is only needed to produce the three artifacts above.
|
||||
try {
|
||||
unlinkSync(iso);
|
||||
} catch {
|
||||
// Non-fatal: leaving it costs disk but nothing else.
|
||||
}
|
||||
}
|
||||
|
||||
function symlinkSafe(target: string, linkPath: string): void {
|
||||
try {
|
||||
symlinkSync(target, linkPath);
|
||||
@@ -182,6 +301,17 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
|
||||
logger.warn(`Ubuntu ${config.ubuntuVersion} artifacts not available -- Ubuntu provisioning disabled`);
|
||||
}
|
||||
|
||||
// VyOS netboot artifacts (non-fatal — same policy as Ubuntu)
|
||||
try {
|
||||
logger.info("Preparing VyOS netboot artifacts...");
|
||||
prepareVyosArtifacts(config);
|
||||
} catch (err) {
|
||||
logger.warn(
|
||||
`VyOS artifacts not available -- VyOS provisioning disabled ` +
|
||||
`(${err instanceof Error ? err.message : String(err)})`,
|
||||
);
|
||||
}
|
||||
|
||||
// Symlink iPXE binaries into HTTP dir for UEFI HTTP Boot
|
||||
for (const name of ["ipxe.efi", "ipxe-arm64.efi"]) {
|
||||
const src = `${config.tftpDir}/${name}`;
|
||||
@@ -261,6 +391,7 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
|
||||
role: msg.role as import("@lab/shared").Role,
|
||||
os: msg.os as import("@lab/shared").OsId,
|
||||
queued_at: new Date().toISOString(),
|
||||
...(msg.vyos ? { vyos: msg.vyos } : {}),
|
||||
};
|
||||
});
|
||||
return { status: "ok", data: { mac: msg.mac, hostname: msg.hostname } };
|
||||
@@ -309,6 +440,32 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
|
||||
return { status: "ok", data: { mac, hostname: msg.hostname } };
|
||||
});
|
||||
|
||||
labdConn.onCommand("command-discover", async (msg) => {
|
||||
if (msg.type !== "command-discover") throw new Error("unexpected");
|
||||
const mac = (msg.mac as string).toLowerCase();
|
||||
const now = new Date().toISOString();
|
||||
const existing = state.load().discovered[mac];
|
||||
state.update((s) => {
|
||||
s.discovered[mac] = {
|
||||
mac,
|
||||
product: (msg.product as string) ?? "unknown",
|
||||
board: (msg.board as string) ?? "unknown",
|
||||
serial: (msg.serial as string) ?? "unknown",
|
||||
manufacturer: (msg.manufacturer as string) ?? "unknown",
|
||||
cpu_model: (msg.cpu_model as string) ?? "unknown",
|
||||
cpu_cores: (msg.cpu_cores as number) ?? 0,
|
||||
memory_gb: (msg.memory_gb as number) ?? 0,
|
||||
arch: (msg.arch as string) ?? "unknown",
|
||||
disks: (msg.disks as Array<{ name: string; size_gb: number; model: string }>) ?? [],
|
||||
nics: (msg.nics as Array<{ name: string; mac: string; state: string }>) ?? [],
|
||||
first_seen: existing?.first_seen ?? now,
|
||||
last_seen: now,
|
||||
};
|
||||
});
|
||||
logger.info(`HARDWARE UPDATED: ${mac} -- ${msg.manufacturer ?? "?"} ${msg.product ?? "?"} (${msg.cpu_model ?? "?"}, ${msg.cpu_cores ?? "?"} cores, ${msg.memory_gb ?? "?"}GB RAM)`);
|
||||
return { status: "ok", data: { mac } };
|
||||
});
|
||||
|
||||
labdConn.onCommand("command-role-update", async (msg) => {
|
||||
if (msg.type !== "command-role-update") throw new Error("unexpected");
|
||||
const mac = msg.mac.toLowerCase();
|
||||
|
||||
@@ -5,8 +5,8 @@
|
||||
// /api/discover - receive hardware discovery reports from PXE-booted machines
|
||||
|
||||
import type { FastifyInstance } from "fastify";
|
||||
import type { HardwareInfo, InstalledInfo, Role } from "@lab/shared";
|
||||
import { isValidOsId, SUPPORTED_ROLES } from "@lab/shared";
|
||||
import type { HardwareInfo, InstalledInfo, Role, VyosInstallSpec } from "@lab/shared";
|
||||
import { isValidOsId, SUPPORTED_ROLES, SUPPORTED_OS } from "@lab/shared";
|
||||
import type { StateManager } from "../services/state.js";
|
||||
import { logger } from "../services/logger.js";
|
||||
import { triggerPostProvisionK3s } from "../services/post-provision.js";
|
||||
@@ -15,6 +15,13 @@ import type { ProgressEvent } from "../services/progress-events.js";
|
||||
import type { InstallLogBuffer } from "../services/install-log.js";
|
||||
import type { SyslogListener } from "../services/syslog-listener.js";
|
||||
|
||||
/**
|
||||
* Seconds after dispatch with zero progress before a machine is called stalled.
|
||||
* Generous: the slowest legitimate gap is fetching a ~600MB VyOS squashfs over
|
||||
* HTTP before the hook can report anything.
|
||||
*/
|
||||
const STALL_THRESHOLD_S = 8 * 60;
|
||||
|
||||
export function registerApiRoutes(
|
||||
app: FastifyInstance,
|
||||
state: StateManager,
|
||||
@@ -34,9 +41,10 @@ export function registerApiRoutes(
|
||||
disk?: string;
|
||||
role?: string;
|
||||
os?: string;
|
||||
vyos?: VyosInstallSpec;
|
||||
};
|
||||
}>("/api/install", async (request, reply) => {
|
||||
const { mac: rawMac, hostname, disk, role, os } = request.body ?? {};
|
||||
const { mac: rawMac, hostname, disk, role, os, vyos } = request.body ?? {};
|
||||
const mac = (rawMac ?? "").toLowerCase().replace(/-/g, ":");
|
||||
|
||||
if (mac === "") {
|
||||
@@ -50,7 +58,7 @@ export function registerApiRoutes(
|
||||
|
||||
const osId = os ?? "fedora-43";
|
||||
if (!isValidOsId(osId)) {
|
||||
return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: fedora-43, ubuntu-26.04` });
|
||||
return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: ${SUPPORTED_OS.join(", ")}` });
|
||||
}
|
||||
|
||||
state.update((s) => {
|
||||
@@ -60,6 +68,7 @@ export function registerApiRoutes(
|
||||
role: validRole as Role,
|
||||
os: osId,
|
||||
queued_at: new Date().toISOString(),
|
||||
...(vyos ? { vyos } : {}),
|
||||
};
|
||||
});
|
||||
|
||||
@@ -139,20 +148,36 @@ export function registerApiRoutes(
|
||||
? detailStr.replace("ready at ", "").trim()
|
||||
: "";
|
||||
|
||||
const hw = s.discovered[mac];
|
||||
const installedInfo: InstalledInfo = {
|
||||
hostname: cfg?.hostname ?? "?",
|
||||
role: cfg?.role ?? "?",
|
||||
...(cfg?.os !== undefined ? { os: cfg.os } : {}),
|
||||
ip,
|
||||
installed_at: new Date().toISOString(),
|
||||
// Preserve hardware info from discovery
|
||||
...(hw ? {
|
||||
product: hw.product,
|
||||
manufacturer: hw.manufacturer,
|
||||
cpu_model: hw.cpu_model,
|
||||
cpu_cores: hw.cpu_cores,
|
||||
memory_gb: hw.memory_gb,
|
||||
arch: hw.arch,
|
||||
} : {}),
|
||||
};
|
||||
s.installed[mac] = installedInfo;
|
||||
|
||||
const admin = installedInfo.role !== "vanilla" && installedInfo.role !== "" ? "lab" : "root";
|
||||
// VyOS: the only login user is "vyos", and a router never runs k3s —
|
||||
// without this guard a non-vanilla role + recorded IP would trigger
|
||||
// the k3s post-provision against a VyOS box.
|
||||
const isVyos = (installedInfo.os ?? "").startsWith("vyos");
|
||||
const admin = isVyos
|
||||
? "vyos"
|
||||
: installedInfo.role !== "vanilla" && installedInfo.role !== "" ? "lab" : "root";
|
||||
console.log(`\n \x1b[0;32m\x1b[1m ssh ${admin}@${ip}\x1b[0m\n`); // eslint-disable-line no-console
|
||||
|
||||
// Auto-install k3s for non-vanilla roles
|
||||
if (installedInfo.role !== "vanilla" && ip !== "") {
|
||||
if (!isVyos && installedInfo.role !== "vanilla" && ip !== "") {
|
||||
void triggerPostProvisionK3s(installedInfo.hostname, ip, installedInfo.role, admin, mac);
|
||||
}
|
||||
}
|
||||
@@ -359,6 +384,23 @@ export function registerApiRoutes(
|
||||
});
|
||||
});
|
||||
|
||||
// Simple machine state query (used by ks-auto for ISO boot dispatch)
|
||||
app.get<{
|
||||
Params: { mac: string };
|
||||
}>("/api/machine-state/:mac", async (request, reply) => {
|
||||
const mac = request.params.mac.toLowerCase().replace(/-/g, ":");
|
||||
const currentState = state.load();
|
||||
|
||||
if (currentState.debug[mac]) return reply.send("debug");
|
||||
if (currentState.install_queue[mac]) {
|
||||
const progress = currentState.install_queue[mac].progress;
|
||||
return reply.send(progress ? "installing" : "queued");
|
||||
}
|
||||
if (currentState.installed[mac]) return reply.send("installed");
|
||||
if (currentState.discovered[mac]) return reply.send("discovered");
|
||||
return reply.send("unknown");
|
||||
});
|
||||
|
||||
// Update a machine's role (e.g. promote infra -> labcontroller)
|
||||
app.post<{
|
||||
Body: {
|
||||
@@ -407,6 +449,15 @@ export function registerApiRoutes(
|
||||
const installedEntry = currentState.installed[mac];
|
||||
|
||||
if (queueEntry) {
|
||||
// A machine that was handed an install script but has reported nothing
|
||||
// since is wedged BEFORE the installer environment came up — a bad
|
||||
// kernel/initrd, no network in the initramfs, or the wrong NIC picked.
|
||||
// Surfacing it here is what makes that diagnosable without a console.
|
||||
const since = queueEntry.progress_at ?? queueEntry.dispatched_at;
|
||||
const stalledForS = since !== undefined && queueEntry.progress === undefined
|
||||
? Math.floor((Date.now() - new Date(since).getTime()) / 1000)
|
||||
: 0;
|
||||
|
||||
return reply.send({
|
||||
mac,
|
||||
hostname: queueEntry.hostname,
|
||||
@@ -414,6 +465,9 @@ export function registerApiRoutes(
|
||||
progress: queueEntry.progress ?? "queued",
|
||||
progress_detail: queueEntry.progress_detail ?? "",
|
||||
progress_at: queueEntry.progress_at ?? queueEntry.queued_at,
|
||||
dispatched_at: queueEntry.dispatched_at,
|
||||
stalled_for_s: stalledForS,
|
||||
stalled: stalledForS > STALL_THRESHOLD_S,
|
||||
role: queueEntry.role,
|
||||
os: queueEntry.os,
|
||||
stages: queueEntry.log ?? [],
|
||||
|
||||
@@ -102,7 +102,8 @@ echo " - Standard Asahi boot infrastructure (m1n1 + U-Boot)"
|
||||
echo " - Fedora Asahi Remix root partition"
|
||||
echo " - LVM data partition (remaining space)"
|
||||
echo ""
|
||||
echo " On first boot, LVM volumes are created automatically."
|
||||
echo " After first boot, SSH in and set up LVM:"
|
||||
echo " ssh lab@<ip> 'curl -sf \${BASTION}/asahi/firstboot.sh | sudo bash'"
|
||||
echo ""
|
||||
|
||||
# Run the installer
|
||||
@@ -150,10 +151,10 @@ fi
|
||||
app.get<{
|
||||
Querystring: { hostname?: string; role?: string; mac?: string; user?: string };
|
||||
}>("/asahi/firstboot.sh", async (request, reply) => {
|
||||
const hostname = request.query.hostname ?? "mac-studio";
|
||||
const hostname = request.query.hostname ?? "unknown";
|
||||
const role = (request.query.role ?? "infra") as Role;
|
||||
const mac = request.query.mac ?? "unknown";
|
||||
const user = request.query.user ?? config.adminUser;
|
||||
const user = request.query.user ?? "lab";
|
||||
|
||||
const script = renderFirstbootScript({
|
||||
hostname,
|
||||
|
||||
@@ -137,7 +137,7 @@ function generateIso(config: BastionConfig, outputPath: string): void {
|
||||
"# Map iPXE arch names to Fedora mirror paths (arm64 -> aarch64)",
|
||||
"set fedarch ${buildarch}",
|
||||
"iseq ${buildarch} arm64 && set fedarch aarch64 ||",
|
||||
`kernel file:/vmlinuz-\${buildarch} inst.ks=${bastionUrl}/discover.ks inst.repo=${FEDORA_MIRROR_BASE}/${config.fedoraVersion}/Everything/\${fedarch}/os inst.text || goto no_kernel`,
|
||||
`kernel file:/vmlinuz-\${buildarch} inst.ks=${bastionUrl}/ks-auto inst.repo=${FEDORA_MIRROR_BASE}/${config.fedoraVersion}/Everything/\${fedarch}/os inst.text || goto no_kernel`,
|
||||
`initrd file:/initrd-\${buildarch} || goto no_kernel`,
|
||||
"boot || shell",
|
||||
"",
|
||||
|
||||
@@ -15,6 +15,7 @@ import {
|
||||
renderLocalBootIpxe,
|
||||
} from "../templates/boot.ipxe.js";
|
||||
import { renderUbuntuInstallIpxe } from "../templates/ubuntu-boot.ipxe.js";
|
||||
import { renderVyosInstallIpxe } from "../templates/vyos-boot.ipxe.js";
|
||||
import { renderDebugKickstart } from "../templates/debug.ks.js";
|
||||
import { logger } from "../services/logger.js";
|
||||
|
||||
@@ -99,8 +100,22 @@ echo "==============================="
|
||||
const os = queueEntry.os ?? "fedora-43";
|
||||
logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os})`);
|
||||
|
||||
// Stamp the handoff so a machine that boots the installer but never
|
||||
// reports can be spotted without a console.
|
||||
state.update((s) => {
|
||||
const entry = s.install_queue[mac];
|
||||
if (entry) entry.dispatched_at = new Date().toISOString();
|
||||
});
|
||||
|
||||
let script: string;
|
||||
if (os.startsWith("ubuntu")) {
|
||||
if (os.startsWith("vyos")) {
|
||||
script = renderVyosInstallIpxe({
|
||||
mac,
|
||||
hostname,
|
||||
serverIp: config.serverIp,
|
||||
httpPort: config.httpPort,
|
||||
});
|
||||
} else if (os.startsWith("ubuntu")) {
|
||||
script = renderUbuntuInstallIpxe({
|
||||
mac,
|
||||
hostname,
|
||||
|
||||
@@ -41,6 +41,150 @@ export function registerKickstartRoutes(
|
||||
return reply.type("text/plain").send(ks);
|
||||
});
|
||||
|
||||
// Auto-detecting kickstart for ISO boot (no-network machines like R1 ARM).
|
||||
// %pre detects MAC, queries bastion state, writes dynamic kickstart to /tmp.
|
||||
// Main body %include's it — so Anaconda gets either discover or install content.
|
||||
app.get("/ks-auto", async (_request, reply) => {
|
||||
const bastionUrl = `http://${config.serverIp}:${config.httpPort}`;
|
||||
|
||||
const ks = `# Lab Bastion -- Auto-detect kickstart (ISO boot)
|
||||
# %pre detects MAC, queries bastion state, writes /tmp/dynamic.ks.
|
||||
# Main body %include's it to get either discovery reboot or full install.
|
||||
|
||||
%pre --erroronfail --log=/tmp/ks-auto.log
|
||||
#!/bin/bash
|
||||
set -x
|
||||
|
||||
# -- Detect MAC address --
|
||||
MAC=$(ip link show | awk '/ether/ && !/00:00:00:00/ {print $2; exit}')
|
||||
echo "Detected MAC: $MAC"
|
||||
|
||||
# -- Wait for network (Linux drivers may take a moment) --
|
||||
for i in $(seq 1 30); do
|
||||
if curl -sf "${bastionUrl}/healthz" >/dev/null 2>&1; then
|
||||
echo "Bastion reachable at ${bastionUrl}"
|
||||
break
|
||||
fi
|
||||
echo "Waiting for network... ($i/30)"
|
||||
sleep 2
|
||||
done
|
||||
|
||||
# -- Query bastion for machine state --
|
||||
STATE=$(curl -sf "${bastionUrl}/api/machine-state/$MAC" 2>/dev/null || echo "unknown")
|
||||
echo "Machine state: $STATE"
|
||||
|
||||
case "$STATE" in
|
||||
queued|installing)
|
||||
echo "=== Machine queued for install. Fetching install kickstart... ==="
|
||||
curl -sf "${bastionUrl}/ks?mac=$MAC" > /tmp/dynamic.ks
|
||||
if [ -s /tmp/dynamic.ks ]; then
|
||||
echo "Install kickstart downloaded ($(wc -l < /tmp/dynamic.ks) lines)"
|
||||
else
|
||||
echo "ERROR: Failed to download install kickstart"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Run any %pre scripts from the downloaded kickstart.
|
||||
# Anaconda only runs %pre from the top-level file, not from %include'd files.
|
||||
python3 -c "
|
||||
import re, subprocess
|
||||
content = open('/tmp/dynamic.ks').read()
|
||||
blocks = re.findall(r'%pre[^\\n]*\\n(.*?)%end', content, re.DOTALL)
|
||||
for i, script in enumerate(blocks):
|
||||
path = f'/tmp/inner-pre-{i}.sh'
|
||||
with open(path, 'w') as f:
|
||||
f.write(script)
|
||||
print(f'Running inner %pre script {i} ({len(script.splitlines())} lines)')
|
||||
subprocess.run(['bash', path], check=False)
|
||||
"
|
||||
;;
|
||||
|
||||
debug)
|
||||
echo "=== Debug mode ==="
|
||||
curl -sf "${bastionUrl}/debug.ks?mac=$MAC" > /tmp/dynamic.ks 2>/dev/null
|
||||
if [ ! -s /tmp/dynamic.ks ]; then
|
||||
echo "rescue" > /tmp/dynamic.ks
|
||||
fi
|
||||
;;
|
||||
|
||||
*)
|
||||
echo "=== Running hardware discovery ==="
|
||||
# Collect hardware info
|
||||
PRODUCT=$(cat /sys/class/dmi/id/product_name 2>/dev/null || echo "unknown")
|
||||
BOARD=$(cat /sys/class/dmi/id/board_name 2>/dev/null || echo "unknown")
|
||||
SERIAL=$(cat /sys/class/dmi/id/product_serial 2>/dev/null || echo "unknown")
|
||||
MANUFACTURER=$(cat /sys/class/dmi/id/sys_vendor 2>/dev/null || echo "unknown")
|
||||
CPUMODEL=$(grep -m1 'model name' /proc/cpuinfo | cut -d: -f2 | sed 's/^ //')
|
||||
CPUCORES=$(grep -c '^processor' /proc/cpuinfo)
|
||||
MEMGB=$(awk '/MemTotal/ {printf "%d", $2/1024/1024}' /proc/meminfo)
|
||||
ARCHTYPE=$(uname -m)
|
||||
|
||||
DISKS_JSON=$(lsblk -Jb -o NAME,SIZE,TYPE,MODEL 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
disks = [d for d in data.get('blockdevices', []) if d.get('type') == 'disk']
|
||||
result = []
|
||||
for d in disks:
|
||||
size_gb = round(int(d.get('size', 0)) / 1073741824, 1)
|
||||
result.append({'name': d.get('name', '?'), 'size_gb': size_gb, 'model': (d.get('model') or 'unknown').strip()})
|
||||
print(json.dumps(result))
|
||||
" 2>/dev/null || echo '[]')
|
||||
|
||||
NICS_JSON=$(ip -j link show 2>/dev/null | python3 -c "
|
||||
import sys, json
|
||||
nics = json.load(sys.stdin)
|
||||
result = []
|
||||
for n in nics:
|
||||
if n.get('link_type') == 'loopback': continue
|
||||
result.append({'name': n.get('ifname', '?'), 'mac': n.get('address', '?'), 'state': n.get('operstate', '?')})
|
||||
print(json.dumps(result))
|
||||
" 2>/dev/null || echo '[]')
|
||||
|
||||
PAYLOAD=$(python3 -c "
|
||||
import json
|
||||
print(json.dumps({
|
||||
'mac': '$MAC', 'product': '$PRODUCT', 'board': '$BOARD', 'serial': '$SERIAL',
|
||||
'manufacturer': '$MANUFACTURER', 'cpu_model': '$CPUMODEL',
|
||||
'cpu_cores': int('$CPUCORES' or 0), 'memory_gb': int('$MEMGB' or 0),
|
||||
'arch': '$ARCHTYPE', 'disks': $DISKS_JSON, 'nics': $NICS_JSON
|
||||
}))
|
||||
")
|
||||
|
||||
curl -sf -X POST "${bastionUrl}/api/discover" \\
|
||||
-H "Content-Type: application/json" \\
|
||||
-d "$PAYLOAD" || true
|
||||
|
||||
echo ""
|
||||
echo "=== Discovery complete ==="
|
||||
echo "Machine MAC: $MAC"
|
||||
echo "Queue for install: labctl provision install $MAC <hostname> --role infra"
|
||||
echo "Then reboot to start installation."
|
||||
echo ""
|
||||
|
||||
# Write a minimal kickstart that just reboots
|
||||
cat > /tmp/dynamic.ks << 'DISCOVER_KS'
|
||||
# Discovery mode -- reboot to allow install queue
|
||||
reboot
|
||||
DISCOVER_KS
|
||||
|
||||
# Force reboot now (don't wait for Anaconda)
|
||||
sleep 3
|
||||
echo 1 > /proc/sys/kernel/sysrq
|
||||
echo b > /proc/sysrq-trigger
|
||||
sleep 5
|
||||
reboot -f
|
||||
;;
|
||||
esac
|
||||
|
||||
%end
|
||||
|
||||
# Include the dynamically chosen kickstart
|
||||
%include /tmp/dynamic.ks
|
||||
`;
|
||||
|
||||
return reply.type("text/plain").send(ks);
|
||||
});
|
||||
|
||||
// Ubuntu autoinstall user-data (cloud-init)
|
||||
app.get<{ Params: { mac: string } }>("/autoinstall/:mac/user-data", async (request, reply) => {
|
||||
const mac = request.params.mac.toLowerCase().replace(/-/g, ":");
|
||||
|
||||
71
bastion/src/bastion/src/routes/vyos.ts
Normal file
71
bastion/src/bastion/src/routes/vyos.ts
Normal file
@@ -0,0 +1,71 @@
|
||||
// VyOS network install routes.
|
||||
//
|
||||
// VyOS has no unattended installer, so the automation is injected via
|
||||
// live-config's `hooks` component: the iPXE script passes
|
||||
// live-config.hooks=<.../vyos/autoinstall.sh>, live-config wgets it and runs it
|
||||
// as root, and that script fetches and executes the generated install driver.
|
||||
|
||||
import type { FastifyInstance } from "fastify";
|
||||
import type { BastionConfig } from "@lab/shared";
|
||||
import type { StateManager } from "../services/state.js";
|
||||
import { buildVyosConfigSpec } from "../templates/vyos-config-spec.js";
|
||||
import { renderVyosInstallPy } from "../templates/vyos-install.py.js";
|
||||
import { logger } from "../services/logger.js";
|
||||
|
||||
function normalizeMac(value: string | undefined): string {
|
||||
return (value ?? "").toLowerCase().replace(/-/g, ":");
|
||||
}
|
||||
|
||||
export function registerVyosRoutes(
|
||||
app: FastifyInstance,
|
||||
config: BastionConfig,
|
||||
state: StateManager,
|
||||
): void {
|
||||
// live-config hook. Kept minimal: everything version-specific lives in the
|
||||
// generated Python. wget is guaranteed present -- live-config used it to
|
||||
// fetch this very script.
|
||||
app.get<{ Querystring: { mac?: string } }>("/vyos/autoinstall.sh", async (request, reply) => {
|
||||
const mac = normalizeMac(request.query.mac);
|
||||
const base = `http://${config.serverIp}:${config.httpPort}`;
|
||||
|
||||
logger.info(`VYOS AUTOINSTALL HOOK served to ${mac || "unknown MAC"}`);
|
||||
|
||||
const script = `#!/bin/sh
|
||||
# Lab PXE Bastion -- VyOS unattended install hook (run by live-config as root)
|
||||
set -eu
|
||||
|
||||
wget -q "${base}/vyos/install.py?mac=${mac}" -O /tmp/vyos-install.py
|
||||
exec python3 /tmp/vyos-install.py
|
||||
`;
|
||||
return reply.type("text/plain").send(script);
|
||||
});
|
||||
|
||||
// Per-MAC install driver, with the machine's config spec baked in.
|
||||
app.get<{ Querystring: { mac?: string } }>("/vyos/install.py", async (request, reply) => {
|
||||
const mac = normalizeMac(request.query.mac);
|
||||
const queueEntry = state.load().install_queue[mac];
|
||||
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: queueEntry?.hostname ?? "vyos",
|
||||
spec: queueEntry?.vyos,
|
||||
defaultPassword: config.vyosDefaultPassword,
|
||||
sshKeys: config.sshKeys,
|
||||
disk: queueEntry?.disk,
|
||||
});
|
||||
|
||||
logger.info(
|
||||
`VYOS INSTALL DRIVER served to ${mac} (${spec.hostname}, ` +
|
||||
`${spec.sets.length} config ops, disk="${spec.disk || "auto"}")`,
|
||||
);
|
||||
|
||||
const script = renderVyosInstallPy({
|
||||
spec,
|
||||
mac,
|
||||
serverIp: config.serverIp,
|
||||
httpPort: config.httpPort,
|
||||
role: queueEntry?.role ?? "vanilla",
|
||||
});
|
||||
|
||||
return reply.type("text/plain").send(script);
|
||||
});
|
||||
}
|
||||
@@ -12,6 +12,7 @@ import { registerDispatchRoutes } from "./routes/dispatch.js";
|
||||
import { registerKickstartRoutes } from "./routes/kickstart.js";
|
||||
import { registerApiRoutes } from "./routes/api.js";
|
||||
import { registerAsahiRoutes } from "./routes/asahi.js";
|
||||
import { registerVyosRoutes } from "./routes/vyos.js";
|
||||
|
||||
|
||||
export function createApp(config: BastionConfig): { app: ReturnType<typeof Fastify>; state: StateManager; installLog: InstallLogBuffer; syslog: SyslogListener } {
|
||||
@@ -47,6 +48,7 @@ export function createApp(config: BastionConfig): { app: ReturnType<typeof Fasti
|
||||
registerKickstartRoutes(app, config, state, syslog);
|
||||
registerApiRoutes(app, state, installLog, syslog);
|
||||
registerAsahiRoutes(app, config);
|
||||
registerVyosRoutes(app, config, state);
|
||||
// boot.iso is generated at startup and served as a static file from httpDir
|
||||
// (static serving supports HTTP Range requests, required by JetKVM streaming)
|
||||
|
||||
|
||||
@@ -166,6 +166,7 @@ export class BastionConnection {
|
||||
case "command-role-update":
|
||||
case "command-debug":
|
||||
case "command-register":
|
||||
case "command-discover":
|
||||
void this.handleCommand(msg);
|
||||
break;
|
||||
}
|
||||
|
||||
@@ -111,6 +111,29 @@ mount_lv() {
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Write fstab function (idempotent) ────────────────────────────
|
||||
write_lab_fstab() {
|
||||
# Remove any previous lab LVM entries (clean slate)
|
||||
sed -i '/# lab-lvm:/d' /etc/fstab
|
||||
sed -i '/# Lab LVM volumes/d' /etc/fstab
|
||||
grep -v "/dev/labvg/" /etc/fstab > /etc/fstab.tmp && mv /etc/fstab.tmp /etc/fstab
|
||||
# Comment out non-LVM entries for mount points we manage
|
||||
for mp in "/var " "/var/log " "/home " "/srv "; do
|
||||
if grep -q "$mp" /etc/fstab; then
|
||||
awk -v m="$mp" '{if($0 !~ /^#/ && index($0,m)) print "# lab-lvm: " $0; else print}' /etc/fstab > /etc/fstab.tmp
|
||||
mv /etc/fstab.tmp /etc/fstab
|
||||
fi
|
||||
done
|
||||
# Add fresh LVM entries
|
||||
echo "# Lab LVM volumes" >> /etc/fstab
|
||||
echo "/dev/labvg/swap none swap defaults 0 0" >> /etc/fstab
|
||||
echo "/dev/labvg/var /var xfs defaults 0 0" >> /etc/fstab
|
||||
echo "/dev/labvg/varlog /var/log xfs defaults 0 0" >> /etc/fstab
|
||||
echo "/dev/labvg/home /home xfs defaults 0 0" >> /etc/fstab
|
||||
echo "/dev/labvg/srv /srv xfs defaults 0 0" >> /etc/fstab
|
||||
${roleFstabLines.join('\n ')}
|
||||
}
|
||||
|
||||
# ── Check for existing VG ────────────────────────────────────────
|
||||
if vgs labvg &>/dev/null; then
|
||||
echo "Volume group 'labvg' already exists — reprovision detected."
|
||||
@@ -129,22 +152,11 @@ ${roleMountLines.map(l => ` ${l}`).join('\n')}
|
||||
echo " Enabled swap"
|
||||
fi
|
||||
|
||||
# Ensure fstab entries exist
|
||||
grep -q "labvg" /etc/fstab || {
|
||||
echo "# Lab LVM volumes (re-added after reprovision)" >> /etc/fstab
|
||||
echo "/dev/labvg/swap none swap defaults 0 0" >> /etc/fstab
|
||||
echo "/dev/labvg/var /var xfs defaults 0 0" >> /etc/fstab
|
||||
echo "/dev/labvg/varlog /var/log xfs defaults 0 0" >> /etc/fstab
|
||||
echo "/dev/labvg/home /home xfs defaults 0 0" >> /etc/fstab
|
||||
echo "/dev/labvg/srv /srv xfs defaults 0 0" >> /etc/fstab
|
||||
${roleFstabLines.map(l => ` ${l}`).join('\n')}
|
||||
}
|
||||
# Ensure fstab entries exist — comment out conflicting btrfs subvol entries
|
||||
write_lab_fstab
|
||||
|
||||
echo "Existing LVM volumes re-mounted."
|
||||
touch "$MARKER"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
else
|
||||
# ── Fresh install: create LVM ────────────────────────────────────
|
||||
echo "Creating LVM on $DATA_PART..."
|
||||
|
||||
@@ -210,34 +222,39 @@ echo "NOTE: /var and /var/log will switch to LVM on next reboot."
|
||||
# Enable swap
|
||||
swapon /dev/labvg/swap 2>/dev/null || true
|
||||
|
||||
# Write fstab entries
|
||||
echo "" >> /etc/fstab
|
||||
echo "# Lab LVM volumes" >> /etc/fstab
|
||||
echo "/dev/labvg/swap none swap defaults 0 0" >> /etc/fstab
|
||||
echo "/dev/labvg/var /var xfs defaults 0 0" >> /etc/fstab
|
||||
echo "/dev/labvg/varlog /var/log xfs defaults 0 0" >> /etc/fstab
|
||||
echo "/dev/labvg/home /home xfs defaults 0 0" >> /etc/fstab
|
||||
echo "/dev/labvg/srv /srv xfs defaults 0 0" >> /etc/fstab
|
||||
${roleFstabLines.join('\n')}
|
||||
write_lab_fstab
|
||||
|
||||
echo "LVM setup complete."
|
||||
lvs labvg
|
||||
|
||||
# ── Set hostname ─────────────────────────────────────────────────
|
||||
hostnamectl set-hostname "${hostname}"
|
||||
fi # end if/else for reprovision vs fresh install
|
||||
|
||||
# ── Set hostname (use configured value, or keep existing) ────────
|
||||
CONF_HOSTNAME="${hostname}"
|
||||
if [ "$CONF_HOSTNAME" != "unknown" ] && [ -n "$CONF_HOSTNAME" ]; then
|
||||
hostnamectl set-hostname "$CONF_HOSTNAME"
|
||||
fi
|
||||
ACTUAL_HOSTNAME=$(hostname)
|
||||
|
||||
# ── Detect MAC address ───────────────────────────────────────────
|
||||
CONF_MAC="${mac}"
|
||||
if [ "$CONF_MAC" = "unknown" ] || [ -z "$CONF_MAC" ]; then
|
||||
CONF_MAC=$(ip -o link show | grep -v "lo:" | grep "state UP" | head -1 | grep -oP 'link/ether \\K[^ ]+' || echo "unknown")
|
||||
fi
|
||||
|
||||
# ── Configure admin user ─────────────────────────────────────────
|
||||
if ! id "${adminUser}" &>/dev/null; then
|
||||
useradd -m -G wheel "${adminUser}"
|
||||
echo "${adminUser} ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/${adminUser}
|
||||
chmod 440 /etc/sudoers.d/${adminUser}
|
||||
ADMIN="${adminUser}"
|
||||
if ! id "$ADMIN" &>/dev/null; then
|
||||
useradd -m -G wheel "$ADMIN"
|
||||
echo "$ADMIN ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/$ADMIN
|
||||
chmod 440 /etc/sudoers.d/$ADMIN
|
||||
fi
|
||||
ADMIN_SSH="/home/${adminUser}/.ssh"
|
||||
ADMIN_SSH="/home/$ADMIN/.ssh"
|
||||
mkdir -p "$ADMIN_SSH"
|
||||
chmod 700 "$ADMIN_SSH"
|
||||
${sshKeyBlock}
|
||||
chmod 600 "$ADMIN_SSH/authorized_keys"
|
||||
chown -R ${adminUser}:${adminUser} "$ADMIN_SSH"
|
||||
chown -R $ADMIN:$ADMIN "$ADMIN_SSH"
|
||||
|
||||
# Also authorize root
|
||||
mkdir -p /root/.ssh
|
||||
@@ -246,14 +263,14 @@ ${rootSshKeyBlock}
|
||||
chmod 600 /root/.ssh/authorized_keys
|
||||
|
||||
# ── Harden SSH (takes effect on next sshd restart/reboot) ────────
|
||||
sed -i 's/^#\\?PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
|
||||
sed -i 's/^#\\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
|
||||
sed -i 's/^#*PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
|
||||
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
|
||||
|
||||
# ── Write provisioning metadata ──────────────────────────────────
|
||||
cat > /etc/lab-provisioned << LABMETA
|
||||
hostname=${hostname}
|
||||
hostname=$ACTUAL_HOSTNAME
|
||||
role=${role}
|
||||
mac=${mac}
|
||||
mac=$CONF_MAC
|
||||
provisioned_at=$(date -Iseconds)
|
||||
method=asahi-firstboot
|
||||
LABMETA
|
||||
@@ -263,9 +280,9 @@ IP=$(hostname -I | awk '{print $1}')
|
||||
echo "Registering with bastion at ${serverIp}:${httpPort}..."
|
||||
curl -sf -X POST "http://${serverIp}:${httpPort}/api/register" \\
|
||||
-H "Content-Type: application/json" \\
|
||||
-d "{\\"mac\\":\\"${mac}\\",\\"hostname\\":\\"${hostname}\\",\\"role\\":\\"${role}\\",\\"ip\\":\\"$IP\\"}" \\
|
||||
2>/dev/null && echo " Registered as ${hostname} ($IP)" \\
|
||||
|| echo " WARNING: Could not reach bastion — register manually with: labctl provision register ${mac} ${hostname} --role ${role} --ip $IP"
|
||||
-d "{\\"mac\\":\\"$CONF_MAC\\",\\"hostname\\":\\"$ACTUAL_HOSTNAME\\",\\"role\\":\\"${role}\\",\\"ip\\":\\"$IP\\"}" \\
|
||||
2>/dev/null && echo " Registered as $ACTUAL_HOSTNAME ($IP)" \\
|
||||
|| echo " WARNING: Could not reach bastion — register manually with: labctl provision register $CONF_MAC $ACTUAL_HOSTNAME --role ${role} --ip $IP"
|
||||
|
||||
# ── Mark done ────────────────────────────────────────────────────
|
||||
touch "$MARKER"
|
||||
|
||||
@@ -40,6 +40,11 @@ export function renderInstallKickstart(params: InstallKickstartParams): string {
|
||||
const now = new Date().toISOString();
|
||||
const hasLonghorn = role === "worker";
|
||||
const hasRancher = role === "infra";
|
||||
// k8s roles get a dedicated 120G image-store LV. 2026-08 incident: the old
|
||||
// 20G LV idled at 85% used, so a single ~5G image pull tripped imagefs
|
||||
// eviction. Must be sized here — longhorn's --grow consumes all remaining
|
||||
// VG space, making post-install lvextend impossible on worker nodes.
|
||||
const hasRancherLv = role === "infra" || role === "worker";
|
||||
const isVanilla = role === "vanilla";
|
||||
|
||||
// -- Auth section --
|
||||
@@ -113,9 +118,9 @@ done
|
||||
? `logvol /var/lib/longhorn --vgname=${vg} --name=longhorn --fstype=xfs --grow --size=1`
|
||||
: "";
|
||||
|
||||
// -- Rancher LV for fresh install (infra role) --
|
||||
const rancherFreshLine = hasRancher
|
||||
? `logvol /var/lib/rancher --vgname=${vg} --name=rancher --fstype=xfs --size=20480`
|
||||
// -- Rancher LV for fresh install (k8s roles: worker + infra) --
|
||||
const rancherFreshLine = hasRancherLv
|
||||
? `logvol /var/lib/rancher --vgname=${vg} --name=rancher --fstype=xfs --size=122880`
|
||||
: "";
|
||||
|
||||
return `# Lab Bastion -- Fedora ${fedoraVersion} server install
|
||||
|
||||
53
bastion/src/bastion/src/templates/vyos-boot.ipxe.ts
Normal file
53
bastion/src/bastion/src/templates/vyos-boot.ipxe.ts
Normal file
@@ -0,0 +1,53 @@
|
||||
// iPXE boot script template for VyOS network install.
|
||||
//
|
||||
// VyOS ships no unattended installer: `install image` is unconditionally
|
||||
// interactive (image_installer.py's install action takes no arguments, and
|
||||
// --no-prompt is wired only to `add`). So PXE boots the *live* system and the
|
||||
// automation is injected through live-config's `hooks` component, which fetches
|
||||
// a script over HTTP and runs it as root late in live boot.
|
||||
//
|
||||
// Unlike the Fedora/Ubuntu paths this boots a live image rather than an
|
||||
// installer, so there is no kickstart/autoinstall equivalent — see
|
||||
// routes/vyos.ts for the hook that actually drives the install.
|
||||
|
||||
export function renderVyosInstallIpxe(params: {
|
||||
mac: string;
|
||||
hostname: string;
|
||||
serverIp: string;
|
||||
httpPort: number;
|
||||
}): string {
|
||||
const base = `http://${params.serverIp}:${params.httpPort}`;
|
||||
|
||||
// Pin the boot NIC by MAC. live-boot otherwise scans for the first
|
||||
// *connected* interface, and on a multi-NIC box that race is lost by
|
||||
// whichever port negotiates slowest: on the Protectli VP2440 the SFP+
|
||||
// pair links first, so live-boot picked the fiber ports (which have no
|
||||
// DHCP), burned 15s per port, and gave up with "Unable to find a live
|
||||
// file system on the network" -- while the copper port that actually PXE
|
||||
// booted came up at 4.6s and was never tried.
|
||||
//
|
||||
// live-boot's Device_from_bootif() strips the "01-" and matches the MAC
|
||||
// against /sys/class/net/*. params.mac is the dispatch key, i.e. exactly
|
||||
// the NIC that PXE booted -- more reliable than iPXE's ${net0} on a box
|
||||
// where the booting NIC may not be net0.
|
||||
const bootif = `01-${params.mac.toLowerCase().replace(/:/g, "-")}`;
|
||||
|
||||
// Deliberately NOT passing `nonetworking` (present in VyOS's own PXE docs):
|
||||
// live-config's hook component needs networking up to fetch the hook over
|
||||
// HTTP. Also no `console=ttyS0` — on hardware without a physical UART that
|
||||
// costs 30s at every systemd boot phase.
|
||||
return `#!ipxe
|
||||
|
||||
echo
|
||||
echo =============================================
|
||||
echo Lab PXE Bastion - INSTALLING VyOS
|
||||
echo Target: ${params.hostname}
|
||||
echo MAC: ${params.mac}
|
||||
echo =============================================
|
||||
echo
|
||||
|
||||
kernel ${base}/vyos-vmlinuz boot=live nopersistence noautologin BOOTIF=${bootif} fetch=${base}/vyos-filesystem.squashfs live-config.hooks=${base}/vyos/autoinstall.sh?mac=${params.mac}
|
||||
initrd ${base}/vyos-initrd
|
||||
boot
|
||||
`;
|
||||
}
|
||||
352
bastion/src/bastion/src/templates/vyos-config-spec.ts
Normal file
352
bastion/src/bastion/src/templates/vyos-config-spec.ts
Normal file
@@ -0,0 +1,352 @@
|
||||
// Builds the VyOS configuration spec applied by the autoinstall hook.
|
||||
//
|
||||
// We deliberately do NOT emit a config.boot file as text. A config.boot carries a
|
||||
// `vyos-config-version` trailer; without a trailer matching the running image,
|
||||
// VyOS runs its migration scripts from version 0 on first boot. Instead the hook
|
||||
// loads the image's own /opt/vyatta/etc/config.boot.default through vyos.configtree
|
||||
// and applies these set operations on top, so syntax and version trailer always
|
||||
// match the exact image being installed.
|
||||
|
||||
import type { VyosInstallSpec } from "@lab/shared";
|
||||
|
||||
export interface VyosSetOp {
|
||||
path: string[];
|
||||
value?: string;
|
||||
/** false appends to a multi-value node (e.g. bond members) instead of replacing. */
|
||||
replace?: boolean;
|
||||
}
|
||||
|
||||
export interface VyosConfigSpec {
|
||||
hostname: string;
|
||||
/** "" means accept the installer default (the running image's version string). */
|
||||
imageName: string;
|
||||
password: string;
|
||||
console: "K" | "S";
|
||||
/** Target disk name (e.g. "nvme0n1"); "" accepts the installer's first-disk default. */
|
||||
disk: string;
|
||||
/**
|
||||
* IP the driver should report in the "complete" callback ("ready at <ip>" —
|
||||
* the exact format routes/api.ts parses installed.ip from). The mgmt
|
||||
* address when static; "" means detect the live DHCP address at runtime.
|
||||
*/
|
||||
reportAddress: string;
|
||||
/** Whether to accept RAID-1 when the installer finds more than one disk. */
|
||||
raid: boolean;
|
||||
/** Overwrite the installed config.boot with the generated one on reinstall. */
|
||||
freshConfig: boolean;
|
||||
sets: VyosSetOp[];
|
||||
/** Paths that are VyOS tag nodes — must be marked as such in the ConfigTree. */
|
||||
tags: string[][];
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalise a target disk to the form the installer expects.
|
||||
*
|
||||
* find_disks() enumerates via `lsblk -Jbp` (-p = full paths), so its valid
|
||||
* responses are "/dev/mmcblk0"-style. A bare "mmcblk0" is rejected by
|
||||
* ask_input()'s valid_responses check and re-prompts forever.
|
||||
*/
|
||||
function normalizeDiskPath(value: string | undefined): string {
|
||||
const raw = (value ?? "").trim();
|
||||
if (raw === "") return "";
|
||||
return raw.startsWith("/dev/") ? raw : `/dev/${raw}`;
|
||||
}
|
||||
|
||||
/** Sentinel marking a value that lives in Pulumi config, not in the bundle. */
|
||||
const SECRET_PREFIX = "@secret:";
|
||||
|
||||
/**
|
||||
* Enable the VyOS HTTP API so the router is manageable the moment it boots.
|
||||
*
|
||||
* This belongs at install time rather than in the Pulumi model: the model is
|
||||
* applied THROUGH this API, so a router that lacks it cannot be brought under
|
||||
* management without a hand-run change on a live firewall. It is also why the
|
||||
* model excludes `service https` outright -- a provider able to rewrite its own
|
||||
* transport can lock itself out permanently.
|
||||
*
|
||||
* `listen-address` is always set. Leaving it unbound would expose a
|
||||
* config-write endpoint on every segment the router touches, the WAN included.
|
||||
*/
|
||||
function apiSets(apiKey: string, listenAddress: string): VyosSetOp[] {
|
||||
const sets: VyosSetOp[] = [
|
||||
{ path: ["service", "https", "api", "keys", "id", "pulumi", "key"], value: apiKey },
|
||||
{ path: ["service", "https", "api", "rest"] },
|
||||
];
|
||||
if (listenAddress !== "") {
|
||||
sets.push({ path: ["service", "https", "listen-address"], value: listenAddress });
|
||||
}
|
||||
return sets;
|
||||
}
|
||||
|
||||
/** Tag nodes introduced by the API config, needed by the installer's ConfigTree. */
|
||||
const API_TAGS: string[][] = [["service", "https", "api", "keys", "id"]];
|
||||
|
||||
/**
|
||||
* The address to bind the API to: an explicit choice, else the management
|
||||
* address with its prefix length stripped. Under DHCP there is no address to
|
||||
* bind at build time, so the caller must pass one or the listener stays unbound
|
||||
* and the API is not enabled at all.
|
||||
*/
|
||||
function apiListenAddress(spec: VyosInstallSpec, mgmtAddress: string): string {
|
||||
if (spec.apiListenAddress !== undefined && spec.apiListenAddress !== "") {
|
||||
return spec.apiListenAddress;
|
||||
}
|
||||
return mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "";
|
||||
}
|
||||
|
||||
/**
|
||||
* Use a Pulumi-rendered bundle as the router's config verbatim.
|
||||
*
|
||||
* Secret-valued nodes are dropped rather than installed with their sentinel
|
||||
* text: writing `@secret:pppoePassword` into config.boot would look configured
|
||||
* while being wrong, which is worse than being absent. The router comes up
|
||||
* without those values and the first `pulumi up` fills them in.
|
||||
*
|
||||
* `system host-name` is forced to the hostname the install was asked for. The
|
||||
* bundle carries the name of whichever router it was exported from, and
|
||||
* installing vyos001's hostname onto vyos002 would collide on the network.
|
||||
*/
|
||||
function buildFromBundle(
|
||||
params: { hostname: string; defaultPassword: string; disk?: string | undefined },
|
||||
spec: VyosInstallSpec,
|
||||
bundle: NonNullable<VyosInstallSpec["bundle"]>,
|
||||
mgmtAddress: string,
|
||||
): VyosConfigSpec {
|
||||
const sets: VyosSetOp[] = [];
|
||||
const dropped: string[] = [];
|
||||
for (const op of bundle.sets) {
|
||||
if (op.value !== undefined && op.value.startsWith(SECRET_PREFIX)) {
|
||||
dropped.push(op.path.join(" "));
|
||||
continue;
|
||||
}
|
||||
if (op.path.length === 2 && op.path[0] === "system" && op.path[1] === "host-name") {
|
||||
continue;
|
||||
}
|
||||
sets.push({
|
||||
path: op.path,
|
||||
...(op.value === undefined ? {} : { value: op.value }),
|
||||
...(op.replace === undefined ? {} : { replace: op.replace }),
|
||||
});
|
||||
}
|
||||
sets.unshift({ path: ["system", "host-name"], value: params.hostname });
|
||||
|
||||
if (dropped.length > 0) {
|
||||
console.warn(
|
||||
`vyos ${params.hostname}: ${dropped.length} secret-valued node(s) left unset by the ` +
|
||||
`bundle; run \`pulumi up\` to supply them: ${dropped.join(", ")}`,
|
||||
);
|
||||
}
|
||||
|
||||
const tags = [...bundle.tags];
|
||||
const api = enableApi(spec, params.hostname, mgmtAddress);
|
||||
if (api.length > 0) {
|
||||
sets.push(...api);
|
||||
tags.push(...API_TAGS);
|
||||
}
|
||||
|
||||
return {
|
||||
hostname: params.hostname,
|
||||
imageName: "",
|
||||
password: spec.password ?? params.defaultPassword,
|
||||
console: "K",
|
||||
disk: normalizeDiskPath(params.disk),
|
||||
reportAddress: mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "",
|
||||
raid: false,
|
||||
freshConfig: spec.freshConfig ?? false,
|
||||
sets,
|
||||
tags,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The API config for this install, or nothing when it cannot be enabled safely.
|
||||
*
|
||||
* Refusing to enable it unbound is deliberate. Under DHCP there is no address
|
||||
* known at build time, and the alternative -- binding to every interface --
|
||||
* would publish a config-write endpoint on the WAN. Better to leave the router
|
||||
* SSH-only and say so than to open it everywhere.
|
||||
*/
|
||||
function enableApi(spec: VyosInstallSpec, hostname: string, mgmtAddress: string): VyosSetOp[] {
|
||||
if (spec.apiKey === undefined || spec.apiKey === "") return [];
|
||||
const listen = apiListenAddress(spec, mgmtAddress);
|
||||
if (listen === "") {
|
||||
console.warn(
|
||||
`vyos ${hostname}: --vyos-api-key given but no address to bind to ` +
|
||||
`(management is "${mgmtAddress}"). Pass --vyos-api-listen <addr>; the HTTP API ` +
|
||||
`has NOT been enabled, so Pulumi cannot manage this router yet.`,
|
||||
);
|
||||
return [];
|
||||
}
|
||||
return apiSets(spec.apiKey, listen);
|
||||
}
|
||||
|
||||
export function buildVyosConfigSpec(params: {
|
||||
hostname: string;
|
||||
spec?: VyosInstallSpec | undefined;
|
||||
defaultPassword: string;
|
||||
sshKeys?: string[] | undefined;
|
||||
disk?: string | undefined;
|
||||
}): VyosConfigSpec {
|
||||
const spec = params.spec ?? {};
|
||||
const mgmt = spec.mgmtInterface ?? "eth0";
|
||||
const mgmtAddress = spec.mgmtAddress ?? "dhcp";
|
||||
|
||||
// A rendered bundle replaces the derived config entirely. Deriving a second
|
||||
// opinion alongside it is the drift the bundle exists to prevent: Pulumi and
|
||||
// labctl would each believe they knew the router's config, and the box would
|
||||
// end up with whichever ran last.
|
||||
if (spec.bundle !== undefined) {
|
||||
return buildFromBundle(params, spec, spec.bundle, mgmtAddress);
|
||||
}
|
||||
const bondMembers = spec.bondMembers ?? [];
|
||||
const vlans = spec.vlans ?? [];
|
||||
|
||||
const sets: VyosSetOp[] = [];
|
||||
const tags: string[][] = [
|
||||
["interfaces", "ethernet"],
|
||||
["system", "login", "user"],
|
||||
];
|
||||
|
||||
const hwIds = spec.hwIds ?? {};
|
||||
const pinHwId = (iface: string): void => {
|
||||
const mac = hwIds[iface];
|
||||
if (mac !== undefined && mac !== "") {
|
||||
sets.push({ path: ["interfaces", "ethernet", iface, "hw-id"], value: mac });
|
||||
}
|
||||
};
|
||||
|
||||
sets.push({ path: ["system", "host-name"], value: params.hostname });
|
||||
|
||||
// Management interface — the NIC that PXE booted, left untagged and unbonded.
|
||||
sets.push({ path: ["interfaces", "ethernet", mgmt, "address"], value: mgmtAddress });
|
||||
pinHwId(mgmt);
|
||||
|
||||
// Tagged management VLAN on the PXE port. Emitted regardless of bonding, so
|
||||
// the box stays reachable on the management VLAN while still booting untagged
|
||||
// on whichever VLAN the bastion's proxy DHCP serves.
|
||||
const mgmtVlan = spec.mgmtVlan;
|
||||
if (mgmtVlan !== undefined) {
|
||||
tags.push(["interfaces", "ethernet", mgmt, "vif"]);
|
||||
const vif = ["interfaces", "ethernet", mgmt, "vif", String(mgmtVlan.id)];
|
||||
sets.push({ path: [...vif, "address"], value: mgmtVlan.address });
|
||||
if (mgmtVlan.description !== undefined && mgmtVlan.description !== "") {
|
||||
sets.push({ path: [...vif, "description"], value: mgmtVlan.description });
|
||||
}
|
||||
}
|
||||
|
||||
// LACP bond. Members must exclude the PXE NIC; firmware PXE cannot run over LACP.
|
||||
const bonded = bondMembers.length > 0;
|
||||
if (bonded) {
|
||||
tags.push(["interfaces", "bonding"]);
|
||||
sets.push({ path: ["interfaces", "bonding", "bond0", "mode"], value: "802.3ad" });
|
||||
sets.push({ path: ["interfaces", "bonding", "bond0", "hash-policy"], value: "layer2+3" });
|
||||
for (const member of bondMembers) {
|
||||
sets.push({
|
||||
path: ["interfaces", "bonding", "bond0", "member", "interface"],
|
||||
value: member,
|
||||
replace: false,
|
||||
});
|
||||
pinHwId(member);
|
||||
}
|
||||
// Address on the trunk's native/untagged VLAN.
|
||||
if (spec.bondAddress !== undefined && spec.bondAddress !== "") {
|
||||
sets.push({ path: ["interfaces", "bonding", "bond0", "address"], value: spec.bondAddress });
|
||||
}
|
||||
}
|
||||
|
||||
// VRRP groups accumulate here; emitted (plus a sync group) after the VLANs.
|
||||
// interface accepts dotted vifs (constraint regex `[0-9]+(.\d+)?`), address
|
||||
// is a tag node (the VIP is the tag value itself), vrid range is 1-255.
|
||||
const vrrpGroups: Array<{ name: string; iface: string; vrid: number; vip: string }> = [];
|
||||
if (bonded && spec.bondVrrp !== undefined && spec.bondVrrp !== "") {
|
||||
// vrid 1 for the untagged group: the native VLAN is never a vif, so this
|
||||
// cannot collide with a vlan-id-derived vrid.
|
||||
vrrpGroups.push({ name: "native", iface: "bond0", vrid: 1, vip: spec.bondVrrp });
|
||||
}
|
||||
|
||||
// Tagged VLAN sub-interfaces hang off the bond when there is one, else off mgmt.
|
||||
const parent = bonded
|
||||
? ["interfaces", "bonding", "bond0"]
|
||||
: ["interfaces", "ethernet", mgmt];
|
||||
if (vlans.length > 0) {
|
||||
tags.push([...parent, "vif"]);
|
||||
const parentName = bonded ? "bond0" : mgmt;
|
||||
for (const vlan of vlans) {
|
||||
const vif = [...parent, "vif", String(vlan.id)];
|
||||
sets.push({ path: [...vif, "address"], value: vlan.address });
|
||||
if (vlan.description !== undefined && vlan.description !== "") {
|
||||
sets.push({ path: [...vif, "description"], value: vlan.description });
|
||||
}
|
||||
if (vlan.vrrp !== undefined && vlan.vrrp !== "") {
|
||||
vrrpGroups.push({
|
||||
name: `vlan${vlan.id}`,
|
||||
iface: `${parentName}.${vlan.id}`,
|
||||
vrid: vlan.id,
|
||||
vip: vlan.vrrp,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Emit VRRP groups plus one sync group so all VLANs fail over together —
|
||||
// without it a single-link event could split mastership across the pair.
|
||||
if (vrrpGroups.length > 0) {
|
||||
tags.push(["high-availability", "vrrp", "group"]);
|
||||
tags.push(["high-availability", "vrrp", "sync-group"]);
|
||||
const priority = String(spec.vrrpPriority ?? 100);
|
||||
for (const g of vrrpGroups) {
|
||||
const base = ["high-availability", "vrrp", "group", g.name];
|
||||
sets.push({ path: [...base, "interface"], value: g.iface });
|
||||
sets.push({ path: [...base, "vrid"], value: String(g.vrid) });
|
||||
sets.push({ path: [...base, "priority"], value: priority });
|
||||
// address is a tag node: the VIP is the path's final segment, no value.
|
||||
sets.push({ path: [...base, "address", g.vip] });
|
||||
tags.push([...base, "address"]);
|
||||
sets.push({
|
||||
path: ["high-availability", "vrrp", "sync-group", "MAIN", "member"],
|
||||
value: g.name,
|
||||
replace: false,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
sets.push({ path: ["service", "ssh", "port"], value: "22" });
|
||||
|
||||
const sshKeys = params.sshKeys ?? [];
|
||||
if (sshKeys.length > 0) {
|
||||
tags.push(["system", "login", "user", "vyos", "authentication", "public-keys"]);
|
||||
sshKeys.forEach((entry, index) => {
|
||||
const parts = entry.trim().split(/\s+/);
|
||||
const type = parts[0] ?? "";
|
||||
const key = parts[1] ?? "";
|
||||
if (!type.startsWith("ssh-") && !type.startsWith("ecdsa-")) return;
|
||||
if (!key) return;
|
||||
const name = parts[2] ?? `lab-key-${index}`;
|
||||
const base = ["system", "login", "user", "vyos", "authentication", "public-keys", name];
|
||||
sets.push({ path: [...base, "type"], value: type });
|
||||
sets.push({ path: [...base, "key"], value: key });
|
||||
});
|
||||
}
|
||||
|
||||
// Enabled here too, not just for bundle installs: every VyOS this bastion
|
||||
// provisions should be manageable from first boot.
|
||||
const api = enableApi(spec, params.hostname, mgmtAddress);
|
||||
if (api.length > 0) {
|
||||
sets.push(...api);
|
||||
tags.push(...API_TAGS);
|
||||
}
|
||||
|
||||
return {
|
||||
hostname: params.hostname,
|
||||
imageName: "",
|
||||
password: spec.password ?? params.defaultPassword,
|
||||
console: "K",
|
||||
disk: normalizeDiskPath(params.disk),
|
||||
// Static mgmt address wins; under DHCP the driver detects the live IP.
|
||||
reportAddress: mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "",
|
||||
raid: false,
|
||||
freshConfig: spec.freshConfig ?? false,
|
||||
sets,
|
||||
tags,
|
||||
};
|
||||
}
|
||||
514
bastion/src/bastion/src/templates/vyos-install.py.ts
Normal file
514
bastion/src/bastion/src/templates/vyos-install.py.ts
Normal file
@@ -0,0 +1,514 @@
|
||||
// Renders the Python program that performs the unattended VyOS install.
|
||||
//
|
||||
// It runs as root inside the live system, fetched and executed by live-config's
|
||||
// `hooks` component (see vyos-boot.ipxe.ts). It does three things:
|
||||
// 1. builds config.boot from the image's own default via vyos.configtree
|
||||
// 2. drives the interactive `install image` through a pty
|
||||
// 3. reports progress back to the bastion, then reboots
|
||||
//
|
||||
// A pty is used rather than piping stdin because the installer reads the
|
||||
// password through getpass(), which opens /dev/tty directly and would ignore a
|
||||
// pipe. Prompts are matched by text rather than replayed positionally: the
|
||||
// installer skips the boot-config question when it finds a previous
|
||||
// installation, so a fixed answer sequence desyncs on reinstall.
|
||||
|
||||
import type { VyosConfigSpec } from "./vyos-config-spec.js";
|
||||
|
||||
export function renderVyosInstallPy(params: {
|
||||
spec: VyosConfigSpec;
|
||||
mac: string;
|
||||
serverIp: string;
|
||||
httpPort: number;
|
||||
role: string;
|
||||
}): string {
|
||||
// Base64 so arbitrary values (passwords, descriptions, SSH keys) can never
|
||||
// terminate the Python string literal that carries them.
|
||||
const specB64 = Buffer.from(JSON.stringify(params.spec), "utf-8").toString("base64");
|
||||
|
||||
return `#!/usr/bin/env python3
|
||||
"""Unattended VyOS install driver -- generated by the lab PXE bastion."""
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import pty
|
||||
import re
|
||||
import select
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
SPEC = json.loads(base64.b64decode("${specB64}").decode("utf-8"))
|
||||
BASTION = "http://${params.serverIp}:${params.httpPort}"
|
||||
MAC = "${params.mac}"
|
||||
ROLE = ${JSON.stringify(params.role ?? "vanilla")}
|
||||
|
||||
INSTALLER = "/usr/libexec/vyos/op_mode/image_installer.py"
|
||||
CONFIG_DIR = "/opt/vyatta/etc/config"
|
||||
# The installer copies the rootfs from the boot MEDIUM path -- which only a
|
||||
# CD/USB boot provides. With fetch= (HTTP netboot) nothing is mounted there
|
||||
# (verified in VM: Errno 2), so the squashfs must be linked or re-fetched into
|
||||
# place before 'install image' runs.
|
||||
ROOTFS_EXPECTED = "/usr/lib/live/mount/medium/live/filesystem.squashfs"
|
||||
SQUASHFS_URL = "http://${params.serverIp}:${params.httpPort}/vyos-filesystem.squashfs"
|
||||
# The live-config hook runs BEFORE vyos-router creates the /opt/vyatta compat
|
||||
# path, so the squashfs's own location must be tried too (verified in VM: only
|
||||
# /usr/share/vyos/config.boot.default exists at hook time).
|
||||
DEFAULT_CONFIG_CANDIDATES = [
|
||||
"/opt/vyatta/etc/config.boot.default",
|
||||
"/usr/share/vyos/config.boot.default",
|
||||
]
|
||||
STALL_TIMEOUT = 900 # seconds without installer output before giving up
|
||||
|
||||
|
||||
def detect_ip():
|
||||
"""Best-effort local IP as seen on the route toward the bastion.
|
||||
|
||||
Matches Fedora's semantics (IP captured during install): under DHCP the
|
||||
installed system will renew on the same NIC/subnet the live env used.
|
||||
"""
|
||||
import socket
|
||||
try:
|
||||
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||
s.connect(("${params.serverIp}", ${params.httpPort}))
|
||||
ip = s.getsockname()[0]
|
||||
s.close()
|
||||
return ip
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
class LogStreamer:
|
||||
"""Stream install output to the bastion's /api/log so 'labctl provision
|
||||
logs -f' works live for VyOS, like Anaconda's syslog does for Fedora.
|
||||
|
||||
Strictly best-effort: a failed POST drops the batch and must never stall
|
||||
the pty read loop or fail the install.
|
||||
"""
|
||||
|
||||
ANSI = re.compile(rb"\\x1b\\[[0-9;?]*[a-zA-Z]|\\x1b[=>]|\\r")
|
||||
|
||||
def __init__(self):
|
||||
self.partial = b""
|
||||
self.pending = []
|
||||
self.last_flush = time.time()
|
||||
|
||||
def feed(self, chunk):
|
||||
"""Raw pty bytes: split into lines, strip ANSI noise, queue."""
|
||||
self.partial += chunk
|
||||
while b"\\n" in self.partial:
|
||||
raw, self.partial = self.partial.split(b"\\n", 1)
|
||||
text = self.ANSI.sub(b"", raw).decode("utf-8", "replace").rstrip()
|
||||
if text:
|
||||
self.pending.append(text)
|
||||
self.maybe_flush()
|
||||
|
||||
def line(self, text):
|
||||
"""A driver-originated message (already a clean string)."""
|
||||
self.pending.append(text)
|
||||
self.maybe_flush()
|
||||
|
||||
def maybe_flush(self):
|
||||
if len(self.pending) >= 20 or (self.pending and time.time() - self.last_flush >= 2):
|
||||
self.flush()
|
||||
|
||||
def flush(self):
|
||||
if not self.pending:
|
||||
return
|
||||
batch, self.pending = self.pending[:200], self.pending[200:]
|
||||
self.last_flush = time.time()
|
||||
try:
|
||||
body = json.dumps({"mac": MAC, "lines": batch}).encode()
|
||||
req = urllib.request.Request(
|
||||
BASTION + "/api/log",
|
||||
data=body,
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
urllib.request.urlopen(req, timeout=5).read()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
STREAM = LogStreamer()
|
||||
|
||||
|
||||
def say(msg):
|
||||
"""Print locally and stream to the bastion log buffer."""
|
||||
print(msg)
|
||||
STREAM.line(str(msg))
|
||||
|
||||
|
||||
def report(stage, detail=""):
|
||||
"""Best-effort progress callback; never fatal."""
|
||||
STREAM.flush()
|
||||
try:
|
||||
body = json.dumps({"mac": MAC, "stage": stage, "detail": detail}).encode()
|
||||
req = urllib.request.Request(
|
||||
BASTION + "/api/progress",
|
||||
data=body,
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
urllib.request.urlopen(req, timeout=5).read()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def build_config():
|
||||
"""Apply our set operations onto the image's own default config.
|
||||
|
||||
Using config.boot.default as the base keeps the vyos-config-version trailer
|
||||
consistent with the running image, so first boot does not run migrations.
|
||||
"""
|
||||
from vyos.configtree import ConfigTree
|
||||
|
||||
default_config = next(
|
||||
(p for p in DEFAULT_CONFIG_CANDIDATES if os.path.exists(p)), None)
|
||||
if default_config is None:
|
||||
raise FileNotFoundError(
|
||||
"no config.boot.default found (tried %s)" % ", ".join(DEFAULT_CONFIG_CANDIDATES))
|
||||
say("base config: %s" % default_config)
|
||||
|
||||
with open(default_config) as handle:
|
||||
config = ConfigTree(handle.read())
|
||||
|
||||
for op in SPEC["sets"]:
|
||||
replace = op.get("replace", True)
|
||||
if "value" in op and op["value"] is not None:
|
||||
config.set(op["path"], value=op["value"], replace=replace)
|
||||
else:
|
||||
config.set(op["path"])
|
||||
|
||||
# Tag nodes must be marked after the nodes exist, as the installer itself does.
|
||||
for tag in SPEC["tags"]:
|
||||
try:
|
||||
config.set_tag(tag)
|
||||
except Exception as err:
|
||||
say("warning: set_tag %s failed: %s" % (tag, err))
|
||||
|
||||
os.makedirs(CONFIG_DIR, exist_ok=True)
|
||||
target = os.path.join(CONFIG_DIR, "config.boot")
|
||||
|
||||
# Re-attach the vyos-config-version footer: ConfigTree.to_string() emits
|
||||
# only the config body, and a config without the footer is treated as
|
||||
# ancient -- the boot migrator then runs every migration over it and (as
|
||||
# observed in the VM test) crashes in system/31-to-32. Building the footer
|
||||
# from the running system pins it to the exact image being installed.
|
||||
body = config.to_string()
|
||||
try:
|
||||
from vyos.component_version import version_info_from_system
|
||||
info = version_info_from_system()
|
||||
info.update_config_body(body)
|
||||
info.write(target)
|
||||
say("wrote %s (footer: %s)" % (target, info.release))
|
||||
except Exception as err:
|
||||
say("warning: version footer failed (%s); writing bare config" % err)
|
||||
with open(target, "w") as handle:
|
||||
handle.write(body)
|
||||
return target
|
||||
|
||||
|
||||
def find_live_squashfs():
|
||||
"""Locate the squashfs live-boot fetched, without walking into the mounted
|
||||
rootfs or overlay (each would mean traversing the entire OS tree)."""
|
||||
explicit = [
|
||||
"/run/live/medium/live/filesystem.squashfs",
|
||||
"/lib/live/mount/medium/live/filesystem.squashfs",
|
||||
]
|
||||
for path in explicit:
|
||||
if os.path.isfile(path) and os.path.getsize(path) > 0:
|
||||
return path
|
||||
for root in ("/run/live", "/lib/live/mount", "/usr/lib/live/mount"):
|
||||
for dirpath, dirs, files in os.walk(root):
|
||||
depth = dirpath.count(os.sep) - root.count(os.sep)
|
||||
dirs[:] = [d for d in dirs
|
||||
if d not in ("rootfs", "overlay")
|
||||
and not d.endswith(".squashfs")
|
||||
and depth < 3]
|
||||
if "filesystem.squashfs" in files:
|
||||
path = os.path.join(dirpath, "filesystem.squashfs")
|
||||
if os.path.isfile(path) and os.path.getsize(path) > 0:
|
||||
return path
|
||||
return None
|
||||
|
||||
|
||||
def ensure_rootfs():
|
||||
"""Make FILE_ROOTFS_SRC exist so the installer can copy the system image."""
|
||||
if os.path.isfile(ROOTFS_EXPECTED) and os.path.getsize(ROOTFS_EXPECTED) > 0:
|
||||
return
|
||||
src = find_live_squashfs()
|
||||
if src is None:
|
||||
say("squashfs not in live mounts; re-fetching %s" % SQUASHFS_URL)
|
||||
src = "/tmp/filesystem.squashfs"
|
||||
urllib.request.urlretrieve(SQUASHFS_URL, src)
|
||||
os.makedirs(os.path.dirname(ROOTFS_EXPECTED), exist_ok=True)
|
||||
if os.path.lexists(ROOTFS_EXPECTED):
|
||||
os.remove(ROOTFS_EXPECTED)
|
||||
os.symlink(src, ROOTFS_EXPECTED)
|
||||
say("rootfs source: %s -> %s" % (ROOTFS_EXPECTED, src))
|
||||
|
||||
|
||||
def build_rules():
|
||||
"""Prompt -> response table for the interactive installer."""
|
||||
password = SPEC["password"].encode() + b"\\n"
|
||||
image_name = SPEC["imageName"].encode() + b"\\n"
|
||||
disk = SPEC["disk"].encode() + b"\\n"
|
||||
console = SPEC["console"].encode() + b"\\n"
|
||||
raid = (b"yes\\n" if SPEC["raid"] else b"no\\n")
|
||||
|
||||
return [
|
||||
(re.compile(rb"Would you like to continue\\?"), b"yes\\n"),
|
||||
(re.compile(rb"What would you like to name this image\\?"), image_name),
|
||||
(re.compile(rb"Please confirm password for the .vyos. user:"), password),
|
||||
(re.compile(rb"Please enter a password for the .vyos. user:"), password),
|
||||
(re.compile(rb"What console should be used by default"), console),
|
||||
# Three RAID variants: "configure RAID-1 mirroring?", "...on them?",
|
||||
# and "choose two disks for RAID-1 mirroring?" -- all default to YES,
|
||||
# so a missed one both hangs the install and risks an unwanted mirror.
|
||||
(re.compile(rb"Would you like to [^?]*RAID-1 mirroring"), raid),
|
||||
(re.compile(rb"Installation will delete all data on (?:the drive|both drives)\\. Continue\\?"), b"yes\\n"),
|
||||
(re.compile(rb"Which one should be used for installation\\?"), disk),
|
||||
(re.compile(rb"Would you like to use all the free space on the drive\\?"), b"yes\\n"),
|
||||
(re.compile(rb"Which file would you like as boot config\\?"), b"1\\n"),
|
||||
# Reinstall path only (search_previous_installation): carrying the old
|
||||
# /config and SSH host keys forward is VyOS's "reinstall without losing
|
||||
# data". Always yes -- freshConfig replaces config.boot afterwards, so
|
||||
# answering no here would also discard non-config data under /config.
|
||||
(re.compile(rb"Would you like to copy data to the new image\\?"), b"yes\\n"),
|
||||
(re.compile(rb"Would you like to copy the encrypted config to the new image\\?"), b"yes\\n"),
|
||||
# More than one previous image found -- take the first offered.
|
||||
(re.compile(rb"From which image would you like to save config information\\?"), b"1\\n"),
|
||||
(re.compile(rb"From which image would you like to copy the encrypted config\\?"), b"1\\n"),
|
||||
]
|
||||
|
||||
|
||||
def run_installer():
|
||||
"""Drive image_installer.py over a pty, answering prompts as they appear."""
|
||||
rules = build_rules()
|
||||
master, slave = pty.openpty()
|
||||
|
||||
proc = subprocess.Popen(
|
||||
[INSTALLER, "--action", "install"],
|
||||
stdin=slave,
|
||||
stdout=slave,
|
||||
stderr=slave,
|
||||
close_fds=True,
|
||||
preexec_fn=os.setsid,
|
||||
)
|
||||
os.close(slave)
|
||||
|
||||
buf = b""
|
||||
transcript = b"" # rolling tail of everything the installer printed
|
||||
last_output = time.time()
|
||||
|
||||
while True:
|
||||
ready, _, _ = select.select([master], [], [], 1.0)
|
||||
|
||||
if ready:
|
||||
try:
|
||||
chunk = os.read(master, 4096)
|
||||
except OSError:
|
||||
break
|
||||
if not chunk:
|
||||
break
|
||||
|
||||
sys.stdout.buffer.write(chunk)
|
||||
sys.stdout.buffer.flush()
|
||||
buf += chunk
|
||||
transcript = (transcript + chunk)[-8000:]
|
||||
STREAM.feed(chunk)
|
||||
last_output = time.time()
|
||||
|
||||
# Answer every prompt currently in the buffer, earliest first, so
|
||||
# ordering is preserved even when the installer skips questions --
|
||||
# and so a single chunk carrying two prompts gets both answers.
|
||||
while True:
|
||||
best = None
|
||||
for pattern, response in rules:
|
||||
found = pattern.search(buf)
|
||||
if found and (best is None or found.start() < best[0].start()):
|
||||
best = (found, response)
|
||||
if best is None:
|
||||
break
|
||||
found, response = best
|
||||
os.write(master, response)
|
||||
transcript = (transcript + b"\\n>>> answered: " + response)[-8000:]
|
||||
STREAM.line(">>> answered: " + response.decode("utf-8", "replace").strip())
|
||||
buf = buf[found.end():]
|
||||
|
||||
# Bound memory if the installer emits a lot without prompting.
|
||||
if len(buf) > 65536:
|
||||
buf = buf[-8192:]
|
||||
|
||||
elif proc.poll() is not None:
|
||||
break
|
||||
|
||||
STREAM.maybe_flush()
|
||||
|
||||
if time.time() - last_output > STALL_TIMEOUT:
|
||||
proc.kill()
|
||||
raise SystemExit("installer produced no output for %ds" % STALL_TIMEOUT)
|
||||
|
||||
os.close(master)
|
||||
return proc.wait(), transcript.decode("utf-8", "replace")
|
||||
|
||||
|
||||
def ensure_network_boot_first():
|
||||
"""Keep network boot first so the bastion intercepts every reboot.
|
||||
|
||||
Port of the Fedora kickstart's %post efibootmgr step (install.ks.ts) --
|
||||
what makes reprovision-by-reboot work. Best-effort: skipped on BIOS boots
|
||||
or when efibootmgr is absent. Runs from the live env after the installer;
|
||||
efibootmgr edits NVRAM, not the disk, so installer cleanup is irrelevant.
|
||||
"""
|
||||
import shutil
|
||||
if not os.path.isdir("/sys/firmware/efi") or shutil.which("efibootmgr") is None:
|
||||
say("boot order: skipped (BIOS boot or efibootmgr missing)")
|
||||
return
|
||||
try:
|
||||
out = subprocess.run(["efibootmgr"], capture_output=True, text=True, timeout=30).stdout
|
||||
order = []
|
||||
network_entry = None
|
||||
for line in out.splitlines():
|
||||
m = re.match(r"^BootOrder:\\s*(.*)$", line)
|
||||
if m:
|
||||
order = [x.strip() for x in m.group(1).split(",") if x.strip()]
|
||||
continue
|
||||
m = re.match(r"^Boot([0-9A-Fa-f]{4})\\*?\\s+(.*)$", line)
|
||||
if m and network_entry is None:
|
||||
if re.search(r"network|pxe|ipv4|ipv6|http", m.group(2), re.IGNORECASE):
|
||||
network_entry = m.group(1).upper()
|
||||
if network_entry is None or not order:
|
||||
say("boot order: no network boot entry found; leaving as is")
|
||||
return
|
||||
new_order = [network_entry] + [x for x in order if x.upper() != network_entry]
|
||||
if [x.upper() for x in order] == [x.upper() for x in new_order]:
|
||||
say("boot order: network entry Boot%s already first" % network_entry)
|
||||
return
|
||||
subprocess.run(["efibootmgr", "-o", ",".join(new_order)],
|
||||
capture_output=True, timeout=30)
|
||||
say("boot order: moved network entry Boot%s first" % network_entry)
|
||||
except Exception as err:
|
||||
say("warning: boot order adjustment failed: %s" % err)
|
||||
|
||||
|
||||
def with_target_mounted(fn):
|
||||
"""Mount the installed root partition, call fn(rw_dir), always unmount.
|
||||
|
||||
The installer has unmounted and cleaned the target by the time this runs,
|
||||
so the block device is free. The partition holding boot/<image>/rw is the
|
||||
VyOS root; the glob also yields the installed image's rw dir directly.
|
||||
"""
|
||||
import glob
|
||||
disk = SPEC["disk"]
|
||||
if not disk:
|
||||
# No pinned disk (installer picked the default) -- enumerate all disks.
|
||||
candidates = ["/dev/" + b for b in os.listdir("/sys/block")
|
||||
if not b.startswith(("loop", "ram", "zram", "sr"))]
|
||||
else:
|
||||
candidates = [disk]
|
||||
|
||||
mnt = "/mnt/lab-target"
|
||||
os.makedirs(mnt, exist_ok=True)
|
||||
for dev in candidates:
|
||||
name = os.path.basename(dev)
|
||||
parts = sorted(p for p in os.listdir("/sys/block/%s" % name)
|
||||
if p.startswith(name)) if os.path.isdir("/sys/block/%s" % name) else []
|
||||
for part in parts:
|
||||
pdev = "/dev/" + part
|
||||
if subprocess.run(["mount", pdev, mnt], capture_output=True).returncode != 0:
|
||||
continue
|
||||
try:
|
||||
rw_dirs = glob.glob(os.path.join(mnt, "boot", "*", "rw"))
|
||||
if rw_dirs:
|
||||
fn(rw_dirs[0])
|
||||
return True
|
||||
finally:
|
||||
subprocess.run(["umount", mnt], capture_output=True)
|
||||
return False
|
||||
|
||||
|
||||
def post_install_target_steps():
|
||||
"""Metadata + optional fresh-config overwrite inside the installed image."""
|
||||
def apply(rw_dir):
|
||||
config_dir = os.path.join(rw_dir, "opt/vyatta/etc/config")
|
||||
os.makedirs(config_dir, exist_ok=True)
|
||||
|
||||
# /config/lab-provisioned -- survives VyOS image upgrades. Mirrors the
|
||||
# Fedora kickstart's /etc/lab-provisioned.
|
||||
try:
|
||||
with open(os.path.join(config_dir, "lab-provisioned"), "w") as handle:
|
||||
handle.write("hostname=%s\\n" % SPEC["hostname"])
|
||||
handle.write("role=%s\\n" % ROLE)
|
||||
handle.write("provisioned=%s\\n" % time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()))
|
||||
handle.write("bastion=%s\\n" % BASTION)
|
||||
say("wrote /config/lab-provisioned")
|
||||
except Exception as err:
|
||||
say("warning: lab-provisioned metadata failed: %s" % err)
|
||||
|
||||
# freshConfig: make the bastion-generated config win over the previous
|
||||
# installation's carried-forward config. Explicit intent -- failure is
|
||||
# fatal (raised out of with_target_mounted).
|
||||
if SPEC.get("freshConfig"):
|
||||
import shutil
|
||||
shutil.copyfile(os.path.join(CONFIG_DIR, "config.boot"),
|
||||
os.path.join(config_dir, "config.boot"))
|
||||
say("freshConfig: replaced installed config.boot with generated config")
|
||||
|
||||
mounted = with_target_mounted(apply)
|
||||
if not mounted:
|
||||
if SPEC.get("freshConfig"):
|
||||
raise RuntimeError("freshConfig requested but installed root partition not found")
|
||||
say("warning: installed root partition not found; skipping metadata")
|
||||
|
||||
|
||||
def main():
|
||||
report("vyos-install", "building config.boot")
|
||||
try:
|
||||
build_config()
|
||||
except Exception as err:
|
||||
report("error", "config generation failed: %s" % err)
|
||||
raise
|
||||
|
||||
report("vyos-install", "staging rootfs for installer")
|
||||
try:
|
||||
ensure_rootfs()
|
||||
except Exception as err:
|
||||
report("error", "rootfs staging failed: %s" % err)
|
||||
raise
|
||||
|
||||
report("vyos-install", "running install image")
|
||||
code, transcript = run_installer()
|
||||
|
||||
if code != 0:
|
||||
# Surface the installer's last words in bastion progress -- the console
|
||||
# they were printed on is usually invisible during unattended installs.
|
||||
report("error", "install image exited %d | tail: %s" % (code, transcript[-4000:]))
|
||||
raise SystemExit(code)
|
||||
|
||||
report("post-install", "boot order + metadata")
|
||||
ensure_network_boot_first()
|
||||
try:
|
||||
post_install_target_steps()
|
||||
except Exception as err:
|
||||
report("error", "post-install target steps failed: %s" % err)
|
||||
raise
|
||||
|
||||
# "complete" is the stage the bastion uses to move a machine out of the
|
||||
# install queue into installed state, and "ready at <ip>" is the exact
|
||||
# detail format it parses installed.ip from -- see routes/api.ts.
|
||||
ip = SPEC.get("reportAddress") or detect_ip()
|
||||
report("complete", "ready at %s" % ip if ip else "VyOS installed, rebooting")
|
||||
os.system("sync")
|
||||
# --force: this driver is a child of live-config.service, whose start job is
|
||||
# still running -- a normal reboot deadlocks waiting for it (verified in VM:
|
||||
# shutdown blocked >1min on "start job is running for live-config"). The
|
||||
# installer has already unmounted and cleaned the target, so an immediate
|
||||
# reboot is safe.
|
||||
os.system("systemctl reboot --force")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
`;
|
||||
}
|
||||
@@ -184,7 +184,8 @@ describe("renderFirstbootScript", () => {
|
||||
|
||||
it("sets hostname", () => {
|
||||
const script = renderFirstbootScript({ ...baseParams, role: "worker" });
|
||||
expect(script).toContain('hostnamectl set-hostname "test-node"');
|
||||
expect(script).toContain('CONF_HOSTNAME="test-node"');
|
||||
expect(script).toContain("hostnamectl set-hostname");
|
||||
});
|
||||
|
||||
it("includes bastion self-registration", () => {
|
||||
|
||||
@@ -22,6 +22,8 @@ function createTestConfig(testDir: string): BastionConfig {
|
||||
dhcpRangeEnd: "",
|
||||
ubuntuVersion: "26.04",
|
||||
ubuntuMirror: "https://releases.ubuntu.com/26.04",
|
||||
vyosIsoUrl: "https://downloads.vyos.io/rolling/current/generic/vyos-rolling-latest.iso",
|
||||
vyosDefaultPassword: "vyos",
|
||||
iface: "eth0",
|
||||
serverIp: "10.0.0.1",
|
||||
network: "10.0.0.0",
|
||||
|
||||
@@ -96,9 +96,9 @@ describe("renderInstallKickstart", () => {
|
||||
expect(ks).toContain("/api/progress");
|
||||
});
|
||||
|
||||
it("infra role has /var/lib/rancher partition", () => {
|
||||
it("infra role has 120G /var/lib/rancher partition", () => {
|
||||
const ks = renderInstallKickstart(baseParams({ role: "infra" }));
|
||||
expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=20480");
|
||||
expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=122880");
|
||||
});
|
||||
|
||||
it("infra role has k3s install", () => {
|
||||
@@ -106,10 +106,14 @@ describe("renderInstallKickstart", () => {
|
||||
expect(ks).toContain("curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true sh -");
|
||||
});
|
||||
|
||||
it("worker role does NOT have /var/lib/rancher partition in fresh install", () => {
|
||||
it("worker role has 120G /var/lib/rancher partition (imageFs must be sized before longhorn --grow)", () => {
|
||||
const ks = renderInstallKickstart(baseParams({ role: "worker" }));
|
||||
// Worker should not have the fresh-install rancher partition line
|
||||
expect(ks).not.toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=20480");
|
||||
expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=122880");
|
||||
});
|
||||
|
||||
it("vanilla role does NOT have /var/lib/rancher partition in fresh install", () => {
|
||||
const ks = renderInstallKickstart(baseParams({ role: "vanilla" }));
|
||||
expect(ks).not.toContain("--name=rancher --fstype=xfs");
|
||||
});
|
||||
|
||||
it("worker role does NOT have k3s install", () => {
|
||||
|
||||
155
bastion/src/bastion/tests/vyos-bundle.test.ts
Normal file
155
bastion/src/bastion/tests/vyos-bundle.test.ts
Normal file
@@ -0,0 +1,155 @@
|
||||
import { describe, it, expect, vi } from "vitest";
|
||||
import type { VyosBundle } from "@lab/shared";
|
||||
import { buildVyosConfigSpec } from "../src/templates/vyos-config-spec.js";
|
||||
|
||||
/**
|
||||
* A bundle is what makes "one config, two apply paths" true rather than
|
||||
* aspirational: `pulumi up` POSTs the subtree model to a running router, labctl
|
||||
* writes the same model into config.boot during a PXE install. These tests pin
|
||||
* the properties that keep the two honest.
|
||||
*/
|
||||
const bundle: VyosBundle = {
|
||||
sets: [
|
||||
{ path: ["system", "host-name"], value: "vyos001" },
|
||||
{ path: ["interfaces", "bonding", "bond0", "address"], value: "192.168.1.252/24" },
|
||||
{ path: ["interfaces", "bonding", "bond0", "member", "interface"], value: "eth1", replace: false },
|
||||
{ path: ["interfaces", "bonding", "bond0", "vif", "53", "disable"] },
|
||||
{ path: ["interfaces", "pppoe", "pppoe0", "authentication", "password"], value: "@secret:pppoePassword" },
|
||||
{ path: ["interfaces", "pppoe", "pppoe0", "mtu"], value: "1492" },
|
||||
],
|
||||
tags: [["interfaces", "bonding", "bond0"], ["interfaces", "ethernet"]],
|
||||
};
|
||||
|
||||
const build = (hostname: string, extra: Record<string, unknown> = {}) =>
|
||||
buildVyosConfigSpec({
|
||||
hostname,
|
||||
spec: { bundle, ...extra },
|
||||
defaultPassword: "changeme",
|
||||
});
|
||||
|
||||
describe("vyos config spec from a Pulumi bundle", () => {
|
||||
it("applies non-secret nodes verbatim, preserving valuelessness and replace:false", () => {
|
||||
const spec = build("vyos001");
|
||||
|
||||
expect(spec.sets).toContainEqual({
|
||||
path: ["interfaces", "bonding", "bond0", "address"],
|
||||
value: "192.168.1.252/24",
|
||||
});
|
||||
// A multi-value node must keep replace:false or the second bond member
|
||||
// overwrites the first.
|
||||
expect(spec.sets).toContainEqual({
|
||||
path: ["interfaces", "bonding", "bond0", "member", "interface"],
|
||||
value: "eth1",
|
||||
replace: false,
|
||||
});
|
||||
// A valueless node must not acquire a value on the way through.
|
||||
expect(spec.sets).toContainEqual({
|
||||
path: ["interfaces", "bonding", "bond0", "vif", "53", "disable"],
|
||||
});
|
||||
expect(spec.tags).toEqual(bundle.tags);
|
||||
});
|
||||
|
||||
it("drops secret-valued nodes instead of installing the sentinel text", () => {
|
||||
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
|
||||
const spec = build("vyos001");
|
||||
|
||||
const values = spec.sets.map((s) => s.value ?? "");
|
||||
expect(values.some((v) => v.startsWith("@secret:"))).toBe(false);
|
||||
expect(spec.sets.some((s) => s.path.includes("authentication"))).toBe(false);
|
||||
// Silently dropping the WAN credential would leave someone debugging a dead
|
||||
// PPPoE link, so it has to be said out loud.
|
||||
expect(warn).toHaveBeenCalledWith(expect.stringContaining("pulumi up"));
|
||||
warn.mockRestore();
|
||||
});
|
||||
|
||||
it("forces the hostname the install was asked for, not the bundle's", () => {
|
||||
// The bundle is exported from one router and reused for its peer; taking the
|
||||
// hostname from it would put two vyos001s on the network.
|
||||
const spec = build("vyos002");
|
||||
const hostnames = spec.sets.filter(
|
||||
(s) => s.path.length === 2 && s.path[0] === "system" && s.path[1] === "host-name",
|
||||
);
|
||||
expect(hostnames).toEqual([{ path: ["system", "host-name"], value: "vyos002" }]);
|
||||
});
|
||||
|
||||
it("still honours installer inputs, which are not router config", () => {
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: "vyos001",
|
||||
spec: { bundle, password: "s3cret", freshConfig: true },
|
||||
defaultPassword: "changeme",
|
||||
disk: "nvme0n1",
|
||||
});
|
||||
expect(spec.password).toBe("s3cret");
|
||||
expect(spec.freshConfig).toBe(true);
|
||||
expect(spec.disk).toBe("/dev/nvme0n1");
|
||||
});
|
||||
|
||||
it("enables the HTTP API at install so Pulumi can manage the router from first boot", () => {
|
||||
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: "vyos001",
|
||||
spec: { bundle, apiKey: "k3y", apiListenAddress: "10.0.1.252" },
|
||||
defaultPassword: "changeme",
|
||||
});
|
||||
|
||||
expect(spec.sets).toContainEqual({
|
||||
path: ["service", "https", "api", "keys", "id", "pulumi", "key"],
|
||||
value: "k3y",
|
||||
});
|
||||
expect(spec.sets).toContainEqual({ path: ["service", "https", "api", "rest"] });
|
||||
expect(spec.sets).toContainEqual({
|
||||
path: ["service", "https", "listen-address"],
|
||||
value: "10.0.1.252",
|
||||
});
|
||||
// The key id is a tag node; without this the installer's ConfigTree rejects it.
|
||||
expect(spec.tags).toContainEqual(["service", "https", "api", "keys", "id"]);
|
||||
warn.mockRestore();
|
||||
});
|
||||
|
||||
it("binds the API to the static management address when none is given", () => {
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: "vyos001",
|
||||
spec: { apiKey: "k3y", mgmtAddress: "192.168.1.252/24" },
|
||||
defaultPassword: "changeme",
|
||||
});
|
||||
expect(spec.sets).toContainEqual({
|
||||
path: ["service", "https", "listen-address"],
|
||||
value: "192.168.1.252",
|
||||
});
|
||||
});
|
||||
|
||||
it("refuses to enable the API unbound rather than exposing it on the WAN", () => {
|
||||
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
|
||||
// Management is DHCP, so there is no address to bind at build time. Binding
|
||||
// to everything would put a config-write endpoint on the WAN.
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: "vyos001",
|
||||
spec: { apiKey: "k3y", mgmtAddress: "dhcp" },
|
||||
defaultPassword: "changeme",
|
||||
});
|
||||
expect(spec.sets.some((s) => s.path[0] === "service" && s.path[1] === "https")).toBe(false);
|
||||
expect(warn).toHaveBeenCalledWith(expect.stringContaining("has NOT been enabled"));
|
||||
warn.mockRestore();
|
||||
});
|
||||
|
||||
it("does not enable the API when no key is supplied", () => {
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: "vyos001",
|
||||
spec: { mgmtAddress: "192.168.1.252/24" },
|
||||
defaultPassword: "changeme",
|
||||
});
|
||||
expect(spec.sets.some((s) => s.path[0] === "service" && s.path[1] === "https")).toBe(false);
|
||||
});
|
||||
|
||||
it("ignores the derived path entirely when a bundle is present", () => {
|
||||
// Belt and braces: even if topology flags reach this far (the CLI rejects
|
||||
// them), the bundle must win rather than merge.
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: "vyos001",
|
||||
spec: { bundle, bondMembers: ["eth2", "eth3"], vlans: [{ id: 99, address: "10.9.9.1/24" }] },
|
||||
defaultPassword: "changeme",
|
||||
});
|
||||
expect(spec.sets.some((s) => s.path.includes("99"))).toBe(false);
|
||||
expect(spec.sets.filter((s) => s.value === "eth2" || s.value === "eth3")).toEqual([]);
|
||||
});
|
||||
});
|
||||
548
bastion/src/bastion/tests/vyos.test.ts
Normal file
548
bastion/src/bastion/tests/vyos.test.ts
Normal file
@@ -0,0 +1,548 @@
|
||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||
import { mkdirSync, rmSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { BastionConfig } from "@lab/shared";
|
||||
import type { FastifyInstance } from "fastify";
|
||||
import { createApp } from "../src/server.js";
|
||||
import type { StateManager } from "../src/services/state.js";
|
||||
import { buildVyosConfigSpec } from "../src/templates/vyos-config-spec.js";
|
||||
import { renderVyosInstallPy } from "../src/templates/vyos-install.py.js";
|
||||
|
||||
function createTestConfig(testDir: string): BastionConfig {
|
||||
return {
|
||||
fedoraVersion: "43",
|
||||
arch: "x86_64",
|
||||
httpPort: 0,
|
||||
timezone: "Europe/London",
|
||||
locale: "en_GB.UTF-8",
|
||||
bastionDir: testDir,
|
||||
domain: "test.local",
|
||||
dhcpMode: "proxy",
|
||||
dhcpRangeStart: "",
|
||||
dhcpRangeEnd: "",
|
||||
ubuntuVersion: "26.04",
|
||||
ubuntuMirror: "https://releases.ubuntu.com/26.04",
|
||||
vyosIsoUrl: "https://example.invalid/vyos.iso",
|
||||
vyosDefaultPassword: "test-pw",
|
||||
iface: "eth0",
|
||||
serverIp: "10.0.0.1",
|
||||
network: "10.0.0.0",
|
||||
gateway: "10.0.0.1",
|
||||
sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST lab@test"],
|
||||
adminUser: "testadmin",
|
||||
syslogPort: 15515,
|
||||
skipDnsmasq: true,
|
||||
skipArtifacts: true,
|
||||
fedoraMirror: "https://example.invalid/fedora",
|
||||
tftpDir: join(testDir, "tftp"),
|
||||
httpDir: join(testDir, "http"),
|
||||
stateFile: join(testDir, "state.json"),
|
||||
};
|
||||
}
|
||||
|
||||
/** Pull the base64 spec back out of the generated Python driver. */
|
||||
function decodeSpecFrom(python: string): Record<string, unknown> {
|
||||
const match = /base64\.b64decode\("([^"]+)"\)/.exec(python);
|
||||
if (!match?.[1]) throw new Error("no base64 spec found in generated driver");
|
||||
return JSON.parse(Buffer.from(match[1], "base64").toString("utf-8"));
|
||||
}
|
||||
|
||||
describe("vyos config spec", () => {
|
||||
it("puts VLANs on the bond when members are given", () => {
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: "fw1",
|
||||
defaultPassword: "pw",
|
||||
spec: {
|
||||
mgmtInterface: "eth0",
|
||||
mgmtAddress: "10.0.8.2/24",
|
||||
bondMembers: ["eth2", "eth3"],
|
||||
vlans: [{ id: 10, address: "10.0.10.1/24", description: "k8s" }],
|
||||
},
|
||||
});
|
||||
|
||||
const paths = spec.sets.map((s) => s.path.join(" "));
|
||||
expect(paths).toContain("interfaces bonding bond0 mode");
|
||||
expect(paths).toContain("interfaces bonding bond0 vif 10 address");
|
||||
// VLANs must hang off the bond, not the management NIC.
|
||||
expect(paths).not.toContain("interfaces ethernet eth0 vif 10 address");
|
||||
|
||||
// Bond members are a multi-value node — appending, not replacing, is what
|
||||
// keeps the second member from overwriting the first.
|
||||
const members = spec.sets.filter(
|
||||
(s) => s.path.join(" ") === "interfaces bonding bond0 member interface",
|
||||
);
|
||||
expect(members.map((m) => m.value)).toEqual(["eth2", "eth3"]);
|
||||
expect(members.every((m) => m.replace === false)).toBe(true);
|
||||
});
|
||||
|
||||
it("falls back to VLANs on the management NIC when unbonded", () => {
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: "fw2",
|
||||
defaultPassword: "pw",
|
||||
spec: { mgmtInterface: "eth1", vlans: [{ id: 20, address: "10.0.20.1/24" }] },
|
||||
});
|
||||
|
||||
const paths = spec.sets.map((s) => s.path.join(" "));
|
||||
expect(paths).toContain("interfaces ethernet eth1 vif 20 address");
|
||||
});
|
||||
|
||||
it("normalises the target disk to a full /dev path", () => {
|
||||
// find_disks() enumerates with `lsblk -Jbp`, so valid responses are full
|
||||
// paths; a bare name fails valid_responses and re-prompts forever.
|
||||
expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw", disk: "/dev/mmcblk0" }).disk)
|
||||
.toBe("/dev/mmcblk0");
|
||||
expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw", disk: "mmcblk0" }).disk)
|
||||
.toBe("/dev/mmcblk0");
|
||||
expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw" }).disk).toBe("");
|
||||
});
|
||||
|
||||
it("defaults to dhcp on eth0 and never opts into RAID", () => {
|
||||
const spec = buildVyosConfigSpec({ hostname: "fw4", defaultPassword: "pw" });
|
||||
const address = spec.sets.find(
|
||||
(s) => s.path.join(" ") === "interfaces ethernet eth0 address",
|
||||
);
|
||||
expect(address?.value).toBe("dhcp");
|
||||
// The installer's RAID prompt defaults to yes; a second disk must not
|
||||
// silently produce a mirror.
|
||||
expect(spec.raid).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("vyos routes", () => {
|
||||
let testDir: string;
|
||||
let app: FastifyInstance;
|
||||
let state: StateManager;
|
||||
const mac = "aa:bb:cc:11:22:33";
|
||||
|
||||
beforeEach(() => {
|
||||
testDir = join(tmpdir(), `bastion-vyos-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
|
||||
mkdirSync(join(testDir, "http"), { recursive: true });
|
||||
mkdirSync(join(testDir, "tftp"), { recursive: true });
|
||||
|
||||
const result = createApp(createTestConfig(testDir));
|
||||
app = result.app;
|
||||
state = result.state;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await app.close();
|
||||
rmSync(testDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("dispatches a queued vyos machine to the live-boot script", async () => {
|
||||
state.update((s) => {
|
||||
s.install_queue[mac] = {
|
||||
hostname: "fw1",
|
||||
disk: "/dev/nvme0n1",
|
||||
role: "worker",
|
||||
os: "vyos-rolling",
|
||||
queued_at: new Date().toISOString(),
|
||||
};
|
||||
});
|
||||
|
||||
const response = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
|
||||
expect(response.statusCode).toBe(200);
|
||||
|
||||
expect(response.body).toContain("/vyos-vmlinuz");
|
||||
expect(response.body).toContain("fetch=http://10.0.0.1:0/vyos-filesystem.squashfs");
|
||||
expect(response.body).toContain(`live-config.hooks=http://10.0.0.1:0/vyos/autoinstall.sh?mac=${mac}`);
|
||||
|
||||
// `nonetworking` appears in VyOS's own PXE docs but breaks the hook fetch,
|
||||
// and console=ttyS0 costs 30s per systemd phase on boards with no UART.
|
||||
expect(response.body).not.toContain("nonetworking");
|
||||
expect(response.body).not.toContain("console=ttyS0");
|
||||
});
|
||||
|
||||
it("serves a hook that fetches and executes the install driver", async () => {
|
||||
const response = await app.inject({ method: "GET", url: `/vyos/autoinstall.sh?mac=${mac}` });
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(response.body).toContain(`/vyos/install.py?mac=${mac}`);
|
||||
expect(response.body).toContain("python3 /tmp/vyos-install.py");
|
||||
});
|
||||
|
||||
it("bakes the machine's config into the generated install driver", async () => {
|
||||
state.update((s) => {
|
||||
s.install_queue[mac] = {
|
||||
hostname: "fw1",
|
||||
disk: "/dev/nvme0n1",
|
||||
role: "worker",
|
||||
os: "vyos-rolling",
|
||||
queued_at: new Date().toISOString(),
|
||||
vyos: {
|
||||
mgmtInterface: "eth0",
|
||||
mgmtAddress: "10.0.8.2/24",
|
||||
bondMembers: ["eth2", "eth3"],
|
||||
vlans: [{ id: 10, address: "10.0.10.1/24" }],
|
||||
password: "s3cret",
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
const response = await app.inject({ method: "GET", url: `/vyos/install.py?mac=${mac}` });
|
||||
expect(response.statusCode).toBe(200);
|
||||
|
||||
// Builds config from the image's own default so the vyos-config-version
|
||||
// trailer matches and first boot skips migrations.
|
||||
expect(response.body).toContain("/opt/vyatta/etc/config.boot.default");
|
||||
expect(response.body).toContain("/usr/libexec/vyos/op_mode/image_installer.py");
|
||||
// "complete" is what moves the machine out of the install queue.
|
||||
expect(response.body).toContain('report("complete"');
|
||||
|
||||
const spec = decodeSpecFrom(response.body);
|
||||
expect(spec["hostname"]).toBe("fw1");
|
||||
expect(spec["password"]).toBe("s3cret");
|
||||
expect(spec["disk"]).toBe("/dev/nvme0n1");
|
||||
|
||||
const paths = (spec["sets"] as Array<{ path: string[] }>).map((s) => s.path.join(" "));
|
||||
expect(paths).toContain("interfaces bonding bond0 vif 10 address");
|
||||
expect(paths).toContain("system host-name");
|
||||
});
|
||||
|
||||
it("falls back to the bastion default password when none is set", async () => {
|
||||
state.update((s) => {
|
||||
s.install_queue[mac] = {
|
||||
hostname: "fw9",
|
||||
disk: "",
|
||||
role: "worker",
|
||||
os: "vyos-rolling",
|
||||
queued_at: new Date().toISOString(),
|
||||
};
|
||||
});
|
||||
|
||||
const response = await app.inject({ method: "GET", url: `/vyos/install.py?mac=${mac}` });
|
||||
const spec = decodeSpecFrom(response.body);
|
||||
expect(spec["password"]).toBe("test-pw");
|
||||
// Empty disk means "accept the installer's first-disk default".
|
||||
expect(spec["disk"]).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("vyos hw-id pinning", () => {
|
||||
it("emits hw-id for the mgmt interface and each bond member", () => {
|
||||
// Discovery sees enp2s0/enp1s0f0np0 under Fedora, but VyOS enumerates its
|
||||
// own eth<N>. Pinning by MAC is what makes the mapping deterministic.
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: "fw1",
|
||||
defaultPassword: "pw",
|
||||
spec: {
|
||||
mgmtInterface: "eth2",
|
||||
bondMembers: ["eth0", "eth1"],
|
||||
hwIds: {
|
||||
eth2: "64:62:66:25:96:47",
|
||||
eth0: "64:62:66:25:96:45",
|
||||
eth1: "64:62:66:25:96:46",
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
const hw = spec.sets.filter((s) => s.path[s.path.length - 1] === "hw-id");
|
||||
expect(hw.map((s) => [s.path[2], s.value])).toEqual([
|
||||
["eth2", "64:62:66:25:96:47"],
|
||||
["eth0", "64:62:66:25:96:45"],
|
||||
["eth1", "64:62:66:25:96:46"],
|
||||
]);
|
||||
});
|
||||
|
||||
it("omits hw-id entirely when no mapping is given", () => {
|
||||
const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" });
|
||||
expect(spec.sets.some((s) => s.path.includes("hw-id"))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("vyos management VLAN", () => {
|
||||
it("puts the mgmt VLAN on the PXE port while the bond carries routed VLANs", () => {
|
||||
// Trunked PXE port: boots untagged on the VLAN the bastion serves, stays
|
||||
// reachable on the tagged management VLAN.
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: "vyos001",
|
||||
defaultPassword: "pw",
|
||||
spec: {
|
||||
mgmtInterface: "eth2",
|
||||
mgmtAddress: "dhcp",
|
||||
mgmtVlan: { id: 3, address: "192.168.3.4/24", description: "kvm" },
|
||||
bondMembers: ["eth0", "eth1"],
|
||||
vlans: [{ id: 2, address: "192.168.8.2/23" }],
|
||||
},
|
||||
});
|
||||
|
||||
const paths = spec.sets.map((s) => s.path.join(" "));
|
||||
expect(paths).toContain("interfaces ethernet eth2 vif 3 address");
|
||||
expect(paths).toContain("interfaces bonding bond0 vif 2 address");
|
||||
// The mgmt VLAN must not land on the bond.
|
||||
expect(paths).not.toContain("interfaces bonding bond0 vif 3 address");
|
||||
expect(spec.tags.map((t) => t.join(" "))).toContain("interfaces ethernet eth2 vif");
|
||||
});
|
||||
});
|
||||
|
||||
describe("vyos VRRP HA", () => {
|
||||
const haSpec = {
|
||||
mgmtInterface: "eth2",
|
||||
mgmtAddress: "dhcp",
|
||||
bondMembers: ["eth0", "eth1"],
|
||||
bondAddress: "192.168.1.252/24",
|
||||
bondVrrp: "192.168.1.254/24",
|
||||
vrrpPriority: 200,
|
||||
vlans: [
|
||||
{ id: 3, address: "192.168.3.4/24", vrrp: "192.168.3.254/24" },
|
||||
{ id: 200, address: "192.168.2.252/24" }, // no VIP on this one
|
||||
],
|
||||
};
|
||||
|
||||
it("emits a vrrp group per VIP with vrid = VLAN id and dotted vif interface", () => {
|
||||
const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw", spec: haSpec });
|
||||
const paths = spec.sets.map((s) => `${s.path.join(" ")}${s.value !== undefined ? "=" + s.value : ""}`);
|
||||
|
||||
expect(paths).toContain("interfaces bonding bond0 address=192.168.1.252/24");
|
||||
// untagged bond group: vrid 1, interface bond0 itself
|
||||
expect(paths).toContain("high-availability vrrp group native interface=bond0");
|
||||
expect(paths).toContain("high-availability vrrp group native vrid=1");
|
||||
// address is a tag node -- VIP is the final path segment, no value
|
||||
expect(paths).toContain("high-availability vrrp group native address 192.168.1.254/24");
|
||||
// VLAN group: vrid = VLAN id, dotted vif
|
||||
expect(paths).toContain("high-availability vrrp group vlan3 interface=bond0.3");
|
||||
expect(paths).toContain("high-availability vrrp group vlan3 vrid=3");
|
||||
expect(paths).toContain("high-availability vrrp group vlan3 address 192.168.3.254/24");
|
||||
// VLAN without a VIP gets no group
|
||||
expect(paths.some((p) => p.includes("group vlan200"))).toBe(false);
|
||||
});
|
||||
|
||||
it("applies the box-wide priority and one sync group over all groups", () => {
|
||||
const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw", spec: haSpec });
|
||||
const prio = spec.sets.filter((s) => s.path[s.path.length - 1] === "priority"
|
||||
&& s.path[0] === "high-availability");
|
||||
expect(prio).toHaveLength(2);
|
||||
expect(prio.every((s) => s.value === "200")).toBe(true);
|
||||
|
||||
// sync group binds the pair: all groups fail over together
|
||||
const members = spec.sets.filter(
|
||||
(s) => s.path.join(" ") === "high-availability vrrp sync-group MAIN member",
|
||||
);
|
||||
expect(members.map((m) => m.value)).toEqual(["native", "vlan3"]);
|
||||
expect(members.every((m) => m.replace === false)).toBe(true);
|
||||
});
|
||||
|
||||
it("emits no high-availability nodes when no VIPs are given", () => {
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: "fw1",
|
||||
defaultPassword: "pw",
|
||||
spec: { bondMembers: ["eth0", "eth1"], vlans: [{ id: 3, address: "192.168.3.4/24" }] },
|
||||
});
|
||||
expect(spec.sets.some((s) => s.path[0] === "high-availability")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("pickLargestInitrd", async () => {
|
||||
const { pickLargestInitrd } = await import("../src/main.js");
|
||||
|
||||
// Verbatim from `xorriso -lsl /live/` on vyos-2026.08.05-0033-rolling.
|
||||
const realListing = `total 8
|
||||
-r--r--r-- 1 0 0 22255 Aug 5 01:33 'filesystem.packages'
|
||||
-r--r--r-- 1 0 0 6 Aug 5 01:33 'filesystem.packages-remove'
|
||||
-r--r--r-- 1 0 0 541192192 Aug 5 01:33 'filesystem.squashfs'
|
||||
-r--r--r-- 1 0 0 50352547 Aug 5 01:33 'initrd.img'
|
||||
-r--r--r-- 1 0 0 50352547 Aug 5 01:33 'initrd.img-6.18.41-vyos'
|
||||
-r--r--r-- 1 0 0 20 Aug 5 01:33 'packages.txt'
|
||||
-r--r--r-- 1 0 0 9135104 Aug 2 19:54 'vmlinuz'
|
||||
-r--r--r-- 1 0 0 9135104 Aug 2 19:54 'vmlinuz-6.18.41-vyos'
|
||||
`;
|
||||
|
||||
it("picks a full-size initrd from a real nightly listing", () => {
|
||||
expect(pickLargestInitrd(realListing)).toEqual({ name: "initrd.img", size: 50352547 });
|
||||
});
|
||||
|
||||
it("ignores 0-byte decoys and symlinks (which report link size, not target size)", () => {
|
||||
const listing = `total 8
|
||||
-r--r--r-- 1 0 0 0 Aug 5 01:33 'initrd.img'
|
||||
lrwxrwxrwx 1 0 0 24 Aug 5 01:33 'initrd.img-link' -> 'initrd.img-6.18.41-vyos'
|
||||
-r--r--r-- 1 0 0 50352547 Aug 5 01:33 'initrd.img-6.18.41-vyos'
|
||||
`;
|
||||
expect(pickLargestInitrd(listing)).toEqual({ name: "initrd.img-6.18.41-vyos", size: 50352547 });
|
||||
});
|
||||
|
||||
it("returns undefined when only decoys exist", () => {
|
||||
expect(pickLargestInitrd("-r--r--r-- 1 0 0 0 Aug 5 01:33 'initrd.img'\n")).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("vyos fedora-parity features", () => {
|
||||
it("computes reportAddress from a static mgmt address, empty for dhcp", () => {
|
||||
const staticSpec = buildVyosConfigSpec({
|
||||
hostname: "fw1", defaultPassword: "pw",
|
||||
spec: { mgmtAddress: "192.168.8.2/23" },
|
||||
});
|
||||
expect(staticSpec.reportAddress).toBe("192.168.8.2");
|
||||
|
||||
const dhcpSpec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" });
|
||||
expect(dhcpSpec.reportAddress).toBe("");
|
||||
});
|
||||
|
||||
it("defaults freshConfig off (reinstall preserves the on-disk config)", () => {
|
||||
expect(buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" }).freshConfig).toBe(false);
|
||||
expect(buildVyosConfigSpec({
|
||||
hostname: "fw1", defaultPassword: "pw", spec: { freshConfig: true },
|
||||
}).freshConfig).toBe(true);
|
||||
});
|
||||
|
||||
it("driver streams logs to /api/log and reports 'ready at' on completion", async () => {
|
||||
const testDir = join(tmpdir(), `bastion-vyos-parity-${Date.now()}`);
|
||||
mkdirSync(join(testDir, "http"), { recursive: true });
|
||||
mkdirSync(join(testDir, "tftp"), { recursive: true });
|
||||
const { app: parityApp, state: parityState } = createApp(createTestConfig(testDir));
|
||||
try {
|
||||
parityState.update((s) => {
|
||||
s.install_queue["aa:bb:cc:44:55:66"] = {
|
||||
hostname: "fw9", disk: "/dev/vda", role: "vanilla",
|
||||
os: "vyos-rolling", queued_at: new Date().toISOString(),
|
||||
};
|
||||
});
|
||||
const response = await parityApp.inject({
|
||||
method: "GET", url: "/vyos/install.py?mac=aa:bb:cc:44:55:66",
|
||||
});
|
||||
expect(response.body).toContain("/api/log");
|
||||
expect(response.body).toContain('"lines": batch');
|
||||
expect(response.body).toContain('report("complete", "ready at %s"');
|
||||
expect(response.body).toContain("ensure_network_boot_first");
|
||||
expect(response.body).toContain("lab-provisioned");
|
||||
expect(response.body).toContain('ROLE = "vanilla"');
|
||||
} finally {
|
||||
await parityApp.close();
|
||||
rmSync(testDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("complete with 'ready at' records installed.ip for a vyos machine", async () => {
|
||||
const testDir = join(tmpdir(), `bastion-vyos-complete-${Date.now()}`);
|
||||
mkdirSync(join(testDir, "http"), { recursive: true });
|
||||
mkdirSync(join(testDir, "tftp"), { recursive: true });
|
||||
const { app: cApp, state: cState } = createApp(createTestConfig(testDir));
|
||||
try {
|
||||
const mac2 = "aa:bb:cc:77:88:99";
|
||||
cState.update((s) => {
|
||||
s.install_queue[mac2] = {
|
||||
hostname: "fw1", disk: "/dev/vda", role: "vanilla",
|
||||
os: "vyos-rolling", queued_at: new Date().toISOString(),
|
||||
};
|
||||
});
|
||||
const response = await cApp.inject({
|
||||
method: "POST", url: "/api/progress",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ mac: mac2, stage: "complete", detail: "ready at 192.168.8.2" }),
|
||||
});
|
||||
expect(response.statusCode).toBe(200);
|
||||
const installed = cState.load().installed[mac2];
|
||||
expect(installed?.ip).toBe("192.168.8.2");
|
||||
expect(installed?.os).toBe("vyos-rolling");
|
||||
} finally {
|
||||
await cApp.close();
|
||||
rmSync(testDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("vyos installer prompt coverage", () => {
|
||||
// Every interactive prompt image_installer.py can emit, copied verbatim from
|
||||
// the MSG_* constants (including the reinstall-only search_previous_installation
|
||||
// ones). An unanswered prompt does not fail loudly -- the installer simply
|
||||
// blocks on stdin until the driver's stall timeout, which is how the reinstall
|
||||
// path silently hung for 15 minutes in the VM test.
|
||||
const PROMPTS: Record<string, string> = {
|
||||
continue: "Would you like to continue? [y/N] ",
|
||||
imageName: "What would you like to name this image? (Default: 1.5-rolling) ",
|
||||
password: 'Please enter a password for the "vyos" user: ',
|
||||
passwordConfirm: 'Please confirm password for the "vyos" user: ',
|
||||
console: "What console should be used by default? (K: KVM, S: Serial)? (Default: K) ",
|
||||
raidConfigure: "Would you like to configure RAID-1 mirroring? [Y/n] ",
|
||||
raidFoundDisks: "Would you like to configure RAID-1 mirroring on them? [Y/n] ",
|
||||
raidChooseDisks: "Would you like to choose two disks for RAID-1 mirroring? [Y/n] ",
|
||||
diskSelect: "Which one should be used for installation? (Default: /dev/vda) ",
|
||||
diskConfirm: "Installation will delete all data on the drive. Continue? [y/N] ",
|
||||
raidConfirm: "Installation will delete all data on both drives. Continue? [y/N] ",
|
||||
rootSizeAll: "Would you like to use all the free space on the drive? [Y/n] ",
|
||||
bootConfig: "Which file would you like as boot config? ",
|
||||
copyData: "Would you like to copy data to the new image? [Y/n] ",
|
||||
chooseCopyData: "From which image would you like to save config information? ",
|
||||
copyEncData: "Would you like to copy the encrypted config to the new image? [Y/n] ",
|
||||
chooseCopyEncData: "From which image would you like to copy the encrypted config? ",
|
||||
};
|
||||
|
||||
it("answers every installer prompt exactly once", () => {
|
||||
const { execFileSync } = require("node:child_process") as typeof import("node:child_process");
|
||||
const { writeFileSync, unlinkSync, mkdtempSync } = require("node:fs") as typeof import("node:fs");
|
||||
|
||||
// Skip cleanly where python3 is unavailable (same spirit as the
|
||||
// ksvalidator-backed kickstart test).
|
||||
try {
|
||||
execFileSync("python3", ["--version"], { stdio: "pipe" });
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
|
||||
const spec = buildVyosConfigSpec({
|
||||
hostname: "fw1", defaultPassword: "pw", disk: "/dev/vda",
|
||||
});
|
||||
const driver = renderVyosInstallPy({
|
||||
spec, mac: "aa:bb:cc:11:22:33", serverIp: "10.0.0.1", httpPort: 8080, role: "vanilla",
|
||||
});
|
||||
|
||||
const dir = mkdtempSync(join(tmpdir(), "vyos-rules-"));
|
||||
const driverPath = join(dir, "driver.py");
|
||||
const checkPath = join(dir, "check.py");
|
||||
writeFileSync(driverPath, driver);
|
||||
writeFileSync(checkPath, `
|
||||
import importlib.util, json, sys
|
||||
spec = importlib.util.spec_from_file_location("drv", ${JSON.stringify(driverPath)})
|
||||
drv = importlib.util.module_from_spec(spec); spec.loader.exec_module(drv)
|
||||
rules = drv.build_rules()
|
||||
prompts = json.loads(sys.argv[1])
|
||||
out = {}
|
||||
for label, text in prompts.items():
|
||||
out[label] = len([r for p, r in rules if p.search(text.encode())])
|
||||
print(json.dumps(out))
|
||||
`);
|
||||
|
||||
try {
|
||||
const stdout = execFileSync("python3", [checkPath, JSON.stringify(PROMPTS)], {
|
||||
encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"],
|
||||
});
|
||||
const counts = JSON.parse(stdout) as Record<string, number>;
|
||||
const unanswered = Object.entries(counts).filter(([, n]) => n !== 1);
|
||||
expect(unanswered).toEqual([]);
|
||||
} finally {
|
||||
try { unlinkSync(driverPath); unlinkSync(checkPath); } catch { /* best effort */ }
|
||||
try { rmSync(dir, { recursive: true, force: true }); } catch { /* best effort */ }
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("vyos boot NIC pinning", () => {
|
||||
it("pins the boot interface by MAC via BOOTIF", async () => {
|
||||
// Without this, live-boot picks the first *connected* NIC. On the VP2440
|
||||
// the SFP+ pair links before the copper PXE port, so live-boot tried the
|
||||
// fiber ports (no DHCP), timed out 15s each, and failed with "Unable to
|
||||
// find a live file system on the network".
|
||||
const testDir = join(tmpdir(), `bastion-vyos-bootif-${Date.now()}`);
|
||||
mkdirSync(join(testDir, "http"), { recursive: true });
|
||||
mkdirSync(join(testDir, "tftp"), { recursive: true });
|
||||
const { app: a, state: st } = createApp(createTestConfig(testDir));
|
||||
try {
|
||||
const m = "64:62:66:25:96:47";
|
||||
st.update((s) => {
|
||||
s.install_queue[m] = {
|
||||
hostname: "vyos001", disk: "/dev/mmcblk0", role: "vanilla",
|
||||
os: "vyos-rolling", queued_at: new Date().toISOString(),
|
||||
};
|
||||
});
|
||||
const res = await a.inject({ method: "GET", url: `/dispatch?mac=${m}` });
|
||||
// live-boot's Device_from_bootif() expects 01-<mac with dashes>
|
||||
expect(res.body).toContain("BOOTIF=01-64-62-66-25-96-47");
|
||||
// and it must be on the kernel line, before fetch= is attempted
|
||||
const kernelLine = res.body.split("\n").find((l) => l.startsWith("kernel "));
|
||||
expect(kernelLine).toContain("BOOTIF=01-64-62-66-25-96-47");
|
||||
expect(kernelLine).toContain("fetch=");
|
||||
} finally {
|
||||
await a.close();
|
||||
rmSync(testDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -90,6 +90,7 @@ export class LabdClient {
|
||||
|
||||
async installMachine(opts: {
|
||||
mac: string; hostname: string; disk?: string; role?: string; os?: string;
|
||||
vyos?: import("@lab/shared").VyosInstallSpec;
|
||||
}): Promise<{ status: string; data?: unknown; error?: string }> {
|
||||
return this.request("POST", "/api/machines/install", { body: opts });
|
||||
}
|
||||
@@ -104,6 +105,16 @@ export class LabdClient {
|
||||
return this.request("POST", "/api/machines/debug", { body: { mac, pxeBoot: opts?.pxeBoot } });
|
||||
}
|
||||
|
||||
async discoverMachine(data: {
|
||||
mac: string; product?: string; board?: string; serial?: string;
|
||||
manufacturer?: string; cpu_model?: string; cpu_cores?: number;
|
||||
memory_gb?: number; arch?: string;
|
||||
disks?: Array<{ name: string; size_gb: number; model: string }>;
|
||||
nics?: Array<{ name: string; mac: string; state: string }>;
|
||||
}): Promise<{ status: string; error?: string }> {
|
||||
return this.request("POST", "/api/machines/discover", { body: data });
|
||||
}
|
||||
|
||||
async forgetMachine(mac: string): Promise<{ status: string }> {
|
||||
return this.request("DELETE", `/api/machines/${encodeURIComponent(mac)}`);
|
||||
}
|
||||
|
||||
@@ -70,7 +70,7 @@ export function registerAppCommand(program: Command): void {
|
||||
.command("install <target>")
|
||||
.description("Install k3s on a target machine (hostname, IP, or MAC)")
|
||||
.option("--role <role>", "k3s role: infra (server) or worker (agent)", "infra")
|
||||
.option("--user <user>", "SSH user", "lab")
|
||||
.option("--user <user>", "SSH user", "root")
|
||||
.option("--k3s-server <url>", "k3s server URL (required for worker role)")
|
||||
.option("--k3s-token <token>", "k3s join token (required for worker role)")
|
||||
.action(async (target: string, opts: {
|
||||
@@ -164,7 +164,7 @@ export function registerAppCommand(program: Command): void {
|
||||
k3sCmd
|
||||
.command("health [target]")
|
||||
.description("Check k3s health (all hosts if no target given)")
|
||||
.option("--user <user>", "SSH user", "lab")
|
||||
.option("--user <user>", "SSH user", "root")
|
||||
.action(async (target: string | undefined, opts: { user: string }) => {
|
||||
const sshKey = findSshKey();
|
||||
|
||||
@@ -304,7 +304,7 @@ export function registerAppCommand(program: Command): void {
|
||||
k3sCmd
|
||||
.command("list")
|
||||
.description("List installed machines and their k3s status")
|
||||
.option("--user <user>", "SSH user", "lab")
|
||||
.option("--user <user>", "SSH user", "root")
|
||||
.action(async (opts: { user: string }) => {
|
||||
let state: BastionState;
|
||||
try {
|
||||
|
||||
@@ -59,9 +59,9 @@ export function registerAsahiCommand(parent: Command): void {
|
||||
console.log(` labvg/longhorn (remaining space)${RESET}`);
|
||||
console.log("");
|
||||
console.log(` After first boot, SSH in and run the firstboot script:`);
|
||||
console.log(` ${BOLD}ssh root@<ip> 'curl -sf ${bastionUrl}/asahi/firstboot.sh?hostname=<name>\\&role=infra | bash'${RESET}`);
|
||||
console.log(` ${BOLD}ssh root@<ip> 'curl -sf ${bastionUrl}/asahi/firstboot.sh | bash'${RESET}`);
|
||||
console.log("");
|
||||
console.log(` This sets up LVM and self-registers with the bastion.`);
|
||||
console.log(` This sets up LVM, detects hostname/MAC, and self-registers.`);
|
||||
console.log(` Then install k3s:`);
|
||||
console.log(` ${BOLD}labctl app k3s install <hostname> --role infra${RESET}`);
|
||||
console.log("");
|
||||
|
||||
@@ -1,10 +1,61 @@
|
||||
// CLI command: provision install
|
||||
// Queue a discovered machine for OS installation via labd.
|
||||
|
||||
import { Command, Option } from "commander";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { Command, Option, InvalidArgumentError } from "commander";
|
||||
import { isValidOsId, SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY } from "@lab/shared";
|
||||
import type { VyosBundle, VyosInstallSpec, VyosVlanSpec } from "@lab/shared";
|
||||
import { getLabdClient } from "../api/config.js";
|
||||
|
||||
/**
|
||||
* Load one router's config out of a Pulumi-rendered bundle.
|
||||
*
|
||||
* The bundle is produced by `kubernetes-deployment` (npm run vyos:bundle) and
|
||||
* holds every router it manages, keyed by name. Selecting by hostname here is
|
||||
* what keeps bring-up and `pulumi up` describing the same box: labctl replays
|
||||
* the declared config rather than deriving its own.
|
||||
*/
|
||||
export function loadVyosBundle(path: string, hostname: string): VyosBundle {
|
||||
let parsed: { version?: number; routers?: Record<string, VyosBundle> };
|
||||
try {
|
||||
parsed = JSON.parse(readFileSync(path, "utf8"));
|
||||
} catch (e) {
|
||||
throw new InvalidArgumentError(`Cannot read VyOS bundle ${path}: ${(e as Error).message}`);
|
||||
}
|
||||
if (parsed.version !== 1) {
|
||||
throw new InvalidArgumentError(
|
||||
`VyOS bundle ${path} has version ${parsed.version ?? "<none>"}; this labctl understands 1`,
|
||||
);
|
||||
}
|
||||
const router = parsed.routers?.[hostname];
|
||||
if (router === undefined) {
|
||||
const known = Object.keys(parsed.routers ?? {}).join(", ") || "<none>";
|
||||
throw new InvalidArgumentError(
|
||||
`VyOS bundle ${path} has no entry for "${hostname}" (has: ${known})`,
|
||||
);
|
||||
}
|
||||
return router;
|
||||
}
|
||||
|
||||
/** Parse a repeated --vlan flag: "<id>:<cidr>[:<description>]". */
|
||||
export function parseVlan(value: string, previous: VyosVlanSpec[] = []): VyosVlanSpec[] {
|
||||
const parts = value.split(":");
|
||||
const id = Number(parts[0]);
|
||||
const address = parts[1] ?? "";
|
||||
// InvalidArgumentError makes commander print a clean message instead of
|
||||
// dumping a stack trace at the operator.
|
||||
if (!Number.isInteger(id) || id < 1 || id > 4094) {
|
||||
throw new InvalidArgumentError(`Invalid VLAN id in "${value}" (expected 1-4094)`);
|
||||
}
|
||||
if (!address.includes("/")) {
|
||||
throw new InvalidArgumentError(
|
||||
`Invalid VLAN address in "${value}" (expected CIDR, e.g. 10.0.10.1/24)`,
|
||||
);
|
||||
}
|
||||
const description = parts.slice(2).join(":");
|
||||
return [...previous, { id, address, ...(description ? { description } : {}) }];
|
||||
}
|
||||
|
||||
function roleTable(): string {
|
||||
const lines: string[] = ["", "Available roles:"];
|
||||
for (const r of ROLE_REGISTRY) {
|
||||
@@ -15,6 +66,38 @@ function roleTable(): string {
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/** Parse a repeated --vlan-vip flag: "<id>:<cidr>" — VRRP VIP for a --vlan entry. */
|
||||
export function parseVlanVip(
|
||||
value: string,
|
||||
previous: Record<number, string> = {},
|
||||
): Record<number, string> {
|
||||
const index = value.indexOf(":");
|
||||
const id = Number(index === -1 ? Number.NaN : value.slice(0, index));
|
||||
const cidr = index === -1 ? "" : value.slice(index + 1).trim();
|
||||
if (!Number.isInteger(id) || id < 1 || id > 4094 || !cidr.includes("/")) {
|
||||
throw new InvalidArgumentError(
|
||||
`Invalid VLAN VIP "${value}" (expected <id>:<cidr>, e.g. 3:192.168.3.254/24)`,
|
||||
);
|
||||
}
|
||||
return { ...previous, [id]: cidr };
|
||||
}
|
||||
|
||||
/** Parse a repeated --vyos-hwid flag: "<iface>=<mac>". */
|
||||
export function parseHwId(
|
||||
value: string,
|
||||
previous: Record<string, string> = {},
|
||||
): Record<string, string> {
|
||||
const index = value.indexOf("=");
|
||||
const iface = index === -1 ? "" : value.slice(0, index).trim();
|
||||
const mac = index === -1 ? "" : value.slice(index + 1).trim().toLowerCase();
|
||||
if (iface === "" || !/^([0-9a-f]{2}:){5}[0-9a-f]{2}$/.test(mac)) {
|
||||
throw new InvalidArgumentError(
|
||||
`Invalid hw-id "${value}" (expected <iface>=<mac>, e.g. eth2=64:62:66:25:96:47)`,
|
||||
);
|
||||
}
|
||||
return { ...previous, [iface]: mac };
|
||||
}
|
||||
|
||||
export function registerInstallCommand(parent: Command): void {
|
||||
parent
|
||||
.command("install <mac> <hostname>")
|
||||
@@ -24,10 +107,51 @@ export function registerInstallCommand(parent: Command): void {
|
||||
.addOption(new Option("--role <role>", "Machine role (see below)").choices([...SUPPORTED_ROLES]).default("worker"))
|
||||
.addOption(new Option("--os <os>", "Operating system").choices([...SUPPORTED_OS]).default("fedora-43"))
|
||||
.option("--disk <device>", "Target disk device (auto-detect if omitted)")
|
||||
.option("--vyos-mgmt <iface>", "VyOS: untagged interface the machine PXE boots from (default eth0)")
|
||||
.option("--vyos-mgmt-address <addr>", "VyOS: CIDR for the management interface, or 'dhcp' (default dhcp)")
|
||||
.option("--vyos-bond <ifaces>", "VyOS: comma-separated LACP bond members (must exclude the PXE NIC)")
|
||||
.option("--vyos-bond-address <cidr>", "VyOS: address on the untagged bond (trunk native VLAN)")
|
||||
.option("--vyos-bond-vrrp <cidr>", "VyOS: VRRP VIP floated on the untagged bond")
|
||||
.option("--vlan-vip <id:cidr>", "VyOS: VRRP VIP for a --vlan entry (repeatable)", parseVlanVip)
|
||||
.option("--vyos-vrrp-priority <n>", "VyOS: VRRP priority for all groups on this box (higher = master)")
|
||||
.option("--vyos-mgmt-vlan <id:cidr[:desc]>", "VyOS: tagged management VLAN on the PXE port")
|
||||
.option("--vlan <id:cidr[:desc]>", "VyOS: tagged VLAN sub-interface on the bond (repeatable)", parseVlan)
|
||||
.option("--vyos-password <password>", "VyOS: password for the 'vyos' user")
|
||||
.option("--vyos-hwid <iface=mac>", "VyOS: pin an interface name to a MAC via hw-id (repeatable)", parseHwId)
|
||||
.option("--vyos-fresh-config", "VyOS: on reinstall, overwrite the preserved config with the generated one")
|
||||
.option(
|
||||
"--vyos-bundle <path>",
|
||||
"VyOS: apply a Pulumi-rendered bundle verbatim (kubernetes-deployment/infra/vyos/vyos-bundle.json). " +
|
||||
"Replaces the derived --vyos-bond/--vlan/... config; secret values are left unset for `pulumi up`.",
|
||||
)
|
||||
.option(
|
||||
"--vyos-api-key <key>",
|
||||
"VyOS: enable the HTTP API with this key so Pulumi can manage the router from first boot",
|
||||
)
|
||||
.option(
|
||||
"--vyos-api-listen <addr>",
|
||||
"VyOS: address the HTTP API binds to (default: the static management address). " +
|
||||
"Required when management is DHCP; the API is never bound to all interfaces.",
|
||||
)
|
||||
.action(async (mac: string, hostname: string, opts: {
|
||||
role: string;
|
||||
os: string;
|
||||
disk?: string;
|
||||
vyosMgmt?: string;
|
||||
vyosMgmtAddress?: string;
|
||||
vyosBond?: string;
|
||||
vyosBondAddress?: string;
|
||||
vyosBondVrrp?: string;
|
||||
vlan?: VyosVlanSpec[];
|
||||
vlanVip?: Record<number, string>;
|
||||
vyosVrrpPriority?: string;
|
||||
vyosMgmtVlan?: string;
|
||||
vyosPassword?: string;
|
||||
vyosHwid?: Record<string, string>;
|
||||
vyosFreshConfig?: boolean;
|
||||
vyosBundle?: string;
|
||||
vyosApiKey?: string;
|
||||
vyosApiListen?: string;
|
||||
}) => {
|
||||
if (!isValidOsId(opts.os)) {
|
||||
console.error(`Unknown OS: ${opts.os}. Supported: ${SUPPORTED_OS.join(", ")}`);
|
||||
@@ -39,6 +163,89 @@ export function registerInstallCommand(parent: Command): void {
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const bondMembers = opts.vyosBond !== undefined && opts.vyosBond !== ""
|
||||
? opts.vyosBond.split(",").map((s) => s.trim()).filter((s) => s.length > 0)
|
||||
: [];
|
||||
|
||||
// Attach --vlan-vip entries to their --vlan definitions. A VIP for a VLAN
|
||||
// that was never defined is a typo that would otherwise vanish silently.
|
||||
const vips = opts.vlanVip ?? {};
|
||||
const vlans = (opts.vlan ?? []).map((v) =>
|
||||
vips[v.id] !== undefined ? { ...v, vrrp: vips[v.id] as string } : v,
|
||||
);
|
||||
for (const id of Object.keys(vips)) {
|
||||
if (!vlans.some((v) => String(v.id) === id)) {
|
||||
console.error(`--vlan-vip ${id}:... has no matching --vlan ${id}:... entry`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
const vrrpPriority = opts.vyosVrrpPriority !== undefined && opts.vyosVrrpPriority !== ""
|
||||
? Number(opts.vyosVrrpPriority)
|
||||
: undefined;
|
||||
if (vrrpPriority !== undefined
|
||||
&& (!Number.isInteger(vrrpPriority) || vrrpPriority < 1 || vrrpPriority > 255)) {
|
||||
console.error(`--vyos-vrrp-priority must be an integer 1-255 (got ${opts.vyosVrrpPriority})`);
|
||||
process.exit(1);
|
||||
}
|
||||
const vyos: VyosInstallSpec = {
|
||||
...(opts.vyosMgmt !== undefined && opts.vyosMgmt !== ""
|
||||
? { mgmtInterface: opts.vyosMgmt } : {}),
|
||||
...(opts.vyosMgmtAddress !== undefined && opts.vyosMgmtAddress !== ""
|
||||
? { mgmtAddress: opts.vyosMgmtAddress } : {}),
|
||||
...(bondMembers.length > 0 ? { bondMembers } : {}),
|
||||
...(opts.vyosBondAddress !== undefined && opts.vyosBondAddress !== ""
|
||||
? { bondAddress: opts.vyosBondAddress } : {}),
|
||||
...(opts.vyosBondVrrp !== undefined && opts.vyosBondVrrp !== ""
|
||||
? { bondVrrp: opts.vyosBondVrrp } : {}),
|
||||
...(vrrpPriority !== undefined ? { vrrpPriority } : {}),
|
||||
...(vlans.length > 0 ? { vlans } : {}),
|
||||
...(opts.vyosPassword !== undefined && opts.vyosPassword !== ""
|
||||
? { password: opts.vyosPassword } : {}),
|
||||
...(opts.vyosHwid !== undefined && Object.keys(opts.vyosHwid).length > 0
|
||||
? { hwIds: opts.vyosHwid } : {}),
|
||||
...(opts.vyosMgmtVlan !== undefined && opts.vyosMgmtVlan !== ""
|
||||
? { mgmtVlan: parseVlan(opts.vyosMgmtVlan)[0] as VyosVlanSpec } : {}),
|
||||
...(opts.vyosFreshConfig === true ? { freshConfig: true } : {}),
|
||||
...(opts.vyosBundle !== undefined && opts.vyosBundle !== ""
|
||||
? { bundle: loadVyosBundle(opts.vyosBundle, hostname) } : {}),
|
||||
...(opts.vyosApiKey !== undefined && opts.vyosApiKey !== ""
|
||||
? { apiKey: opts.vyosApiKey } : {}),
|
||||
...(opts.vyosApiListen !== undefined && opts.vyosApiListen !== ""
|
||||
? { apiListenAddress: opts.vyosApiListen } : {}),
|
||||
};
|
||||
const hasVyosOptions = Object.keys(vyos).length > 0;
|
||||
|
||||
// A bundle already describes the whole router. Accepting derived topology
|
||||
// flags alongside it would silently discard them (the bundle wins in
|
||||
// buildVyosConfigSpec), so say so rather than appear to honour both.
|
||||
if (vyos.bundle !== undefined) {
|
||||
const derived = ["mgmtInterface", "mgmtAddress", "bondMembers", "bondAddress",
|
||||
"bondVrrp", "vrrpPriority", "vlans", "mgmtVlan"] as const;
|
||||
const conflicting = derived.filter((k) => vyos[k] !== undefined);
|
||||
if (conflicting.length > 0) {
|
||||
console.error(
|
||||
`--vyos-bundle describes the whole router; these would be ignored: ${conflicting.join(", ")}`,
|
||||
);
|
||||
console.error("Remove them, or change the bundle in kubernetes-deployment and re-render.");
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
if (hasVyosOptions && !opts.os.startsWith("vyos")) {
|
||||
console.error(`VyOS options require --os vyos-rolling (got --os ${opts.os})`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Firmware PXE cannot run over LACP, so the NIC that boots the installer
|
||||
// must stay out of the bond — otherwise the next reinstall has no path in.
|
||||
const mgmt = vyos.mgmtInterface ?? "eth0";
|
||||
if (bondMembers.includes(mgmt)) {
|
||||
console.error(`--vyos-bond must not include the PXE/management interface "${mgmt}"`);
|
||||
console.error("PXE cannot boot over an LACP bond; keep that NIC unbonded.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await getLabdClient().installMachine({
|
||||
mac,
|
||||
@@ -46,11 +253,14 @@ export function registerInstallCommand(parent: Command): void {
|
||||
role: opts.role,
|
||||
os: opts.os,
|
||||
...(opts.disk ? { disk: opts.disk } : {}),
|
||||
...(hasVyosOptions ? { vyos } : {}),
|
||||
});
|
||||
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
console.log("");
|
||||
const osLabel = opts.os.startsWith("ubuntu") ? "Ubuntu" : "Fedora";
|
||||
const osLabel = opts.os.startsWith("ubuntu")
|
||||
? "Ubuntu"
|
||||
: opts.os.startsWith("vyos") ? "VyOS" : "Fedora";
|
||||
console.log(`Power on the machine to start ${osLabel} installation.`);
|
||||
|
||||
const roleInfo = ROLE_REGISTRY.find(r => r.name === opts.role);
|
||||
|
||||
@@ -38,7 +38,7 @@ export function registerLabcontrollerCommands(appCmd: Command): void {
|
||||
lcCmd
|
||||
.command("deploy <target>")
|
||||
.description("Deploy labcontroller stack to a k3s node")
|
||||
.option("--user <user>", "SSH user", "lab")
|
||||
.option("--user <user>", "SSH user", "root")
|
||||
.option("--crdb-replicas <n>", "CockroachDB replicas", "1")
|
||||
.action(async (target: string, opts: {
|
||||
user: string;
|
||||
@@ -193,7 +193,7 @@ export function registerLabcontrollerCommands(appCmd: Command): void {
|
||||
lcCmd
|
||||
.command("status [target]")
|
||||
.description("Check labcontroller deployment status (all hosts if no target)")
|
||||
.option("--user <user>", "SSH user", "lab")
|
||||
.option("--user <user>", "SSH user", "root")
|
||||
.action(async (target: string | undefined, opts: { user: string }) => {
|
||||
const sshKey = findSshKey();
|
||||
const sshOpts = sshKey ? { keyPath: sshKey } : {};
|
||||
|
||||
@@ -69,10 +69,10 @@ export function registerListCommand(parent: Command): void {
|
||||
const hostname = inst?.hostname ?? queued?.hostname ?? "-";
|
||||
const role = inst?.role ?? queued?.role ?? "-";
|
||||
const ip = inst?.ip ?? "-";
|
||||
const cpu = hw?.cpu_model ?? "-";
|
||||
const cores = hw?.cpu_cores != null ? String(hw.cpu_cores) : "-";
|
||||
const ram = hw?.memory_gb != null ? `${hw.memory_gb}GB` : "-";
|
||||
const product = hw?.product ?? "-";
|
||||
const cpu = hw?.cpu_model ?? inst?.cpu_model ?? "-";
|
||||
const cores = (hw?.cpu_cores ?? inst?.cpu_cores) != null ? String(hw?.cpu_cores ?? inst?.cpu_cores) : "-";
|
||||
const ram = (hw?.memory_gb ?? inst?.memory_gb) != null ? `${hw?.memory_gb ?? inst?.memory_gb}GB` : "-";
|
||||
const product = hw?.product ?? inst?.product ?? "-";
|
||||
|
||||
const color = statusColor(status);
|
||||
|
||||
|
||||
97
bastion/src/cli/src/commands/recheck.ts
Normal file
97
bastion/src/cli/src/commands/recheck.ts
Normal file
@@ -0,0 +1,97 @@
|
||||
// CLI command: provision recheck
|
||||
// SSH into all installed machines, collect hardware info, update bastion state.
|
||||
|
||||
import type { Command } from "commander";
|
||||
import { sshExec } from "@lab/modules";
|
||||
import { getLabdClient } from "../api/config.js";
|
||||
|
||||
const BOLD = "\x1b[1m";
|
||||
const GREEN = "\x1b[0;32m";
|
||||
const RED = "\x1b[0;31m";
|
||||
const DIM = "\x1b[2m";
|
||||
const RESET = "\x1b[0m";
|
||||
|
||||
const SSH_OPTS = { timeoutMs: 30_000 };
|
||||
|
||||
// Shell script that collects hardware info as JSON.
|
||||
// Kept simple — no Python, pure shell + awk.
|
||||
const HW_COLLECT_SCRIPT = [
|
||||
'P=$(cat /sys/class/dmi/id/product_name 2>/dev/null || echo unknown)',
|
||||
'B=$(cat /sys/class/dmi/id/board_name 2>/dev/null || echo unknown)',
|
||||
'S=$(cat /sys/class/dmi/id/product_serial 2>/dev/null || echo unknown)',
|
||||
'M=$(cat /sys/class/dmi/id/sys_vendor 2>/dev/null || echo unknown)',
|
||||
'C=$(grep -m1 "model name" /proc/cpuinfo 2>/dev/null | cut -d: -f2 | sed "s/^ //" || grep -m1 Model /proc/cpuinfo 2>/dev/null | cut -d: -f2 | sed "s/^ //" || echo unknown)',
|
||||
'N=$(grep -c "^processor" /proc/cpuinfo 2>/dev/null || echo 0)',
|
||||
'R=$(awk "/MemTotal/ {printf \\"%d\\", \\$2/1024/1024}" /proc/meminfo 2>/dev/null || echo 0)',
|
||||
'A=$(uname -m)',
|
||||
'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A"',
|
||||
].join("; ");
|
||||
|
||||
export function registerRecheckCommand(parent: Command): void {
|
||||
parent
|
||||
.command("recheck")
|
||||
.description("Refresh hardware info for all installed machines via SSH")
|
||||
.option("--user <user>", "SSH user", "root")
|
||||
.option("--target <hostname>", "Only recheck a specific machine (by hostname or MAC)")
|
||||
.action(async (opts: { user: string; target?: string }) => {
|
||||
const client = getLabdClient();
|
||||
let state;
|
||||
try {
|
||||
state = await client.getMachines();
|
||||
} catch (err) {
|
||||
console.error(`Cannot reach labd: ${err instanceof Error ? err.message : String(err)}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Build list of machines to check
|
||||
const targets: Array<{ mac: string; hostname: string; ip: string; sshUser: string }> = [];
|
||||
const userIsDefault = opts.user === "root";
|
||||
for (const [mac, info] of Object.entries(state.installed)) {
|
||||
if (!info.ip) continue;
|
||||
if (opts.target && info.hostname !== opts.target && mac !== opts.target) continue;
|
||||
// VyOS boxes only have the "vyos" login; honor an explicit --user.
|
||||
const sshUser = userIsDefault && (info.os ?? "").startsWith("vyos") ? "vyos" : opts.user;
|
||||
targets.push({ mac, hostname: info.hostname, ip: info.ip, sshUser });
|
||||
}
|
||||
|
||||
if (targets.length === 0) {
|
||||
console.log("No installed machines with IPs to check.");
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`\n${BOLD}Rechecking ${targets.length} machine(s)...${RESET}\n`);
|
||||
|
||||
let updated = 0;
|
||||
let failed = 0;
|
||||
|
||||
for (const { mac, hostname, ip, sshUser } of targets) {
|
||||
process.stdout.write(` ${hostname.padEnd(24)} ${DIM}(${ip})${RESET} `);
|
||||
|
||||
try {
|
||||
const t0 = Date.now();
|
||||
const result = await sshExec(ip, sshUser, HW_COLLECT_SCRIPT, SSH_OPTS);
|
||||
const elapsed = Date.now() - t0;
|
||||
if (result.exitCode !== 0) {
|
||||
console.log(`${RED}SSH failed (exit ${result.exitCode}, ${elapsed}ms)${RESET}`);
|
||||
if (result.stderr) console.log(` ${DIM}${result.stderr.substring(0, 200)}${RESET}`);
|
||||
console.log(`${RED}SSH failed (exit ${result.exitCode})${RESET}`);
|
||||
failed++;
|
||||
continue;
|
||||
}
|
||||
|
||||
const hwData = JSON.parse(result.stdout.trim());
|
||||
await client.discoverMachine({ mac, ...hwData });
|
||||
const cpu = hwData.cpu_model || "?";
|
||||
const cores = hwData.cpu_cores || "?";
|
||||
const mem = hwData.memory_gb || "?";
|
||||
console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${RESET}`);
|
||||
updated++;
|
||||
} catch (err) {
|
||||
console.log(`${RED}FAIL${RESET} ${DIM}${err instanceof Error ? err.message : String(err)}${RESET}`);
|
||||
failed++;
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n${BOLD}Done:${RESET} ${updated} updated, ${failed} failed\n`);
|
||||
});
|
||||
}
|
||||
@@ -24,12 +24,12 @@ function roleTable(): string {
|
||||
function resolveTarget(
|
||||
target: string,
|
||||
state: BastionState,
|
||||
): { mac: string; hostname: string; ip: string } | null {
|
||||
): { mac: string; hostname: string; ip: string; os?: string } | null {
|
||||
const normalized = target.toLowerCase().replace(/-/g, ":");
|
||||
|
||||
if (state.installed[normalized]) {
|
||||
const info = state.installed[normalized];
|
||||
return { mac: normalized, hostname: info.hostname, ip: info.ip };
|
||||
return { mac: normalized, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
|
||||
}
|
||||
|
||||
if (state.discovered[normalized]) {
|
||||
@@ -38,13 +38,13 @@ function resolveTarget(
|
||||
|
||||
for (const [mac, info] of Object.entries(state.installed)) {
|
||||
if (info.hostname === target || info.hostname.startsWith(target + ".")) {
|
||||
return { mac, hostname: info.hostname, ip: info.ip };
|
||||
return { mac, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
|
||||
}
|
||||
}
|
||||
|
||||
for (const [mac, info] of Object.entries(state.installed)) {
|
||||
if (info.ip === target) {
|
||||
return { mac, hostname: info.hostname, ip: info.ip };
|
||||
return { mac, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
|
||||
}
|
||||
}
|
||||
|
||||
@@ -60,10 +60,12 @@ export function registerReprovisionCommand(parent: Command): void {
|
||||
.addOption(new Option("--role <role>", "Machine role (see below)").choices([...SUPPORTED_ROLES]).default("worker"))
|
||||
.addOption(new Option("--os <os>", "Operating system").choices([...SUPPORTED_OS]).default("fedora-43"))
|
||||
.option("--disk <device>", "Target disk device (auto-detect if omitted)")
|
||||
.option("--user <user>", "SSH user for the reboot (default: vyos for VyOS machines, else current user)")
|
||||
.action(async (target: string, hostnameOverride: string | undefined, opts: {
|
||||
role: string;
|
||||
os: string;
|
||||
disk?: string;
|
||||
user?: string;
|
||||
}) => {
|
||||
if (!isValidOsId(opts.os)) {
|
||||
console.error(`Unknown OS: ${opts.os}. Supported: ${SUPPORTED_OS.join(", ")}`);
|
||||
@@ -123,7 +125,11 @@ export function registerReprovisionCommand(parent: Command): void {
|
||||
return;
|
||||
}
|
||||
|
||||
const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
|
||||
// SSH user: explicit flag > the machine's current OS (VyOS boxes only
|
||||
// have the "vyos" login) > the invoking user.
|
||||
const currentOsIsVyos = (resolved.os ?? "").startsWith("vyos");
|
||||
const adminUser = opts.user
|
||||
?? (currentOsIsVyos ? "vyos" : (process.env["SUDO_USER"] ?? process.env["USER"] ?? ""));
|
||||
const effectiveUser = adminUser === "root" ? "" : adminUser;
|
||||
|
||||
if (effectiveUser === "") {
|
||||
|
||||
@@ -11,7 +11,7 @@ export function registerStartCommand(parent: Command): void {
|
||||
.command("start")
|
||||
.description("Start the bastion server (HTTP + dnsmasq PXE)")
|
||||
.option("--port <port>", "HTTP port", "8080")
|
||||
.option("--dir <dir>", "Bastion data directory", "/tmp/lab-bastion")
|
||||
.option("--dir <dir>", "Bastion data directory", process.env["BASTION_DIR"] ?? "/tmp/lab-bastion")
|
||||
.option("--domain <domain>", "Internal domain for hostnames", "ad.itaz.eu")
|
||||
.option("--dhcp-mode <mode>", "DHCP mode: proxy or full", "proxy")
|
||||
.option("--fedora <version>", "Fedora version", "43")
|
||||
|
||||
@@ -8,7 +8,7 @@ export function registerStopCommand(parent: Command): void {
|
||||
parent
|
||||
.command("stop")
|
||||
.description("Stop a running bastion server")
|
||||
.option("--dir <dir>", "Bastion data directory", "/tmp/lab-bastion")
|
||||
.option("--dir <dir>", "Bastion data directory", process.env["BASTION_DIR"] ?? "/tmp/lab-bastion")
|
||||
.action((opts: { dir: string }) => {
|
||||
const pidFile = `${opts.dir}/bastion.pid`;
|
||||
|
||||
|
||||
@@ -20,6 +20,7 @@ import { registerRegisterCommand } from "./commands/register.js";
|
||||
import { registerAsahiCommand } from "./commands/asahi.js";
|
||||
import { registerLogsCommand } from "./commands/logs.js";
|
||||
import { registerMakeIsoCommand } from "./commands/makeiso.js";
|
||||
import { registerRecheckCommand } from "./commands/recheck.js";
|
||||
import { registerConfigCommand } from "./commands/config.js";
|
||||
import { registerLoginCommand } from "./commands/login.js";
|
||||
import { registerDoctorCommand } from "./commands/doctor.js";
|
||||
@@ -104,6 +105,7 @@ export function createProgram(): Command {
|
||||
registerAsahiCommand(provisionCmd);
|
||||
registerLogsCommand(provisionCmd);
|
||||
registerMakeIsoCommand(provisionCmd);
|
||||
registerRecheckCommand(provisionCmd);
|
||||
|
||||
// config list/get/set/path
|
||||
registerConfigCommand(program);
|
||||
|
||||
35
bastion/src/cli/tests/install-vyos.test.ts
Normal file
35
bastion/src/cli/tests/install-vyos.test.ts
Normal file
@@ -0,0 +1,35 @@
|
||||
// Tests for VyOS install option parsing.
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { parseVlan } from "../src/commands/install.js";
|
||||
|
||||
describe("parseVlan", () => {
|
||||
it("parses id and CIDR", () => {
|
||||
expect(parseVlan("10:10.0.10.1/24")).toEqual([{ id: 10, address: "10.0.10.1/24" }]);
|
||||
});
|
||||
|
||||
it("accumulates across repeated flags", () => {
|
||||
const first = parseVlan("10:10.0.10.1/24");
|
||||
const both = parseVlan("20:10.0.20.1/24", first);
|
||||
expect(both).toHaveLength(2);
|
||||
expect(both[1]).toEqual({ id: 20, address: "10.0.20.1/24" });
|
||||
});
|
||||
|
||||
it("keeps a description, including one containing colons", () => {
|
||||
expect(parseVlan("30:10.0.30.1/24:mgmt:secondary")).toEqual([
|
||||
{ id: 30, address: "10.0.30.1/24", description: "mgmt:secondary" },
|
||||
]);
|
||||
});
|
||||
|
||||
it("rejects an address that is not CIDR", () => {
|
||||
// A bare address would produce a VyOS config that fails to commit on first
|
||||
// boot, long after the operator has stopped watching.
|
||||
expect(() => parseVlan("10:10.0.10.1")).toThrow(/CIDR/);
|
||||
});
|
||||
|
||||
it("rejects out-of-range and non-numeric VLAN ids", () => {
|
||||
expect(() => parseVlan("0:10.0.10.1/24")).toThrow(/1-4094/);
|
||||
expect(() => parseVlan("4095:10.0.10.1/24")).toThrow(/1-4094/);
|
||||
expect(() => parseVlan("abc:10.0.10.1/24")).toThrow(/1-4094/);
|
||||
});
|
||||
});
|
||||
23
bastion/src/core/package.json
Normal file
23
bastion/src/core/package.json
Normal file
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"name": "@lab/core",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
"types": "./dist/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"import": "./dist/index.js",
|
||||
"types": "./dist/index.d.ts"
|
||||
}
|
||||
},
|
||||
"scripts": {
|
||||
"build": "tsc --build",
|
||||
"clean": "rimraf dist",
|
||||
"test": "vitest",
|
||||
"test:run": "vitest run"
|
||||
},
|
||||
"dependencies": {
|
||||
"@pulumi/pulumi": "^3.0.0"
|
||||
}
|
||||
}
|
||||
75
bastion/src/core/src/audit.ts
Normal file
75
bastion/src/core/src/audit.ts
Normal file
@@ -0,0 +1,75 @@
|
||||
// Audit event types for the labctl platform.
|
||||
// Every mutation is tracked with correlation IDs for causal chains.
|
||||
|
||||
export type AuditEventKind =
|
||||
| "resource_created"
|
||||
| "resource_updated"
|
||||
| "resource_deleted"
|
||||
| "resource_state_change"
|
||||
| "plan_generated"
|
||||
| "apply_started"
|
||||
| "apply_step"
|
||||
| "apply_completed"
|
||||
| "driver_translate"
|
||||
| "driver_execute"
|
||||
| "driver_error"
|
||||
| "fleet_discovery"
|
||||
| "fleet_classification"
|
||||
| "fleet_approval"
|
||||
| "fleet_auto_approve"
|
||||
| "pipeline_started"
|
||||
| "pipeline_step_started"
|
||||
| "pipeline_step_completed"
|
||||
| "pipeline_completed"
|
||||
| "deploy_started"
|
||||
| "deploy_completed"
|
||||
| "deploy_failed"
|
||||
| "drift_detected"
|
||||
| "drift_corrected"
|
||||
| "sync_triggered"
|
||||
| "sync_completed"
|
||||
| "auth_login"
|
||||
| "auth_logout"
|
||||
| "auth_bootstrap"
|
||||
| "rbac_decision"
|
||||
| "impersonation"
|
||||
| "server_started"
|
||||
| "controller_started"
|
||||
| "agent_connected"
|
||||
| "agent_disconnected"
|
||||
| "bastion_registered";
|
||||
|
||||
export type AuditSource =
|
||||
| "cli"
|
||||
| "labd"
|
||||
| "agent"
|
||||
| "driver"
|
||||
| "fleet-controller"
|
||||
| "sync-controller";
|
||||
|
||||
export type AuditResult = "success" | "failure" | "denied" | "skipped";
|
||||
|
||||
export interface AuditEvent {
|
||||
id: string;
|
||||
timestamp: Date;
|
||||
eventKind: AuditEventKind;
|
||||
source: AuditSource;
|
||||
verified: boolean;
|
||||
|
||||
userId?: string;
|
||||
userName?: string;
|
||||
sessionId?: string;
|
||||
environmentName?: string;
|
||||
accountName?: string;
|
||||
|
||||
resourceKind?: string;
|
||||
resourceName?: string;
|
||||
|
||||
correlationId: string;
|
||||
parentEventId?: string;
|
||||
|
||||
details: Record<string, unknown>;
|
||||
result: AuditResult;
|
||||
error?: string;
|
||||
durationMs?: number;
|
||||
}
|
||||
50
bastion/src/core/src/auth.ts
Normal file
50
bastion/src/core/src/auth.ts
Normal file
@@ -0,0 +1,50 @@
|
||||
// Auth types for the labctl platform.
|
||||
// Bearer token auth for CLI/SDK. mTLS stays for agent/bastion.
|
||||
|
||||
export type UserRole = "USER" | "ADMIN";
|
||||
|
||||
export interface User {
|
||||
id: string;
|
||||
email: string;
|
||||
name?: string;
|
||||
role: UserRole;
|
||||
createdAt: Date;
|
||||
}
|
||||
|
||||
export interface Session {
|
||||
id: string;
|
||||
userId: string;
|
||||
token: string;
|
||||
expiresAt: Date;
|
||||
createdAt: Date;
|
||||
}
|
||||
|
||||
export interface Group {
|
||||
id: string;
|
||||
name: string;
|
||||
description?: string;
|
||||
}
|
||||
|
||||
export type SubjectKind = "User" | "Group" | "ServiceAccount";
|
||||
|
||||
export interface RoleBinding {
|
||||
role: "view" | "edit" | "create" | "delete" | "run" | "admin";
|
||||
resource: string;
|
||||
name?: string;
|
||||
environment?: string;
|
||||
action?: string;
|
||||
}
|
||||
|
||||
export interface RbacSubject {
|
||||
kind: SubjectKind;
|
||||
name: string;
|
||||
}
|
||||
|
||||
export interface RbacDefinition {
|
||||
id: string;
|
||||
name: string;
|
||||
subjects: RbacSubject[];
|
||||
roleBindings: RoleBinding[];
|
||||
createdAt: Date;
|
||||
updatedAt: Date;
|
||||
}
|
||||
24
bastion/src/core/src/environment.ts
Normal file
24
bastion/src/core/src/environment.ts
Normal file
@@ -0,0 +1,24 @@
|
||||
// Environment and Account types.
|
||||
// An Environment is a logical boundary (production, staging, dev).
|
||||
// An Account is a configured driver instance with credentials.
|
||||
|
||||
export interface Environment {
|
||||
id: string;
|
||||
name: string;
|
||||
status: "active" | "archived";
|
||||
createdAt: Date;
|
||||
}
|
||||
|
||||
export interface Account {
|
||||
id: string;
|
||||
name: string;
|
||||
driver: string;
|
||||
config: Record<string, unknown>;
|
||||
createdAt: Date;
|
||||
}
|
||||
|
||||
export interface Binding {
|
||||
id: string;
|
||||
environmentId: string;
|
||||
accountId: string;
|
||||
}
|
||||
9
bastion/src/core/src/index.ts
Normal file
9
bastion/src/core/src/index.ts
Normal file
@@ -0,0 +1,9 @@
|
||||
// @lab/core — foundation types for the labctl platform.
|
||||
// Phase 1 stub: resource types, auth types, audit types, Output<T>.
|
||||
// Phase 5 adds: CompositeResource, evaluator integration, full SDK.
|
||||
|
||||
export * from "./resource.js";
|
||||
export * from "./environment.js";
|
||||
export * from "./audit.js";
|
||||
export * from "./auth.js";
|
||||
export { Output, output, all, interpolate, secret } from "./output.js";
|
||||
5
bastion/src/core/src/output.ts
Normal file
5
bastion/src/core/src/output.ts
Normal file
@@ -0,0 +1,5 @@
|
||||
// Re-export Pulumi's Output<T> type for use across the platform.
|
||||
// Cloud drivers use this for future values (endpoints, IPs, kubeconfigs).
|
||||
// Phase 1: type re-export only. Phase 5 adds full evaluator integration.
|
||||
|
||||
export { Output, output, all, interpolate, secret } from "@pulumi/pulumi";
|
||||
83
bastion/src/core/src/resource.ts
Normal file
83
bastion/src/core/src/resource.ts
Normal file
@@ -0,0 +1,83 @@
|
||||
// Core resource types for the labctl platform.
|
||||
// Every managed thing (Server, Database, App, Cluster) is a Resource.
|
||||
|
||||
export type ResourceOrigin = "file" | "cli" | "fleet" | "imported";
|
||||
export type ResourceManagedBy = "gitops" | "manual" | "auto";
|
||||
|
||||
export type ResourceStatus =
|
||||
| "pending"
|
||||
| "creating"
|
||||
| "ready"
|
||||
| "updating"
|
||||
| "deleting"
|
||||
| "error"
|
||||
| "unknown";
|
||||
|
||||
export interface ResourceMetadata {
|
||||
kind: string;
|
||||
name: string;
|
||||
environmentId: string;
|
||||
accountId: string;
|
||||
origin: ResourceOrigin;
|
||||
managedBy: ResourceManagedBy;
|
||||
sourceRef?: string;
|
||||
}
|
||||
|
||||
export interface ResourceState {
|
||||
status: ResourceStatus;
|
||||
message?: string;
|
||||
lastReconciled?: Date;
|
||||
platformRef?: string;
|
||||
}
|
||||
|
||||
export interface Resource<TSpec = Record<string, unknown>> {
|
||||
id: string;
|
||||
metadata: ResourceMetadata;
|
||||
desiredSpec: TSpec;
|
||||
actualSpec?: TSpec;
|
||||
state: ResourceState;
|
||||
createdAt: Date;
|
||||
updatedAt: Date;
|
||||
}
|
||||
|
||||
// Well-known resource kinds. Drivers register additional kinds.
|
||||
export const RESOURCE_KINDS = {
|
||||
SERVER: "server",
|
||||
DATABASE: "database",
|
||||
CACHE: "cache",
|
||||
CLUSTER: "cluster",
|
||||
APP: "app",
|
||||
SERVICE: "service",
|
||||
CRONJOB: "cronjob",
|
||||
NETWORK: "network",
|
||||
LOADBALANCER: "loadbalancer",
|
||||
DNSZONE: "dnszone",
|
||||
CERTIFICATE: "certificate",
|
||||
OBJECTSTORE: "objectstore",
|
||||
QUEUE: "queue",
|
||||
SECRET: "secret",
|
||||
FLEET: "fleet",
|
||||
} as const;
|
||||
|
||||
export type ResourceKind = (typeof RESOURCE_KINDS)[keyof typeof RESOURCE_KINDS];
|
||||
|
||||
// Resource aliases for CLI (kubectl-style shortnames)
|
||||
export const RESOURCE_ALIASES: Record<string, string> = {
|
||||
srv: "server",
|
||||
db: "database",
|
||||
cl: "cluster",
|
||||
svc: "service",
|
||||
cj: "cronjob",
|
||||
lb: "loadbalancer",
|
||||
dns: "dnszone",
|
||||
cert: "certificate",
|
||||
os: "objectstore",
|
||||
mq: "queue",
|
||||
sec: "secret",
|
||||
fl: "fleet",
|
||||
};
|
||||
|
||||
export function resolveResourceKind(input: string): string {
|
||||
const lower = input.toLowerCase();
|
||||
return RESOURCE_ALIASES[lower] ?? lower;
|
||||
}
|
||||
8
bastion/src/core/tsconfig.json
Normal file
8
bastion/src/core/tsconfig.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "dist"
|
||||
},
|
||||
"include": ["src/**/*.ts"]
|
||||
}
|
||||
@@ -26,8 +26,10 @@
|
||||
"dependencies": {
|
||||
"@fastify/rate-limit": "^10.3.0",
|
||||
"@fastify/websocket": "^11.0.2",
|
||||
"@lab/core": "workspace:^",
|
||||
"@lab/shared": "workspace:*",
|
||||
"@prisma/client": "^6.9.0",
|
||||
"bcryptjs": "^3.0.3",
|
||||
"fastify": "^5.3.3",
|
||||
"winston": "^3.17.0",
|
||||
"ws": "^8.19.0",
|
||||
@@ -37,6 +39,7 @@
|
||||
"seed": "tsx prisma/seed.ts"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bcryptjs": "^3.0.0",
|
||||
"@types/node": "^22.14.1",
|
||||
"@types/ws": "^8.18.1",
|
||||
"prisma": "^6.9.0",
|
||||
|
||||
@@ -7,6 +7,225 @@ datasource db {
|
||||
url = env("DATABASE_URL")
|
||||
}
|
||||
|
||||
// ── Auth (mcpctl pattern: email/password + bearer token sessions) ──
|
||||
|
||||
model User {
|
||||
id String @id @default(cuid())
|
||||
email String @unique
|
||||
password String // bcrypt
|
||||
name String?
|
||||
role UserRole @default(USER)
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
|
||||
sessions Session[]
|
||||
auditLogs AuditEvent[]
|
||||
groups GroupMember[]
|
||||
}
|
||||
|
||||
enum UserRole {
|
||||
USER
|
||||
ADMIN
|
||||
}
|
||||
|
||||
model Session {
|
||||
id String @id @default(cuid())
|
||||
userId String
|
||||
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
|
||||
token String @unique
|
||||
expiresAt DateTime
|
||||
createdAt DateTime @default(now())
|
||||
|
||||
@@index([userId])
|
||||
@@index([token])
|
||||
}
|
||||
|
||||
model Group {
|
||||
id String @id @default(cuid())
|
||||
name String @unique
|
||||
description String?
|
||||
createdAt DateTime @default(now())
|
||||
members GroupMember[]
|
||||
}
|
||||
|
||||
model GroupMember {
|
||||
id String @id @default(cuid())
|
||||
groupId String
|
||||
group Group @relation(fields: [groupId], references: [id], onDelete: Cascade)
|
||||
userId String
|
||||
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
|
||||
|
||||
@@unique([groupId, userId])
|
||||
}
|
||||
|
||||
model ServiceAccount {
|
||||
id String @id @default(cuid())
|
||||
name String @unique
|
||||
token String @unique
|
||||
createdAt DateTime @default(now())
|
||||
}
|
||||
|
||||
// ── RBAC (mcpctl pattern: named definitions with JSON subjects/bindings) ──
|
||||
|
||||
model RbacDefinition {
|
||||
id String @id @default(cuid())
|
||||
name String @unique
|
||||
subjects Json // [{kind: "User"|"Group"|"ServiceAccount", name: string}]
|
||||
roleBindings Json // [{role, resource, name?, environment?, action?}]
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
}
|
||||
|
||||
// ── Audit (mcpctl pattern: fire-and-forget with correlation IDs) ──
|
||||
|
||||
model AuditEvent {
|
||||
id String @id @default(cuid())
|
||||
timestamp DateTime @default(now())
|
||||
eventKind String
|
||||
source String // cli | labd | agent | driver | fleet-controller | sync-controller
|
||||
verified Boolean @default(false)
|
||||
|
||||
userId String?
|
||||
user User? @relation(fields: [userId], references: [id])
|
||||
userName String?
|
||||
sessionId String?
|
||||
environmentName String?
|
||||
accountName String?
|
||||
|
||||
resourceKind String?
|
||||
resourceName String?
|
||||
|
||||
correlationId String
|
||||
parentEventId String?
|
||||
|
||||
details Json @default("{}")
|
||||
result String // success | failure | denied | skipped
|
||||
error String?
|
||||
durationMs Int?
|
||||
|
||||
@@index([correlationId])
|
||||
@@index([eventKind, timestamp])
|
||||
@@index([environmentName, timestamp])
|
||||
@@index([resourceKind, resourceName])
|
||||
@@index([userId, timestamp])
|
||||
}
|
||||
|
||||
// ── Core infrastructure ──
|
||||
|
||||
model Environment {
|
||||
id String @id @default(cuid())
|
||||
name String @unique
|
||||
status String @default("active") // active | archived
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
|
||||
bindings Binding[]
|
||||
resources Resource[]
|
||||
}
|
||||
|
||||
model Account {
|
||||
id String @id @default(cuid())
|
||||
name String @unique
|
||||
driver String // baremetal-pxe | aws | gcp | kubernetes | ovh
|
||||
config Json @default("{}")
|
||||
// Credentials stored in Infisical, referenced by secretPath
|
||||
secretPath String?
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
|
||||
bindings Binding[]
|
||||
resources Resource[]
|
||||
}
|
||||
|
||||
model Binding {
|
||||
id String @id @default(cuid())
|
||||
environmentId String
|
||||
environment Environment @relation(fields: [environmentId], references: [id], onDelete: Cascade)
|
||||
accountId String
|
||||
account Account @relation(fields: [accountId], references: [id], onDelete: Cascade)
|
||||
|
||||
@@unique([environmentId, accountId])
|
||||
}
|
||||
|
||||
model Resource {
|
||||
id String @id @default(cuid())
|
||||
kind String
|
||||
name String
|
||||
environmentId String
|
||||
environment Environment @relation(fields: [environmentId], references: [id])
|
||||
accountId String
|
||||
account Account @relation(fields: [accountId], references: [id])
|
||||
origin String @default("cli") // file | cli | fleet | imported
|
||||
managedBy String @default("manual") // gitops | manual | auto
|
||||
sourceRef String?
|
||||
desiredSpec Json @default("{}")
|
||||
actualSpec Json?
|
||||
platformRef String?
|
||||
status String @default("pending") // pending | creating | ready | updating | deleting | error
|
||||
statusMessage String?
|
||||
lastReconciled DateTime?
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
|
||||
@@unique([kind, name, environmentId])
|
||||
@@index([environmentId])
|
||||
@@index([accountId])
|
||||
@@index([kind, status])
|
||||
}
|
||||
|
||||
model Secret {
|
||||
id String @id @default(cuid())
|
||||
name String @unique
|
||||
// Encrypted data — application-layer encryption as fallback if Infisical unavailable
|
||||
data Json @default("{}")
|
||||
version Int @default(1)
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
}
|
||||
|
||||
// ── Fleet ──
|
||||
|
||||
model Fleet {
|
||||
id String @id @default(cuid())
|
||||
name String
|
||||
environmentId String
|
||||
accountId String
|
||||
selector Json // fact-matching rules
|
||||
onboardPipeline Json // step definitions
|
||||
offboardPipeline Json?
|
||||
approvalConfig Json?
|
||||
status String @default("active")
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
|
||||
members FleetMember[]
|
||||
}
|
||||
|
||||
model FleetMember {
|
||||
id String @id @default(cuid())
|
||||
fleetId String
|
||||
fleet Fleet @relation(fields: [fleetId], references: [id], onDelete: Cascade)
|
||||
serverId String
|
||||
status String // discovered | pending | onboarding | active | offboarding | removed
|
||||
joinedAt DateTime @default(now())
|
||||
|
||||
@@index([fleetId])
|
||||
}
|
||||
|
||||
// ── Git sources (for sync controller) ──
|
||||
|
||||
model GitSource {
|
||||
id String @id @default(cuid())
|
||||
name String @unique
|
||||
repo String
|
||||
branch String @default("main")
|
||||
path String @default("environments/")
|
||||
lastSync DateTime?
|
||||
createdAt DateTime @default(now())
|
||||
}
|
||||
|
||||
// ── Existing v1.0 models (kept for bastion/agent compatibility) ──
|
||||
|
||||
model Server {
|
||||
id String @id @default(uuid())
|
||||
hostname String @unique
|
||||
@@ -17,13 +236,12 @@ model Server {
|
||||
labels Json @default("{}")
|
||||
ip String?
|
||||
agentVersion String?
|
||||
status String @default("unknown") // unknown, online, offline, provisioning
|
||||
status String @default("unknown")
|
||||
lastHeartbeat DateTime?
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
|
||||
agent Agent?
|
||||
auditLogs AuditLog[]
|
||||
}
|
||||
|
||||
model Agent {
|
||||
@@ -33,112 +251,29 @@ model Agent {
|
||||
certificatePem String?
|
||||
enrolledAt DateTime @default(now())
|
||||
lastSeen DateTime?
|
||||
facts Json? // hardware facts reported by agent
|
||||
|
||||
@@index([serverId])
|
||||
}
|
||||
|
||||
model User {
|
||||
id String @id @default(uuid())
|
||||
username String @unique
|
||||
displayName String?
|
||||
certFingerprint String? @unique
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
|
||||
roleBindings UserRole[]
|
||||
auditLogs AuditLog[]
|
||||
}
|
||||
|
||||
model Role {
|
||||
id String @id @default(uuid())
|
||||
name String @unique
|
||||
description String?
|
||||
createdAt DateTime @default(now())
|
||||
|
||||
permissions Permission[]
|
||||
userBindings UserRole[]
|
||||
}
|
||||
|
||||
model Permission {
|
||||
id String @id @default(uuid())
|
||||
roleId String
|
||||
role Role @relation(fields: [roleId], references: [id], onDelete: Cascade)
|
||||
type String @default("allow") // allow or deny
|
||||
action String // read, exec, apply, destroy, manage, admin, kubectl, *
|
||||
cloud String @default("*")
|
||||
environment String @default("*")
|
||||
server String @default("*")
|
||||
|
||||
@@index([roleId])
|
||||
}
|
||||
|
||||
model UserRole {
|
||||
id String @id @default(uuid())
|
||||
userId String
|
||||
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
|
||||
roleId String
|
||||
role Role @relation(fields: [roleId], references: [id], onDelete: Cascade)
|
||||
|
||||
@@unique([userId, roleId])
|
||||
@@index([userId])
|
||||
@@index([roleId])
|
||||
}
|
||||
|
||||
model JoinToken {
|
||||
id String @id @default(uuid())
|
||||
token String @unique
|
||||
type String @default("one-time") // one-time or reusable
|
||||
type String @default("one-time")
|
||||
label String?
|
||||
usedBy String? // server hostname that used it
|
||||
usedBy String?
|
||||
usedAt DateTime?
|
||||
revokedAt DateTime?
|
||||
createdAt DateTime @default(now())
|
||||
expiresAt DateTime?
|
||||
}
|
||||
|
||||
model AuditLog {
|
||||
id String @id @default(uuid())
|
||||
userId String?
|
||||
user User? @relation(fields: [userId], references: [id])
|
||||
serverId String?
|
||||
server Server? @relation(fields: [serverId], references: [id])
|
||||
sessionId String?
|
||||
action String // exec, kubectl, apply, login, rbac-denied, etc.
|
||||
resourceType String? // server, cluster, role, app, etc.
|
||||
resourceName String?
|
||||
args String? // sanitized command args
|
||||
result String @default("success") // success, denied, error
|
||||
durationMs Int?
|
||||
sourceIp String?
|
||||
timestamp DateTime @default(now())
|
||||
|
||||
@@index([userId])
|
||||
@@index([serverId])
|
||||
@@index([sessionId])
|
||||
@@index([timestamp])
|
||||
@@index([action])
|
||||
}
|
||||
|
||||
model PulumiRun {
|
||||
id String @id @default(uuid())
|
||||
userId String
|
||||
stackName String
|
||||
action String // up, preview, destroy
|
||||
status String @default("pending") // pending, running, succeeded, failed
|
||||
output String?
|
||||
startedAt DateTime @default(now())
|
||||
completedAt DateTime?
|
||||
|
||||
@@index([userId])
|
||||
@@index([stackName])
|
||||
}
|
||||
|
||||
model Bastion {
|
||||
id String @id @default(uuid())
|
||||
hostname String @unique
|
||||
network String
|
||||
serverIp String
|
||||
status String @default("offline") // online, offline
|
||||
status String @default("offline")
|
||||
lastHeartbeat DateTime?
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
@@ -149,7 +284,7 @@ model Cluster {
|
||||
name String @unique
|
||||
cloud String @default("baremetal")
|
||||
environment String @default("default")
|
||||
kubeconfigEnc String? // encrypted kubeconfig
|
||||
kubeconfigEnc String?
|
||||
labels Json @default("{}")
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
|
||||
65
bastion/src/labd/src/middleware/bearer-auth.ts
Normal file
65
bastion/src/labd/src/middleware/bearer-auth.ts
Normal file
@@ -0,0 +1,65 @@
|
||||
// Bearer token auth middleware for Fastify.
|
||||
// Validates Authorization header, resolves user identity, attaches to request.
|
||||
|
||||
import type { FastifyRequest, FastifyReply } from "fastify";
|
||||
import type { AuthService } from "../services/auth.js";
|
||||
|
||||
declare module "fastify" {
|
||||
interface FastifyRequest {
|
||||
userId?: string;
|
||||
userEmail?: string;
|
||||
userRole?: string;
|
||||
}
|
||||
}
|
||||
|
||||
// Paths that don't require authentication
|
||||
const PUBLIC_PATHS = new Set([
|
||||
"/health",
|
||||
"/api/auth/login",
|
||||
"/ws/bastion",
|
||||
"/ws/agent",
|
||||
"/api/auth/enroll",
|
||||
]);
|
||||
|
||||
export function createBearerAuthMiddleware(authService: AuthService) {
|
||||
return async function bearerAuth(
|
||||
request: FastifyRequest,
|
||||
reply: FastifyReply,
|
||||
): Promise<void> {
|
||||
// Skip auth for public paths
|
||||
if (PUBLIC_PATHS.has(request.url.split("?")[0] ?? "")) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Skip auth for WebSocket upgrade requests (handled by their own auth)
|
||||
if (request.headers.upgrade === "websocket") {
|
||||
return;
|
||||
}
|
||||
|
||||
const authHeader = request.headers.authorization;
|
||||
if (!authHeader) {
|
||||
void reply.code(401).send({ error: "Authorization header required" });
|
||||
return;
|
||||
}
|
||||
|
||||
if (!authHeader.startsWith("Bearer ")) {
|
||||
void reply.code(401).send({ error: "Invalid authorization format, expected: Bearer <token>" });
|
||||
return;
|
||||
}
|
||||
|
||||
const token = authHeader.slice(7);
|
||||
if (token.length === 0) {
|
||||
void reply.code(401).send({ error: "Empty bearer token" });
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const identity = await authService.validateToken(token);
|
||||
request.userId = identity.userId;
|
||||
request.userEmail = identity.email;
|
||||
request.userRole = identity.role;
|
||||
} catch {
|
||||
void reply.code(401).send({ error: "Invalid or expired token. Run: labctl login" });
|
||||
}
|
||||
};
|
||||
}
|
||||
@@ -10,6 +10,7 @@ import type { FastifyInstance } from "fastify";
|
||||
import type { DbClient } from "../server.js";
|
||||
import { bastionRegistry } from "../services/bastion-registry.js";
|
||||
import { generateRequestId } from "@lab/shared";
|
||||
import type { VyosInstallSpec } from "@lab/shared";
|
||||
|
||||
const COMMAND_TIMEOUT_MS = 15_000;
|
||||
|
||||
@@ -84,7 +85,6 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
|
||||
app.get("/api/machines", async () => {
|
||||
const live = bastionRegistry.getAggregatedState();
|
||||
|
||||
// Merge DB records for machines not currently in any bastion's live state
|
||||
try {
|
||||
const dbServers = (await db.server.findMany({})) as Array<{
|
||||
mac: string | null; hostname: string; role: string; ip: string | null;
|
||||
@@ -93,9 +93,49 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
|
||||
for (const s of dbServers) {
|
||||
if (!s.mac) continue;
|
||||
const mac = s.mac.toLowerCase();
|
||||
// Only add from DB if not already in live state
|
||||
|
||||
// DB knows this machine has been installed at some point if it has a real
|
||||
// hostname+role (not just product-name-as-hostname and role="unknown").
|
||||
// Status alone is unreliable: a rediscovery can re-set it without erasing the
|
||||
// install identity. If the bastion restarted and lost its installed map, the
|
||||
// machine will only show up in live.discovered — promote it here so the CLI
|
||||
// still sees hostname/role/IP.
|
||||
const dbKnowsInstalled =
|
||||
s.role !== "unknown" && s.role !== "" &&
|
||||
s.hostname !== "" && s.hostname !== s.mac;
|
||||
|
||||
if (dbKnowsInstalled && !(mac in live.installed) && !(mac in live.install_queue)) {
|
||||
const hw = live.discovered[mac];
|
||||
live.installed[mac] = {
|
||||
hostname: s.hostname,
|
||||
role: s.role,
|
||||
ip: s.ip ?? "",
|
||||
installed_at: "",
|
||||
bastionId: hw?.bastionId ?? "db",
|
||||
...(hw ? {
|
||||
product: hw.product,
|
||||
manufacturer: hw.manufacturer,
|
||||
cpu_model: hw.cpu_model,
|
||||
cpu_cores: hw.cpu_cores,
|
||||
memory_gb: hw.memory_gb,
|
||||
arch: hw.arch,
|
||||
} : {}),
|
||||
};
|
||||
delete live.discovered[mac];
|
||||
continue;
|
||||
}
|
||||
|
||||
// Unknown-to-live MAC: fall back to whatever the DB says.
|
||||
if (!(mac in live.discovered) && !(mac in live.install_queue) && !(mac in live.installed)) {
|
||||
if (s.status === "discovered") {
|
||||
if (s.status === "online" || s.status === "offline") {
|
||||
live.installed[mac] = {
|
||||
hostname: s.hostname,
|
||||
role: s.role,
|
||||
ip: s.ip ?? "",
|
||||
installed_at: "",
|
||||
bastionId: "db",
|
||||
};
|
||||
} else {
|
||||
live.discovered[mac] = {
|
||||
mac,
|
||||
product: String(s.labels?.product ?? "unknown"),
|
||||
@@ -112,14 +152,6 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
|
||||
last_seen: "",
|
||||
bastionId: "db",
|
||||
};
|
||||
} else if (s.status === "online" || s.status === "offline") {
|
||||
live.installed[mac] = {
|
||||
hostname: s.hostname,
|
||||
role: s.role,
|
||||
ip: s.ip ?? "",
|
||||
installed_at: "",
|
||||
bastionId: "db",
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -132,9 +164,9 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
|
||||
|
||||
// Queue install — route to correct bastion by MAC
|
||||
app.post<{
|
||||
Body: { mac?: string; hostname?: string; disk?: string; role?: string; os?: string };
|
||||
Body: { mac?: string; hostname?: string; disk?: string; role?: string; os?: string; vyos?: VyosInstallSpec };
|
||||
}>("/api/machines/install", async (request, reply) => {
|
||||
const { mac, hostname, disk, role, os } = request.body ?? {};
|
||||
const { mac, hostname, disk, role, os, vyos } = request.body ?? {};
|
||||
if (!mac || !hostname) {
|
||||
return reply.code(400).send({ error: "mac and hostname are required" });
|
||||
}
|
||||
@@ -152,6 +184,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
|
||||
const result = await sendCommand(all[0]!.bastionId, {
|
||||
type: "command-install",
|
||||
mac, hostname, disk: disk ?? "", role: role ?? "infra", os: os ?? "fedora-43",
|
||||
...(vyos ? { vyos } : {}),
|
||||
});
|
||||
return reply.code(result.status === "ok" ? 200 : 500).send(result);
|
||||
} catch (err) {
|
||||
@@ -165,6 +198,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
|
||||
const result = await sendCommand(bastion.bastionId, {
|
||||
type: "command-install",
|
||||
mac, hostname, disk: disk ?? "", role: role ?? "infra", os: os ?? "fedora-43",
|
||||
...(vyos ? { vyos } : {}),
|
||||
});
|
||||
return reply.code(result.status === "ok" ? 200 : 500).send(result);
|
||||
} catch (err) {
|
||||
@@ -260,6 +294,37 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
|
||||
}
|
||||
});
|
||||
|
||||
// Update hardware info (discovery data) for a machine
|
||||
app.post<{
|
||||
Body: {
|
||||
mac?: string; product?: string; board?: string; serial?: string;
|
||||
manufacturer?: string; cpu_model?: string; cpu_cores?: number;
|
||||
memory_gb?: number; arch?: string;
|
||||
disks?: Array<{ name: string; size_gb: number; model: string }>;
|
||||
nics?: Array<{ name: string; mac: string; state: string }>;
|
||||
};
|
||||
}>("/api/machines/discover", async (request, reply) => {
|
||||
const data = request.body ?? {};
|
||||
const mac = (data.mac ?? "").toLowerCase().replace(/-/g, ":");
|
||||
if (!mac) {
|
||||
return reply.code(400).send({ error: "mac is required" });
|
||||
}
|
||||
|
||||
const bastion = bastionRegistry.findBastionByMac(mac);
|
||||
const target = bastion ?? (bastionRegistry.getAll().length === 1 ? bastionRegistry.getAll()[0] : null);
|
||||
|
||||
if (!target) {
|
||||
return reply.code(503).send({ error: "No bastion found for this MAC" });
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await sendCommand(target.bastionId, { type: "command-discover", ...data, mac });
|
||||
return reply.code(result.status === "ok" ? 200 : 500).send(result);
|
||||
} catch (err) {
|
||||
return reply.code(500).send({ error: err instanceof Error ? err.message : String(err) });
|
||||
}
|
||||
});
|
||||
|
||||
// Update role
|
||||
app.post<{
|
||||
Body: { mac?: string; role?: string };
|
||||
|
||||
191
bastion/src/labd/src/routes/environments.ts
Normal file
191
bastion/src/labd/src/routes/environments.ts
Normal file
@@ -0,0 +1,191 @@
|
||||
// Environment and Account management routes.
|
||||
// GET/POST /api/environments — list/create environments
|
||||
// GET/POST /api/accounts — list/create accounts
|
||||
// POST /api/accounts/bind — bind account to environment
|
||||
// GET /api/bindings — list bindings
|
||||
|
||||
import type { FastifyInstance } from "fastify";
|
||||
import type { PrismaClient, Prisma } from "@prisma/client";
|
||||
import type { RbacService } from "../services/rbac.js";
|
||||
import type { AuditService } from "../services/audit.js";
|
||||
|
||||
export function registerEnvironmentRoutes(
|
||||
app: FastifyInstance,
|
||||
db: PrismaClient,
|
||||
rbacService: RbacService,
|
||||
auditService: AuditService,
|
||||
): void {
|
||||
// List environments
|
||||
app.get("/api/environments", async (_request, reply) => {
|
||||
const envs = await db.environment.findMany({ orderBy: { name: "asc" } });
|
||||
return reply.send(envs);
|
||||
});
|
||||
|
||||
// Create environment
|
||||
app.post<{
|
||||
Body: { name?: string };
|
||||
}>("/api/environments", async (request, reply) => {
|
||||
const { name } = request.body ?? {};
|
||||
if (!name) {
|
||||
return reply.code(400).send({ error: "name is required" });
|
||||
}
|
||||
|
||||
const rbac = await rbacService.check({
|
||||
userId: request.userId!,
|
||||
userEmail: request.userEmail!,
|
||||
userRole: request.userRole!,
|
||||
action: "admin",
|
||||
resource: "environments",
|
||||
});
|
||||
if (!rbac.allowed) {
|
||||
return reply.code(403).send({ error: rbac.reason });
|
||||
}
|
||||
|
||||
try {
|
||||
const env = await db.environment.create({ data: { name } });
|
||||
auditService.emit({
|
||||
eventKind: "resource_created",
|
||||
source: "labd",
|
||||
verified: true,
|
||||
userId: request.userId ?? null,
|
||||
resourceKind: "environment",
|
||||
resourceName: name,
|
||||
result: "success",
|
||||
});
|
||||
return reply.code(201).send(env);
|
||||
} catch (err) {
|
||||
if (err instanceof Error && err.message.includes("Unique constraint")) {
|
||||
return reply.code(409).send({ error: `Environment '${name}' already exists` });
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
});
|
||||
|
||||
// List accounts
|
||||
app.get("/api/accounts", async (_request, reply) => {
|
||||
const accounts = await db.account.findMany({
|
||||
orderBy: { name: "asc" },
|
||||
select: { id: true, name: true, driver: true, config: true, createdAt: true, updatedAt: true },
|
||||
});
|
||||
return reply.send(accounts);
|
||||
});
|
||||
|
||||
// Create account
|
||||
app.post<{
|
||||
Body: { name?: string; driver?: string; config?: Record<string, unknown> };
|
||||
}>("/api/accounts", async (request, reply) => {
|
||||
const { name, driver, config } = request.body ?? {};
|
||||
if (!name || !driver) {
|
||||
return reply.code(400).send({ error: "name and driver are required" });
|
||||
}
|
||||
|
||||
const rbac = await rbacService.check({
|
||||
userId: request.userId!,
|
||||
userEmail: request.userEmail!,
|
||||
userRole: request.userRole!,
|
||||
action: "admin",
|
||||
resource: "accounts",
|
||||
});
|
||||
if (!rbac.allowed) {
|
||||
return reply.code(403).send({ error: rbac.reason });
|
||||
}
|
||||
|
||||
try {
|
||||
const account = await db.account.create({
|
||||
data: { name, driver, config: (config ?? {}) as Prisma.InputJsonValue },
|
||||
});
|
||||
auditService.emit({
|
||||
eventKind: "resource_created",
|
||||
source: "labd",
|
||||
verified: true,
|
||||
userId: request.userId ?? null,
|
||||
resourceKind: "account",
|
||||
resourceName: name,
|
||||
result: "success",
|
||||
details: { driver },
|
||||
});
|
||||
return reply.code(201).send(account);
|
||||
} catch (err) {
|
||||
if (err instanceof Error && err.message.includes("Unique constraint")) {
|
||||
return reply.code(409).send({ error: `Account '${name}' already exists` });
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
});
|
||||
|
||||
// Bind account to environment
|
||||
app.post<{
|
||||
Body: { environmentId?: string; accountId?: string };
|
||||
}>("/api/accounts/bind", async (request, reply) => {
|
||||
const { environmentId, accountId } = request.body ?? {};
|
||||
if (!environmentId || !accountId) {
|
||||
return reply.code(400).send({ error: "environmentId and accountId are required" });
|
||||
}
|
||||
|
||||
const rbac = await rbacService.check({
|
||||
userId: request.userId!,
|
||||
userEmail: request.userEmail!,
|
||||
userRole: request.userRole!,
|
||||
action: "admin",
|
||||
resource: "accounts",
|
||||
});
|
||||
if (!rbac.allowed) {
|
||||
return reply.code(403).send({ error: rbac.reason });
|
||||
}
|
||||
|
||||
try {
|
||||
const binding = await db.binding.create({
|
||||
data: { environmentId, accountId },
|
||||
});
|
||||
return reply.code(201).send(binding);
|
||||
} catch (err) {
|
||||
if (err instanceof Error && err.message.includes("Unique constraint")) {
|
||||
return reply.code(409).send({ error: "This account is already bound to this environment" });
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
});
|
||||
|
||||
// List bindings
|
||||
app.get("/api/bindings", async (_request, reply) => {
|
||||
const bindings = await db.binding.findMany({
|
||||
include: { environment: true, account: true },
|
||||
});
|
||||
return reply.send(bindings);
|
||||
});
|
||||
|
||||
// Audit event query
|
||||
app.get<{
|
||||
Querystring: {
|
||||
last?: string;
|
||||
kind?: string;
|
||||
env?: string;
|
||||
correlation?: string;
|
||||
limit?: string;
|
||||
};
|
||||
}>("/api/events", async (request, reply) => {
|
||||
const { last, kind, env, correlation, limit } = request.query as { last?: string; kind?: string; env?: string; correlation?: string; limit?: string };
|
||||
|
||||
const where: Record<string, unknown> = {};
|
||||
|
||||
if (last) {
|
||||
const match = last.match(/^(\d+)(h|d|m)$/);
|
||||
if (match) {
|
||||
const [, num, unit] = match;
|
||||
const ms = { h: 3_600_000, d: 86_400_000, m: 60_000 }[unit!]!;
|
||||
where.timestamp = { gte: new Date(Date.now() - parseInt(num!) * ms) };
|
||||
}
|
||||
}
|
||||
if (kind) where.eventKind = kind;
|
||||
if (env) where.environmentName = env;
|
||||
if (correlation) where.correlationId = correlation;
|
||||
|
||||
const events = await db.auditEvent.findMany({
|
||||
where,
|
||||
orderBy: { timestamp: "desc" },
|
||||
take: Math.min(parseInt(limit ?? "100"), 500),
|
||||
});
|
||||
|
||||
return reply.send(events);
|
||||
});
|
||||
}
|
||||
196
bastion/src/labd/src/routes/resources.ts
Normal file
196
bastion/src/labd/src/routes/resources.ts
Normal file
@@ -0,0 +1,196 @@
|
||||
// Resource CRUD routes with RBAC enforcement.
|
||||
// GET /api/resources — list (filtered by RBAC scope)
|
||||
// GET /api/resources/:id — get
|
||||
// POST /api/resources — create
|
||||
// PUT /api/resources/:id — update
|
||||
// DELETE /api/resources/:id — delete (marks as deleting)
|
||||
|
||||
import type { FastifyInstance } from "fastify";
|
||||
import type { ResourceStore, CreateResourceInput } from "../services/resource-store.js";
|
||||
import type { RbacService } from "../services/rbac.js";
|
||||
import type { AuditService } from "../services/audit.js";
|
||||
import { resolveResourceKind } from "@lab/core";
|
||||
|
||||
export function registerResourceRoutes(
|
||||
app: FastifyInstance,
|
||||
resourceStore: ResourceStore,
|
||||
rbacService: RbacService,
|
||||
auditService: AuditService,
|
||||
): void {
|
||||
// List resources (filtered by kind, environment, status)
|
||||
app.get<{
|
||||
Querystring: { kind?: string; environment?: string; status?: string };
|
||||
}>("/api/resources", async (request, reply) => {
|
||||
const rbac = await rbacService.check({
|
||||
userId: request.userId!,
|
||||
userEmail: request.userEmail!,
|
||||
userRole: request.userRole!,
|
||||
action: "view",
|
||||
resource: request.query.kind ? resolveResourceKind(request.query.kind) : undefined,
|
||||
});
|
||||
|
||||
if (!rbac.allowed) {
|
||||
return reply.code(403).send({ error: rbac.reason });
|
||||
}
|
||||
|
||||
const resources = await resourceStore.list({
|
||||
kind: request.query.kind ? resolveResourceKind(request.query.kind) : undefined,
|
||||
environmentId: request.query.environment,
|
||||
status: request.query.status,
|
||||
});
|
||||
|
||||
return reply.send(resources);
|
||||
});
|
||||
|
||||
// Get single resource
|
||||
app.get<{
|
||||
Params: { id: string };
|
||||
}>("/api/resources/:id", async (request, reply) => {
|
||||
const resource = await resourceStore.get(request.params.id);
|
||||
if (!resource) {
|
||||
return reply.code(404).send({ error: "Resource not found" });
|
||||
}
|
||||
|
||||
const rbac = await rbacService.check({
|
||||
userId: request.userId!,
|
||||
userEmail: request.userEmail!,
|
||||
userRole: request.userRole!,
|
||||
action: "view",
|
||||
resource: resource.kind,
|
||||
name: resource.name,
|
||||
});
|
||||
|
||||
if (!rbac.allowed) {
|
||||
return reply.code(403).send({ error: rbac.reason });
|
||||
}
|
||||
|
||||
return reply.send(resource);
|
||||
});
|
||||
|
||||
// Create resource
|
||||
app.post<{
|
||||
Body: CreateResourceInput;
|
||||
}>("/api/resources", async (request, reply) => {
|
||||
const input = request.body;
|
||||
if (!input?.kind || !input?.name || !input?.environmentId || !input?.accountId) {
|
||||
return reply.code(400).send({ error: "kind, name, environmentId, and accountId are required" });
|
||||
}
|
||||
|
||||
const kind = resolveResourceKind(input.kind);
|
||||
|
||||
const rbac = await rbacService.check({
|
||||
userId: request.userId!,
|
||||
userEmail: request.userEmail!,
|
||||
userRole: request.userRole!,
|
||||
action: "create",
|
||||
resource: kind,
|
||||
});
|
||||
|
||||
if (!rbac.allowed) {
|
||||
return reply.code(403).send({ error: rbac.reason });
|
||||
}
|
||||
|
||||
const correlationId = auditService.createCorrelation();
|
||||
|
||||
try {
|
||||
const resource = await resourceStore.create({ ...input, kind });
|
||||
|
||||
auditService.emit({
|
||||
eventKind: "resource_created",
|
||||
source: "labd",
|
||||
verified: true,
|
||||
userId: request.userId ?? null,
|
||||
userName: request.userEmail ?? null,
|
||||
resourceKind: kind,
|
||||
resourceName: input.name,
|
||||
correlationId,
|
||||
result: "success",
|
||||
});
|
||||
|
||||
return reply.code(201).send(resource);
|
||||
} catch (err) {
|
||||
// Prisma unique constraint violation
|
||||
if (err instanceof Error && err.message.includes("Unique constraint")) {
|
||||
return reply.code(409).send({ error: `Resource ${kind}/${input.name} already exists in this environment` });
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
});
|
||||
|
||||
// Update resource
|
||||
app.put<{
|
||||
Params: { id: string };
|
||||
Body: { desiredSpec?: Record<string, unknown>; status?: string };
|
||||
}>("/api/resources/:id", async (request, reply) => {
|
||||
const resource = await resourceStore.get(request.params.id);
|
||||
if (!resource) {
|
||||
return reply.code(404).send({ error: "Resource not found" });
|
||||
}
|
||||
|
||||
const rbac = await rbacService.check({
|
||||
userId: request.userId!,
|
||||
userEmail: request.userEmail!,
|
||||
userRole: request.userRole!,
|
||||
action: "edit",
|
||||
resource: resource.kind,
|
||||
name: resource.name,
|
||||
});
|
||||
|
||||
if (!rbac.allowed) {
|
||||
return reply.code(403).send({ error: rbac.reason });
|
||||
}
|
||||
|
||||
const updated = await resourceStore.update(request.params.id, request.body);
|
||||
|
||||
auditService.emit({
|
||||
eventKind: "resource_updated",
|
||||
source: "labd",
|
||||
verified: true,
|
||||
userId: request.userId ?? null,
|
||||
userName: request.userEmail ?? null,
|
||||
resourceKind: resource.kind,
|
||||
resourceName: resource.name,
|
||||
result: "success",
|
||||
});
|
||||
|
||||
return reply.send(updated);
|
||||
});
|
||||
|
||||
// Delete resource (marks as deleting)
|
||||
app.delete<{
|
||||
Params: { id: string };
|
||||
}>("/api/resources/:id", async (request, reply) => {
|
||||
const resource = await resourceStore.get(request.params.id);
|
||||
if (!resource) {
|
||||
return reply.code(404).send({ error: "Resource not found" });
|
||||
}
|
||||
|
||||
const rbac = await rbacService.check({
|
||||
userId: request.userId!,
|
||||
userEmail: request.userEmail!,
|
||||
userRole: request.userRole!,
|
||||
action: "delete",
|
||||
resource: resource.kind,
|
||||
name: resource.name,
|
||||
});
|
||||
|
||||
if (!rbac.allowed) {
|
||||
return reply.code(403).send({ error: rbac.reason });
|
||||
}
|
||||
|
||||
await resourceStore.delete(request.params.id);
|
||||
|
||||
auditService.emit({
|
||||
eventKind: "resource_deleted",
|
||||
source: "labd",
|
||||
verified: true,
|
||||
userId: request.userId ?? null,
|
||||
userName: request.userEmail ?? null,
|
||||
resourceKind: resource.kind,
|
||||
resourceName: resource.name,
|
||||
result: "success",
|
||||
});
|
||||
|
||||
return reply.send({ status: "deleting", id: request.params.id });
|
||||
});
|
||||
}
|
||||
81
bastion/src/labd/src/routes/v2-auth.ts
Normal file
81
bastion/src/labd/src/routes/v2-auth.ts
Normal file
@@ -0,0 +1,81 @@
|
||||
// v2 Auth routes: bearer token login/logout.
|
||||
// POST /api/auth/login — email + password → session token
|
||||
// POST /api/auth/logout — revoke session
|
||||
|
||||
import type { FastifyInstance } from "fastify";
|
||||
import type { AuthService } from "../services/auth.js";
|
||||
import type { AuditService } from "../services/audit.js";
|
||||
import { AuthError } from "../services/auth.js";
|
||||
|
||||
export function registerV2AuthRoutes(
|
||||
app: FastifyInstance,
|
||||
authService: AuthService,
|
||||
auditService: AuditService,
|
||||
): void {
|
||||
app.post<{
|
||||
Body: { email?: string; password?: string };
|
||||
}>("/api/auth/login", async (request, reply) => {
|
||||
const { email, password } = request.body ?? {};
|
||||
|
||||
if (!email || !password) {
|
||||
return reply.code(400).send({ error: "email and password are required" });
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await authService.login(email, password);
|
||||
|
||||
auditService.emit({
|
||||
eventKind: result.isBootstrap ? "auth_bootstrap" : "auth_login",
|
||||
source: "labd",
|
||||
verified: true,
|
||||
userId: result.userId,
|
||||
userName: email,
|
||||
result: "success",
|
||||
details: { isBootstrap: result.isBootstrap },
|
||||
});
|
||||
|
||||
return reply.send({
|
||||
token: result.token,
|
||||
expiresAt: result.expiresAt.toISOString(),
|
||||
isBootstrap: result.isBootstrap,
|
||||
});
|
||||
} catch (err) {
|
||||
if (err instanceof AuthError) {
|
||||
auditService.emit({
|
||||
eventKind: "auth_login",
|
||||
source: "labd",
|
||||
verified: true,
|
||||
userName: email,
|
||||
result: "failure",
|
||||
error: err.message,
|
||||
});
|
||||
return reply.code(401).send({ error: err.message });
|
||||
}
|
||||
return reply.code(500).send({ error: "Login failed" });
|
||||
}
|
||||
});
|
||||
|
||||
app.post("/api/auth/logout", async (request, reply) => {
|
||||
const token = request.headers.authorization?.slice(7);
|
||||
if (!token) {
|
||||
return reply.code(400).send({ error: "Authorization header required" });
|
||||
}
|
||||
|
||||
try {
|
||||
await authService.logout(token);
|
||||
auditService.emit({
|
||||
eventKind: "auth_logout",
|
||||
source: "labd",
|
||||
verified: true,
|
||||
userId: request.userId ?? null,
|
||||
result: "success",
|
||||
});
|
||||
return reply.send({ status: "logged_out" });
|
||||
} catch (err) {
|
||||
if (err instanceof AuthError) {
|
||||
return reply.code(400).send({ error: err.message });
|
||||
}
|
||||
return reply.code(500).send({ error: "Logout failed" });
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
import Fastify from "fastify";
|
||||
import websocket from "@fastify/websocket";
|
||||
import type { PrismaClient } from "@prisma/client";
|
||||
import type { LabdConfig } from "./config.js";
|
||||
import { logger } from "./services/logger.js";
|
||||
import { registerHealthRoutes } from "./routes/health.js";
|
||||
@@ -9,8 +10,16 @@ import { registerServerRoutes } from "./routes/servers.js";
|
||||
import { registerAuthRoutes } from "./routes/auth.js";
|
||||
import { registerAgentRoutes } from "./routes/agents.js";
|
||||
import { registerBastionRoutes } from "./routes/bastions.js";
|
||||
import { registerV2AuthRoutes } from "./routes/v2-auth.js";
|
||||
import { registerEnvironmentRoutes } from "./routes/environments.js";
|
||||
import { registerResourceRoutes } from "./routes/resources.js";
|
||||
import { setupRateLimiting } from "./middleware/rate-limit.js";
|
||||
import { createBearerAuthMiddleware } from "./middleware/bearer-auth.js";
|
||||
import { bastionRegistry } from "./services/bastion-registry.js";
|
||||
import { AuthService } from "./services/auth.js";
|
||||
import { RbacService } from "./services/rbac.js";
|
||||
import { ResourceStore } from "./services/resource-store.js";
|
||||
import { AuditService } from "./services/audit.js";
|
||||
import { isBastionMessage } from "@lab/shared";
|
||||
|
||||
export interface DbClient {
|
||||
@@ -37,6 +46,7 @@ export interface DbClient {
|
||||
|
||||
export async function createApp(_config: LabdConfig, db: DbClient): Promise<{
|
||||
app: ReturnType<typeof Fastify>;
|
||||
auditService: AuditService;
|
||||
}> {
|
||||
const app = Fastify({
|
||||
logger: false, // We use winston instead
|
||||
@@ -48,13 +58,39 @@ export async function createApp(_config: LabdConfig, db: DbClient): Promise<{
|
||||
// Register WebSocket support
|
||||
void app.register(websocket);
|
||||
|
||||
// Register route handlers
|
||||
// v2 services. The structural DbClient is a subset of the real PrismaClient;
|
||||
// at runtime db IS the PrismaClient instance, so the cast is safe. Tests that
|
||||
// exercise v2 routes provide a PrismaClient-shaped mock (see auth-bootstrap,
|
||||
// rbac-deny, audit-correlation tests).
|
||||
const prisma = db as unknown as PrismaClient;
|
||||
const authService = new AuthService(prisma);
|
||||
const rbacService = new RbacService(prisma);
|
||||
const resourceStore = new ResourceStore(prisma);
|
||||
const auditService = new AuditService(prisma);
|
||||
auditService.start();
|
||||
|
||||
// Register v1 (legacy) route handlers
|
||||
registerHealthRoutes(app, db);
|
||||
registerServerRoutes(app, db);
|
||||
registerAuthRoutes(app, db);
|
||||
registerAgentRoutes(app);
|
||||
registerBastionRoutes(app, db);
|
||||
|
||||
// v2 routes live in a scope with bearer-auth as preHandler. Public paths
|
||||
// (login, /health, websockets) are skipped inside the middleware itself.
|
||||
// v1 routes above are unaffected — they're registered on the root scope.
|
||||
await app.register(async (scope) => {
|
||||
scope.addHook("preHandler", createBearerAuthMiddleware(authService));
|
||||
registerV2AuthRoutes(scope, authService, auditService);
|
||||
registerEnvironmentRoutes(scope, prisma, rbacService, auditService);
|
||||
registerResourceRoutes(scope, resourceStore, rbacService, auditService);
|
||||
});
|
||||
|
||||
// Flush pending audit events on shutdown so we never lose the last batch.
|
||||
app.addHook("onClose", async () => {
|
||||
auditService.stop();
|
||||
});
|
||||
|
||||
// WebSocket handler for agent connections
|
||||
app.register(async (fastify) => {
|
||||
fastify.get("/ws/agent", { websocket: true }, (socket, _request) => {
|
||||
@@ -192,7 +228,9 @@ export async function createApp(_config: LabdConfig, db: DbClient): Promise<{
|
||||
labels: { cpu: hw.cpu_model, cores: hw.cpu_cores, memory_gb: hw.memory_gb, arch: hw.arch, product: hw.product, manufacturer: hw.manufacturer },
|
||||
},
|
||||
update: {
|
||||
status: "discovered",
|
||||
// Leave status alone — a previously "online"/"offline" record
|
||||
// must not be downgraded to "discovered" just because the bastion
|
||||
// restarted and re-discovered the MAC via DHCP/PXE.
|
||||
lastHeartbeat: new Date(),
|
||||
labels: { cpu: hw.cpu_model, cores: hw.cpu_cores, memory_gb: hw.memory_gb, arch: hw.arch, product: hw.product, manufacturer: hw.manufacturer },
|
||||
},
|
||||
@@ -265,5 +303,5 @@ export async function createApp(_config: LabdConfig, db: DbClient): Promise<{
|
||||
logger.info(`HTTP: ${request.ip} ${request.method} ${request.url}`);
|
||||
});
|
||||
|
||||
return { app };
|
||||
return { app, auditService };
|
||||
}
|
||||
|
||||
106
bastion/src/labd/src/services/audit.ts
Normal file
106
bastion/src/labd/src/services/audit.ts
Normal file
@@ -0,0 +1,106 @@
|
||||
// Audit service: fire-and-forget event collection with batching.
|
||||
// Batches 50 events or flushes every 5 seconds, whichever comes first.
|
||||
// Failures never block the operation being audited.
|
||||
|
||||
import { randomBytes } from "node:crypto";
|
||||
import type { PrismaClient, Prisma } from "@prisma/client";
|
||||
import { logger } from "./logger.js";
|
||||
|
||||
const BATCH_SIZE = 50;
|
||||
const FLUSH_INTERVAL_MS = 5_000;
|
||||
|
||||
export interface AuditEventInput {
|
||||
eventKind: string;
|
||||
source: string;
|
||||
verified?: boolean;
|
||||
userId?: string | null;
|
||||
userName?: string | null;
|
||||
sessionId?: string | null;
|
||||
environmentName?: string | null;
|
||||
accountName?: string | null;
|
||||
resourceKind?: string | null;
|
||||
resourceName?: string | null;
|
||||
correlationId?: string | null;
|
||||
parentEventId?: string | null;
|
||||
details?: Record<string, unknown>;
|
||||
result: string;
|
||||
error?: string | null;
|
||||
durationMs?: number | null;
|
||||
}
|
||||
|
||||
export class AuditService {
|
||||
private batch: AuditEventInput[] = [];
|
||||
private timer: ReturnType<typeof setInterval> | null = null;
|
||||
|
||||
constructor(private readonly db: PrismaClient) {}
|
||||
|
||||
start(): void {
|
||||
this.timer = setInterval(() => {
|
||||
void this.flush();
|
||||
}, FLUSH_INTERVAL_MS);
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
if (this.timer) {
|
||||
clearInterval(this.timer);
|
||||
this.timer = null;
|
||||
}
|
||||
void this.flush();
|
||||
}
|
||||
|
||||
emit(event: AuditEventInput): void {
|
||||
// Generate correlation ID if not provided
|
||||
if (!event.correlationId) {
|
||||
event.correlationId = `corr_${randomBytes(8).toString("hex")}`;
|
||||
}
|
||||
|
||||
this.batch.push(event);
|
||||
|
||||
if (this.batch.length >= BATCH_SIZE) {
|
||||
void this.flush();
|
||||
}
|
||||
}
|
||||
|
||||
/** Create a correlation context for a chain of related events. */
|
||||
createCorrelation(): string {
|
||||
return `corr_${randomBytes(8).toString("hex")}`;
|
||||
}
|
||||
|
||||
/** Flush all pending events synchronously. Tests await this; production
|
||||
* relies on the interval timer or stop() during shutdown. */
|
||||
async flushPending(): Promise<void> {
|
||||
await this.flush();
|
||||
}
|
||||
|
||||
private async flush(): Promise<void> {
|
||||
if (this.batch.length === 0) return;
|
||||
|
||||
const events = this.batch.splice(0);
|
||||
try {
|
||||
await this.db.auditEvent.createMany({
|
||||
data: events.map((e) => ({
|
||||
eventKind: e.eventKind,
|
||||
source: e.source,
|
||||
verified: e.verified ?? false,
|
||||
userId: e.userId ?? null,
|
||||
userName: e.userName ?? null,
|
||||
sessionId: e.sessionId ?? null,
|
||||
environmentName: e.environmentName ?? null,
|
||||
accountName: e.accountName ?? null,
|
||||
resourceKind: e.resourceKind ?? null,
|
||||
resourceName: e.resourceName ?? null,
|
||||
correlationId: e.correlationId ?? `corr_${randomBytes(8).toString("hex")}`,
|
||||
parentEventId: e.parentEventId ?? null,
|
||||
details: (e.details ?? {}) as Prisma.InputJsonValue,
|
||||
result: e.result,
|
||||
error: e.error ?? null,
|
||||
durationMs: e.durationMs ?? null,
|
||||
})),
|
||||
});
|
||||
logger.info(`AUDIT: flushed ${events.length} events`);
|
||||
} catch (err) {
|
||||
// Fire-and-forget: audit failures never block operations
|
||||
logger.warn(`AUDIT: failed to flush ${events.length} events: ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
119
bastion/src/labd/src/services/auth.ts
Normal file
119
bastion/src/labd/src/services/auth.ts
Normal file
@@ -0,0 +1,119 @@
|
||||
// Auth service: bearer token authentication with bootstrap flow.
|
||||
// First login creates the admin user. Subsequent logins return session tokens.
|
||||
|
||||
import { randomBytes } from "node:crypto";
|
||||
import bcrypt from "bcryptjs";
|
||||
import type { PrismaClient } from "@prisma/client";
|
||||
import { logger } from "./logger.js";
|
||||
|
||||
const SESSION_EXPIRY_DAYS = 30;
|
||||
const BCRYPT_ROUNDS = 12;
|
||||
|
||||
export interface LoginResult {
|
||||
token: string;
|
||||
expiresAt: Date;
|
||||
userId: string;
|
||||
isBootstrap: boolean;
|
||||
}
|
||||
|
||||
export class AuthService {
|
||||
constructor(private readonly db: PrismaClient) {}
|
||||
|
||||
async login(email: string, password: string): Promise<LoginResult> {
|
||||
const userCount = await this.db.user.count();
|
||||
|
||||
// Bootstrap: first login creates admin user
|
||||
if (userCount === 0) {
|
||||
return this.bootstrap(email, password);
|
||||
}
|
||||
|
||||
const user = await this.db.user.findUnique({ where: { email } });
|
||||
if (!user) {
|
||||
// Same error for unknown user and wrong password (no enumeration)
|
||||
throw new AuthError("Invalid email or password");
|
||||
}
|
||||
|
||||
const valid = await bcrypt.compare(password, user.password);
|
||||
if (!valid) {
|
||||
throw new AuthError("Invalid email or password");
|
||||
}
|
||||
|
||||
const session = await this.createSession(user.id);
|
||||
logger.info(`AUTH LOGIN: ${email} (${user.id.slice(0, 8)}...)`);
|
||||
|
||||
return {
|
||||
token: session.token,
|
||||
expiresAt: session.expiresAt,
|
||||
userId: user.id,
|
||||
isBootstrap: false,
|
||||
};
|
||||
}
|
||||
|
||||
async logout(token: string): Promise<void> {
|
||||
const session = await this.db.session.findUnique({ where: { token } });
|
||||
if (!session) {
|
||||
throw new AuthError("Invalid session");
|
||||
}
|
||||
await this.db.session.delete({ where: { id: session.id } });
|
||||
logger.info(`AUTH LOGOUT: session ${session.id.slice(0, 8)}...`);
|
||||
}
|
||||
|
||||
async validateToken(token: string): Promise<{ userId: string; email: string; role: string }> {
|
||||
const session = await this.db.session.findUnique({
|
||||
where: { token },
|
||||
include: { user: true },
|
||||
});
|
||||
|
||||
if (!session) {
|
||||
throw new AuthError("Invalid token");
|
||||
}
|
||||
if (session.expiresAt < new Date()) {
|
||||
await this.db.session.delete({ where: { id: session.id } });
|
||||
throw new AuthError("Token expired");
|
||||
}
|
||||
|
||||
return {
|
||||
userId: session.user.id,
|
||||
email: session.user.email,
|
||||
role: session.user.role,
|
||||
};
|
||||
}
|
||||
|
||||
private async bootstrap(email: string, password: string): Promise<LoginResult> {
|
||||
const hashed = await bcrypt.hash(password, BCRYPT_ROUNDS);
|
||||
const user = await this.db.user.create({
|
||||
data: {
|
||||
email,
|
||||
password: hashed,
|
||||
role: "ADMIN",
|
||||
name: email.split("@")[0] ?? null,
|
||||
},
|
||||
});
|
||||
|
||||
const session = await this.createSession(user.id);
|
||||
logger.info(`AUTH BOOTSTRAP: created admin user ${email} (${user.id.slice(0, 8)}...)`);
|
||||
|
||||
return {
|
||||
token: session.token,
|
||||
expiresAt: session.expiresAt,
|
||||
userId: user.id,
|
||||
isBootstrap: true,
|
||||
};
|
||||
}
|
||||
|
||||
private async createSession(userId: string) {
|
||||
const token = randomBytes(32).toString("hex");
|
||||
const expiresAt = new Date(Date.now() + SESSION_EXPIRY_DAYS * 24 * 60 * 60 * 1000);
|
||||
|
||||
return this.db.session.create({
|
||||
data: { userId, token, expiresAt },
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export class AuthError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = "AuthError";
|
||||
}
|
||||
}
|
||||
123
bastion/src/labd/src/services/rbac.ts
Normal file
123
bastion/src/labd/src/services/rbac.ts
Normal file
@@ -0,0 +1,123 @@
|
||||
// RBAC service: environment-scoped permission checks.
|
||||
// Uses named RbacDefinition records with JSON subjects and roleBindings.
|
||||
//
|
||||
// Resolution flow:
|
||||
// 1. Find all RbacDefinitions where subjects match the current user/groups
|
||||
// 2. Collect all roleBindings from matching definitions
|
||||
// 3. Check if any binding grants the requested action on the requested resource
|
||||
|
||||
import type { PrismaClient } from "@prisma/client";
|
||||
import { logger } from "./logger.js";
|
||||
|
||||
export interface RbacCheck {
|
||||
userId: string;
|
||||
userEmail: string;
|
||||
userRole: string;
|
||||
action: string; // "view" | "edit" | "create" | "delete" | "run" | "admin"
|
||||
resource?: string | undefined; // "servers" | "databases" | "clusters" | "*"
|
||||
name?: string | undefined; // specific resource name
|
||||
environment?: string | undefined; // specific environment name
|
||||
}
|
||||
|
||||
export interface RbacResult {
|
||||
allowed: boolean;
|
||||
reason: string;
|
||||
matchedDefinition?: string;
|
||||
}
|
||||
|
||||
interface StoredSubject {
|
||||
kind: string;
|
||||
name: string;
|
||||
}
|
||||
|
||||
interface StoredBinding {
|
||||
role: string;
|
||||
resource?: string;
|
||||
name?: string;
|
||||
environment?: string;
|
||||
action?: string;
|
||||
}
|
||||
|
||||
export class RbacService {
|
||||
constructor(private readonly db: PrismaClient) {}
|
||||
|
||||
async check(req: RbacCheck): Promise<RbacResult> {
|
||||
// Admin users bypass RBAC
|
||||
if (req.userRole === "ADMIN") {
|
||||
return { allowed: true, reason: "admin role" };
|
||||
}
|
||||
|
||||
// Collect user's group memberships
|
||||
const memberships = await this.db.groupMember.findMany({
|
||||
where: { userId: req.userId },
|
||||
include: { group: true },
|
||||
});
|
||||
const groupNames = memberships.map((m) => m.group.name);
|
||||
|
||||
// Find all RBAC definitions
|
||||
const definitions = await this.db.rbacDefinition.findMany();
|
||||
|
||||
for (const def of definitions) {
|
||||
const subjects = def.subjects as unknown as StoredSubject[];
|
||||
const bindings = def.roleBindings as unknown as StoredBinding[];
|
||||
|
||||
// Check if this definition's subjects match the user
|
||||
const subjectMatch = subjects.some((s) => {
|
||||
if (s.kind === "User" && s.name === req.userEmail) return true;
|
||||
if (s.kind === "Group" && groupNames.includes(s.name)) return true;
|
||||
return false;
|
||||
});
|
||||
|
||||
if (!subjectMatch) continue;
|
||||
|
||||
// Check if any binding grants the requested permission
|
||||
for (const binding of bindings) {
|
||||
if (this.bindingMatches(binding, req)) {
|
||||
logger.info(`RBAC ALLOW: ${req.userEmail} ${req.action} ${req.resource ?? "*"}${req.name ? `/${req.name}` : ""} via ${def.name}`);
|
||||
return {
|
||||
allowed: true,
|
||||
reason: `granted by ${def.name}`,
|
||||
matchedDefinition: def.name,
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
logger.info(`RBAC DENY: ${req.userEmail} ${req.action} ${req.resource ?? "*"}${req.name ? `/${req.name}` : ""}`);
|
||||
return {
|
||||
allowed: false,
|
||||
reason: `no matching role binding for ${req.action} on ${req.resource ?? "*"}`,
|
||||
};
|
||||
}
|
||||
|
||||
private bindingMatches(binding: StoredBinding, req: RbacCheck): boolean {
|
||||
// Check role grants the action
|
||||
if (!this.roleGrantsAction(binding.role, req.action)) return false;
|
||||
|
||||
// Check resource scope
|
||||
if (binding.resource && binding.resource !== "*" && binding.resource !== req.resource) return false;
|
||||
|
||||
// Check name scope
|
||||
if (binding.name && binding.name !== req.name) return false;
|
||||
|
||||
// Check environment scope
|
||||
if (binding.environment && binding.environment !== req.environment) return false;
|
||||
|
||||
// Check operation scope (for "run" role with specific actions)
|
||||
if (binding.action && binding.action !== "*" && binding.action !== req.action) return false;
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
private roleGrantsAction(role: string, action: string): boolean {
|
||||
const grants: Record<string, string[]> = {
|
||||
admin: ["view", "edit", "create", "delete", "run", "admin"],
|
||||
edit: ["view", "edit", "create", "delete"],
|
||||
create: ["create"],
|
||||
delete: ["delete"],
|
||||
view: ["view"],
|
||||
run: ["run"],
|
||||
};
|
||||
return grants[role]?.includes(action) ?? false;
|
||||
}
|
||||
}
|
||||
108
bastion/src/labd/src/services/resource-store.ts
Normal file
108
bastion/src/labd/src/services/resource-store.ts
Normal file
@@ -0,0 +1,108 @@
|
||||
// Resource store: CRUD for generic resources with origin/managedBy tracking.
|
||||
// All mutations go through this service so RBAC and audit are applied consistently.
|
||||
|
||||
import type { PrismaClient, Resource as PrismaResource, Prisma } from "@prisma/client";
|
||||
import { logger } from "./logger.js";
|
||||
|
||||
export interface CreateResourceInput {
|
||||
kind: string;
|
||||
name: string;
|
||||
environmentId: string;
|
||||
accountId: string;
|
||||
origin?: string;
|
||||
managedBy?: string;
|
||||
sourceRef?: string;
|
||||
desiredSpec: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface UpdateResourceInput {
|
||||
desiredSpec?: Record<string, unknown>;
|
||||
status?: string;
|
||||
statusMessage?: string;
|
||||
actualSpec?: Record<string, unknown>;
|
||||
platformRef?: string;
|
||||
}
|
||||
|
||||
export interface ListResourcesFilter {
|
||||
kind?: string | undefined;
|
||||
environmentId?: string | undefined;
|
||||
accountId?: string | undefined;
|
||||
status?: string | undefined;
|
||||
}
|
||||
|
||||
export class ResourceStore {
|
||||
constructor(private readonly db: PrismaClient) {}
|
||||
|
||||
async create(input: CreateResourceInput): Promise<PrismaResource> {
|
||||
const resource = await this.db.resource.create({
|
||||
data: {
|
||||
kind: input.kind,
|
||||
name: input.name,
|
||||
environmentId: input.environmentId,
|
||||
accountId: input.accountId,
|
||||
origin: input.origin ?? "cli",
|
||||
managedBy: input.managedBy ?? "manual",
|
||||
sourceRef: input.sourceRef ?? null,
|
||||
desiredSpec: input.desiredSpec as Prisma.InputJsonValue,
|
||||
status: "pending",
|
||||
},
|
||||
});
|
||||
|
||||
logger.info(`RESOURCE CREATED: ${input.kind}/${input.name} in env ${input.environmentId.slice(0, 8)}...`);
|
||||
return resource;
|
||||
}
|
||||
|
||||
async get(id: string): Promise<PrismaResource | null> {
|
||||
return this.db.resource.findUnique({ where: { id } });
|
||||
}
|
||||
|
||||
async getByKindNameEnv(kind: string, name: string, environmentId: string): Promise<PrismaResource | null> {
|
||||
return this.db.resource.findUnique({
|
||||
where: { kind_name_environmentId: { kind, name, environmentId } },
|
||||
});
|
||||
}
|
||||
|
||||
async list(filter: ListResourcesFilter = {}): Promise<PrismaResource[]> {
|
||||
return this.db.resource.findMany({
|
||||
where: {
|
||||
...(filter.kind ? { kind: filter.kind } : {}),
|
||||
...(filter.environmentId ? { environmentId: filter.environmentId } : {}),
|
||||
...(filter.accountId ? { accountId: filter.accountId } : {}),
|
||||
...(filter.status ? { status: filter.status } : {}),
|
||||
},
|
||||
orderBy: { createdAt: "desc" },
|
||||
});
|
||||
}
|
||||
|
||||
async update(id: string, input: UpdateResourceInput): Promise<PrismaResource> {
|
||||
const data: Prisma.ResourceUpdateInput = {};
|
||||
if (input.desiredSpec !== undefined) data.desiredSpec = input.desiredSpec as Prisma.InputJsonValue;
|
||||
if (input.status !== undefined) data.status = input.status;
|
||||
if (input.statusMessage !== undefined) data.statusMessage = input.statusMessage;
|
||||
if (input.actualSpec !== undefined) data.actualSpec = input.actualSpec as Prisma.InputJsonValue;
|
||||
if (input.platformRef !== undefined) data.platformRef = input.platformRef;
|
||||
if (input.status === "ready") data.lastReconciled = new Date();
|
||||
|
||||
const resource = await this.db.resource.update({ where: { id }, data });
|
||||
|
||||
logger.info(`RESOURCE UPDATED: ${resource.kind}/${resource.name} -> ${input.status ?? "spec change"}`);
|
||||
return resource;
|
||||
}
|
||||
|
||||
async delete(id: string): Promise<void> {
|
||||
const resource = await this.db.resource.findUnique({ where: { id } });
|
||||
if (!resource) return;
|
||||
|
||||
// Mark as deleting first (driver handles actual deletion)
|
||||
await this.db.resource.update({
|
||||
where: { id },
|
||||
data: { status: "deleting" },
|
||||
});
|
||||
|
||||
logger.info(`RESOURCE DELETING: ${resource.kind}/${resource.name}`);
|
||||
}
|
||||
|
||||
async hardDelete(id: string): Promise<void> {
|
||||
await this.db.resource.delete({ where: { id } });
|
||||
}
|
||||
}
|
||||
144
bastion/src/labd/tests/bastions-machines.test.ts
Normal file
144
bastion/src/labd/tests/bastions-machines.test.ts
Normal file
@@ -0,0 +1,144 @@
|
||||
import { describe, it, expect, vi, beforeEach } from "vitest";
|
||||
import Fastify from "fastify";
|
||||
import { registerBastionRoutes } from "../src/routes/bastions.js";
|
||||
import { bastionRegistry } from "../src/services/bastion-registry.js";
|
||||
import type { DbClient } from "../src/server.js";
|
||||
import type { BastionState } from "@lab/shared";
|
||||
|
||||
function createMockDb(servers: unknown[] = []): DbClient {
|
||||
return {
|
||||
$queryRaw: vi.fn().mockResolvedValue([{ "?column?": 1 }]),
|
||||
server: {
|
||||
findMany: vi.fn().mockResolvedValue(servers),
|
||||
findUnique: vi.fn().mockResolvedValue(null),
|
||||
upsert: vi.fn().mockResolvedValue({}),
|
||||
},
|
||||
joinToken: {
|
||||
findUnique: vi.fn().mockResolvedValue(null),
|
||||
findMany: vi.fn().mockResolvedValue([]),
|
||||
create: vi.fn().mockResolvedValue({ id: "t" }),
|
||||
update: vi.fn().mockResolvedValue({}),
|
||||
},
|
||||
bastion: {
|
||||
upsert: vi.fn().mockResolvedValue({}),
|
||||
findMany: vi.fn().mockResolvedValue([]),
|
||||
findUnique: vi.fn().mockResolvedValue(null),
|
||||
update: vi.fn().mockResolvedValue({}),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function registerFakeBastion(bastionId: string, state: BastionState): void {
|
||||
bastionRegistry.register({
|
||||
bastionId,
|
||||
hostname: "fake",
|
||||
network: "192.168.8.0/24",
|
||||
serverIp: "192.168.8.11",
|
||||
// socket is referenced only on commands, not during aggregation
|
||||
socket: { on: () => undefined, off: () => undefined, send: () => undefined, close: () => undefined } as never,
|
||||
connectedAt: new Date(),
|
||||
lastHeartbeat: new Date(),
|
||||
state,
|
||||
});
|
||||
}
|
||||
|
||||
describe("GET /api/machines aggregation", () => {
|
||||
beforeEach(() => {
|
||||
for (const b of bastionRegistry.getAll()) bastionRegistry.unregister(b.bastionId);
|
||||
});
|
||||
|
||||
it("promotes a live-discovered MAC to installed when the DB has a real hostname+role for it", async () => {
|
||||
// Simulates the worker0-k8s0 bug: bastion restarted, lost its installed map,
|
||||
// rediscovered the machine via DHCP/PXE. DB still has hostname=worker0-k8s0,
|
||||
// role=infra, ip=192.168.8.23. Without the fix, the CLI sees a "discovered"
|
||||
// row with no hostname/role/IP. With the fix, the row is promoted to
|
||||
// "installed" with full identity preserved.
|
||||
const mac = "78:55:36:08:28:fb";
|
||||
registerFakeBastion("b1", {
|
||||
discovered: {
|
||||
[mac]: {
|
||||
mac, product: "SER", board: "SER", serial: "x", manufacturer: "AZW",
|
||||
cpu_model: "AMD Ryzen 7 255", cpu_cores: 16, memory_gb: 58, arch: "x86_64",
|
||||
disks: [], nics: [], first_seen: "", last_seen: "",
|
||||
},
|
||||
},
|
||||
install_queue: {},
|
||||
installed: {},
|
||||
debug: {},
|
||||
});
|
||||
|
||||
const app = Fastify({ logger: false });
|
||||
const db = createMockDb([
|
||||
{ mac, hostname: "worker0-k8s0", role: "infra", ip: "192.168.8.23", status: "discovered", labels: {} },
|
||||
]);
|
||||
registerBastionRoutes(app, db);
|
||||
|
||||
const res = await app.inject({ method: "GET", url: "/api/machines" });
|
||||
expect(res.statusCode).toBe(200);
|
||||
const body = JSON.parse(res.body);
|
||||
|
||||
expect(body.discovered[mac]).toBeUndefined();
|
||||
expect(body.installed[mac]).toMatchObject({
|
||||
hostname: "worker0-k8s0",
|
||||
role: "infra",
|
||||
ip: "192.168.8.23",
|
||||
cpu_model: "AMD Ryzen 7 255",
|
||||
cpu_cores: 16,
|
||||
memory_gb: 58,
|
||||
});
|
||||
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("leaves a fresh-discovery MAC in discovered when DB only has a discovery-shaped record", async () => {
|
||||
const mac = "aa:bb:cc:dd:ee:ff";
|
||||
registerFakeBastion("b1", {
|
||||
discovered: {
|
||||
[mac]: {
|
||||
mac, product: "SER", board: "SER", serial: "x", manufacturer: "AZW",
|
||||
cpu_model: "AMD Ryzen 7", cpu_cores: 8, memory_gb: 32, arch: "x86_64",
|
||||
disks: [], nics: [], first_seen: "", last_seen: "",
|
||||
},
|
||||
},
|
||||
install_queue: {},
|
||||
installed: {},
|
||||
debug: {},
|
||||
});
|
||||
|
||||
const app = Fastify({ logger: false });
|
||||
// Matches what labd writes on first discovery: hostname=product, role="unknown"
|
||||
const db = createMockDb([
|
||||
{ mac, hostname: "SER", role: "unknown", ip: null, status: "discovered", labels: {} },
|
||||
]);
|
||||
registerBastionRoutes(app, db);
|
||||
|
||||
const res = await app.inject({ method: "GET", url: "/api/machines" });
|
||||
const body = JSON.parse(res.body);
|
||||
|
||||
expect(body.discovered[mac]).toBeDefined();
|
||||
expect(body.installed[mac]).toBeUndefined();
|
||||
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("falls back to DB for MACs not in any live bucket", async () => {
|
||||
const mac = "11:22:33:44:55:66";
|
||||
// No bastions connected
|
||||
const app = Fastify({ logger: false });
|
||||
const db = createMockDb([
|
||||
{ mac, hostname: "worker1-k8s0", role: "infra", ip: "192.168.8.13", status: "online", labels: {} },
|
||||
]);
|
||||
registerBastionRoutes(app, db);
|
||||
|
||||
const res = await app.inject({ method: "GET", url: "/api/machines" });
|
||||
const body = JSON.parse(res.body);
|
||||
|
||||
expect(body.installed[mac]).toMatchObject({
|
||||
hostname: "worker1-k8s0",
|
||||
role: "infra",
|
||||
ip: "192.168.8.13",
|
||||
});
|
||||
|
||||
await app.close();
|
||||
});
|
||||
});
|
||||
425
bastion/src/labd/tests/v2-smoke.test.ts
Normal file
425
bastion/src/labd/tests/v2-smoke.test.ts
Normal file
@@ -0,0 +1,425 @@
|
||||
// End-to-end smoke tests for the v2.0 Phase 1 surface (auth bootstrap, RBAC,
|
||||
// audit correlation). These exercise the wiring in createApp(): the bearer
|
||||
// auth middleware, the v2 routes scope, and the AuditService lifecycle.
|
||||
//
|
||||
// We don't spin up CockroachDB. Instead we provide a PrismaClient-shaped
|
||||
// in-memory mock that matches the surface the v2 services actually touch.
|
||||
// Tests follow the project convention of using mock DBs + Fastify.inject().
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
import bcrypt from "bcryptjs";
|
||||
import { createApp } from "../src/server.js";
|
||||
import type { DbClient } from "../src/server.js";
|
||||
import type { AuditService } from "../src/services/audit.js";
|
||||
|
||||
const TEST_CONFIG = { port: 0, host: "127.0.0.1", databaseUrl: "", caDir: "/tmp", logLevel: "silent" };
|
||||
|
||||
interface UserRow { id: string; email: string; password: string; role: string; name: string | null; }
|
||||
interface SessionRow { id: string; userId: string; token: string; expiresAt: Date; user?: UserRow; }
|
||||
interface RbacDefRow { id: string; name: string; subjects: unknown; roleBindings: unknown; }
|
||||
interface AuditEventRow {
|
||||
id: string;
|
||||
eventKind: string;
|
||||
source: string;
|
||||
verified: boolean;
|
||||
userId: string | null;
|
||||
userName: string | null;
|
||||
environmentName: string | null;
|
||||
resourceKind: string | null;
|
||||
correlationId: string | null;
|
||||
parentEventId: string | null;
|
||||
details: unknown;
|
||||
result: string;
|
||||
error: string | null;
|
||||
durationMs: number | null;
|
||||
timestamp: Date;
|
||||
}
|
||||
|
||||
interface Stores {
|
||||
users: Map<string, UserRow>;
|
||||
sessions: Map<string, SessionRow>;
|
||||
groupMembers: Array<{ userId: string; group: { name: string } }>;
|
||||
rbacDefs: RbacDefRow[];
|
||||
auditEvents: AuditEventRow[];
|
||||
resources: Array<Record<string, unknown>>;
|
||||
}
|
||||
|
||||
function makeStores(): Stores {
|
||||
return {
|
||||
users: new Map(),
|
||||
sessions: new Map(),
|
||||
groupMembers: [],
|
||||
rbacDefs: [],
|
||||
auditEvents: [],
|
||||
resources: [],
|
||||
};
|
||||
}
|
||||
|
||||
function makeMockDb(s: Stores): DbClient {
|
||||
let idCounter = 0;
|
||||
const newId = (prefix: string): string => `${prefix}-${++idCounter}`;
|
||||
|
||||
return {
|
||||
$queryRaw: vi.fn(async () => [{ "?column?": 1 }]),
|
||||
server: { findMany: vi.fn(async () => []), findUnique: vi.fn(), upsert: vi.fn() },
|
||||
joinToken: { findUnique: vi.fn(), findMany: vi.fn(), create: vi.fn(), update: vi.fn() },
|
||||
bastion: { upsert: vi.fn(), findMany: vi.fn(), findUnique: vi.fn(), update: vi.fn() },
|
||||
|
||||
user: {
|
||||
count: vi.fn(async () => s.users.size),
|
||||
findUnique: vi.fn(async (args: { where: { email?: string; id?: string } }) => {
|
||||
if (args.where.email) {
|
||||
for (const u of s.users.values()) if (u.email === args.where.email) return u;
|
||||
}
|
||||
if (args.where.id) return s.users.get(args.where.id) ?? null;
|
||||
return null;
|
||||
}),
|
||||
create: vi.fn(async (args: { data: Omit<UserRow, "id"> }) => {
|
||||
const id = newId("user");
|
||||
const row: UserRow = { id, ...args.data };
|
||||
s.users.set(id, row);
|
||||
return row;
|
||||
}),
|
||||
},
|
||||
session: {
|
||||
findUnique: vi.fn(async (args: { where: { token?: string; id?: string }; include?: { user?: boolean } }) => {
|
||||
let session: SessionRow | undefined;
|
||||
if (args.where.token) {
|
||||
for (const sess of s.sessions.values()) if (sess.token === args.where.token) { session = sess; break; }
|
||||
} else if (args.where.id) {
|
||||
session = s.sessions.get(args.where.id);
|
||||
}
|
||||
if (!session) return null;
|
||||
if (args.include?.user) {
|
||||
return { ...session, user: s.users.get(session.userId)! };
|
||||
}
|
||||
return session;
|
||||
}),
|
||||
create: vi.fn(async (args: { data: { userId: string; token: string; expiresAt: Date } }) => {
|
||||
const id = newId("sess");
|
||||
const row: SessionRow = { id, ...args.data };
|
||||
s.sessions.set(id, row);
|
||||
return row;
|
||||
}),
|
||||
delete: vi.fn(async (args: { where: { id: string } }) => {
|
||||
s.sessions.delete(args.where.id);
|
||||
return null;
|
||||
}),
|
||||
},
|
||||
groupMember: {
|
||||
findMany: vi.fn(async (args: { where: { userId: string } }) =>
|
||||
s.groupMembers.filter((m) => m.userId === args.where.userId),
|
||||
),
|
||||
},
|
||||
rbacDefinition: {
|
||||
findMany: vi.fn(async () => s.rbacDefs),
|
||||
},
|
||||
auditEvent: {
|
||||
createMany: vi.fn(async (args: { data: Array<Omit<AuditEventRow, "id" | "timestamp">> }) => {
|
||||
const ts = new Date();
|
||||
for (const e of args.data) {
|
||||
s.auditEvents.push({ id: newId("evt"), timestamp: ts, ...e });
|
||||
}
|
||||
return { count: args.data.length };
|
||||
}),
|
||||
findMany: vi.fn(async (args: { where?: Record<string, unknown>; orderBy?: unknown; take?: number }) => {
|
||||
const where = args.where ?? {};
|
||||
const filtered = s.auditEvents.filter((e) => {
|
||||
if (where["eventKind"] && e.eventKind !== where["eventKind"]) return false;
|
||||
if (where["correlationId"] && e.correlationId !== where["correlationId"]) return false;
|
||||
if (where["environmentName"] && e.environmentName !== where["environmentName"]) return false;
|
||||
return true;
|
||||
});
|
||||
return filtered.slice(0, args.take ?? 100);
|
||||
}),
|
||||
},
|
||||
resource: {
|
||||
findMany: vi.fn(async () => s.resources),
|
||||
findUnique: vi.fn(),
|
||||
create: vi.fn(),
|
||||
update: vi.fn(),
|
||||
delete: vi.fn(),
|
||||
},
|
||||
environment: { findMany: vi.fn(async () => []), findUnique: vi.fn(), create: vi.fn() },
|
||||
account: { findMany: vi.fn(async () => []), findUnique: vi.fn(), create: vi.fn() },
|
||||
binding: { findMany: vi.fn(async () => []), create: vi.fn() },
|
||||
} as unknown as DbClient;
|
||||
}
|
||||
|
||||
async function buildApp(s: Stores) {
|
||||
const db = makeMockDb(s);
|
||||
const result = await createApp(TEST_CONFIG, db);
|
||||
await result.app.ready();
|
||||
return result;
|
||||
}
|
||||
|
||||
describe("v2 auth: bootstrap flow", () => {
|
||||
let stores: Stores;
|
||||
let app: Awaited<ReturnType<typeof buildApp>>["app"];
|
||||
let auditService: AuditService;
|
||||
|
||||
beforeEach(async () => {
|
||||
stores = makeStores();
|
||||
const built = await buildApp(stores);
|
||||
app = built.app;
|
||||
auditService = built.auditService;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await app.close(); // triggers auditService.stop()
|
||||
});
|
||||
|
||||
it("first login with no users seeds the admin and returns a session token", async () => {
|
||||
expect(stores.users.size).toBe(0);
|
||||
|
||||
const resp = await app.inject({
|
||||
method: "POST",
|
||||
url: "/api/auth/login",
|
||||
payload: { email: "admin@itaz.eu", password: "s3cret-pw" },
|
||||
});
|
||||
|
||||
expect(resp.statusCode).toBe(200);
|
||||
const body = resp.json();
|
||||
expect(body.isBootstrap).toBe(true);
|
||||
expect(body.token).toMatch(/^[a-f0-9]{64}$/);
|
||||
expect(typeof body.expiresAt).toBe("string");
|
||||
|
||||
expect(stores.users.size).toBe(1);
|
||||
const created = [...stores.users.values()][0]!;
|
||||
expect(created.email).toBe("admin@itaz.eu");
|
||||
expect(created.role).toBe("ADMIN");
|
||||
// Password is hashed, not stored plaintext.
|
||||
expect(created.password).not.toBe("s3cret-pw");
|
||||
expect(await bcrypt.compare("s3cret-pw", created.password)).toBe(true);
|
||||
|
||||
// Bootstrap emits an audit event.
|
||||
await auditService.flushPending();
|
||||
const bootstrapEvents = stores.auditEvents.filter((e) => e.eventKind === "auth_bootstrap");
|
||||
expect(bootstrapEvents).toHaveLength(1);
|
||||
expect(bootstrapEvents[0]!.result).toBe("success");
|
||||
expect(bootstrapEvents[0]!.userName).toBe("admin@itaz.eu");
|
||||
});
|
||||
|
||||
it("returns 400 for missing credentials", async () => {
|
||||
const resp = await app.inject({ method: "POST", url: "/api/auth/login", payload: {} });
|
||||
expect(resp.statusCode).toBe(400);
|
||||
});
|
||||
|
||||
it("second login uses normal flow (no isBootstrap)", async () => {
|
||||
// Bootstrap once
|
||||
await app.inject({
|
||||
method: "POST",
|
||||
url: "/api/auth/login",
|
||||
payload: { email: "admin@itaz.eu", password: "s3cret-pw" },
|
||||
});
|
||||
expect(stores.users.size).toBe(1);
|
||||
|
||||
// Login again
|
||||
const resp = await app.inject({
|
||||
method: "POST",
|
||||
url: "/api/auth/login",
|
||||
payload: { email: "admin@itaz.eu", password: "s3cret-pw" },
|
||||
});
|
||||
|
||||
expect(resp.statusCode).toBe(200);
|
||||
expect(resp.json().isBootstrap).toBe(false);
|
||||
expect(stores.users.size).toBe(1); // no new user
|
||||
});
|
||||
|
||||
it("rejects wrong password with 401", async () => {
|
||||
// Seed admin
|
||||
await app.inject({
|
||||
method: "POST",
|
||||
url: "/api/auth/login",
|
||||
payload: { email: "admin@itaz.eu", password: "s3cret-pw" },
|
||||
});
|
||||
|
||||
const resp = await app.inject({
|
||||
method: "POST",
|
||||
url: "/api/auth/login",
|
||||
payload: { email: "admin@itaz.eu", password: "wrong" },
|
||||
});
|
||||
expect(resp.statusCode).toBe(401);
|
||||
|
||||
// Failed login is also audited.
|
||||
await auditService.flushPending();
|
||||
const fails = stores.auditEvents.filter((e) => e.eventKind === "auth_login" && e.result === "failure");
|
||||
expect(fails).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("v2 RBAC: env-scoped denial", () => {
|
||||
let stores: Stores;
|
||||
let app: Awaited<ReturnType<typeof buildApp>>["app"];
|
||||
|
||||
async function seedSession(role: string): Promise<string> {
|
||||
stores.users.set("u-1", {
|
||||
id: "u-1",
|
||||
email: `${role.toLowerCase()}@itaz.eu`,
|
||||
password: "x",
|
||||
role,
|
||||
name: null,
|
||||
});
|
||||
const token = "test-token-" + role;
|
||||
stores.sessions.set("s-1", {
|
||||
id: "s-1",
|
||||
userId: "u-1",
|
||||
token,
|
||||
expiresAt: new Date(Date.now() + 86_400_000),
|
||||
});
|
||||
return token;
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
stores = makeStores();
|
||||
app = (await buildApp(stores)).app;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("non-admin user with no role bindings gets 403 on /api/resources", async () => {
|
||||
const token = await seedSession("EDITOR"); // not admin, no bindings
|
||||
|
||||
const resp = await app.inject({
|
||||
method: "GET",
|
||||
url: "/api/resources",
|
||||
headers: { authorization: `Bearer ${token}` },
|
||||
});
|
||||
|
||||
expect(resp.statusCode).toBe(403);
|
||||
expect(resp.json().error).toMatch(/no matching role binding/);
|
||||
});
|
||||
|
||||
it("missing/empty bearer token gets 401 (auth, not RBAC)", async () => {
|
||||
const r1 = await app.inject({ method: "GET", url: "/api/resources" });
|
||||
expect(r1.statusCode).toBe(401);
|
||||
|
||||
const r2 = await app.inject({
|
||||
method: "GET",
|
||||
url: "/api/resources",
|
||||
headers: { authorization: "Bearer " },
|
||||
});
|
||||
expect(r2.statusCode).toBe(401);
|
||||
});
|
||||
|
||||
it("invalid bearer token gets 401", async () => {
|
||||
const resp = await app.inject({
|
||||
method: "GET",
|
||||
url: "/api/resources",
|
||||
headers: { authorization: "Bearer not-a-real-token" },
|
||||
});
|
||||
expect(resp.statusCode).toBe(401);
|
||||
});
|
||||
|
||||
it("admin role bypasses RBAC", async () => {
|
||||
const token = await seedSession("ADMIN");
|
||||
|
||||
const resp = await app.inject({
|
||||
method: "GET",
|
||||
url: "/api/resources",
|
||||
headers: { authorization: `Bearer ${token}` },
|
||||
});
|
||||
|
||||
expect(resp.statusCode).toBe(200);
|
||||
expect(resp.json()).toEqual([]);
|
||||
});
|
||||
|
||||
it("user with binding for env A is denied for resources in env B", async () => {
|
||||
const token = await seedSession("EDITOR");
|
||||
stores.groupMembers.push({ userId: "u-1", group: { name: "team-a" } });
|
||||
stores.rbacDefs.push({
|
||||
id: "rbac-1",
|
||||
name: "team-a-edit-on-env-a",
|
||||
subjects: [{ kind: "Group", name: "team-a" }],
|
||||
roleBindings: [{ role: "edit", environment: "env-a" }],
|
||||
});
|
||||
|
||||
// List in env-a → should pass RBAC (no env query so it's global view, but
|
||||
// the binding scope is environment-specific → for global list the binding
|
||||
// doesn't apply when an environment scope is set on the binding).
|
||||
// Smoke test the targeted denial: trying to create in env-b is rejected.
|
||||
const respB = await app.inject({
|
||||
method: "POST",
|
||||
url: "/api/resources",
|
||||
headers: { authorization: `Bearer ${token}` },
|
||||
payload: { kind: "database", name: "x", environmentId: "env-b", accountId: "acc-1" },
|
||||
});
|
||||
|
||||
expect(respB.statusCode).toBe(403);
|
||||
expect(respB.json().error).toMatch(/no matching role binding/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("v2 audit: correlation chain visible via /api/events", () => {
|
||||
let stores: Stores;
|
||||
let app: Awaited<ReturnType<typeof buildApp>>["app"];
|
||||
let auditService: AuditService;
|
||||
|
||||
beforeEach(async () => {
|
||||
stores = makeStores();
|
||||
const built = await buildApp(stores);
|
||||
app = built.app;
|
||||
auditService = built.auditService;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("emitted audit events are queryable by correlation id", async () => {
|
||||
// Seed admin so /api/events is accessible (it sits behind bearer auth)
|
||||
const loginResp = await app.inject({
|
||||
method: "POST",
|
||||
url: "/api/auth/login",
|
||||
payload: { email: "admin@itaz.eu", password: "pw" },
|
||||
});
|
||||
const token = loginResp.json().token;
|
||||
|
||||
// Force flush so the bootstrap event is in the DB
|
||||
await auditService.flushPending();
|
||||
|
||||
expect(stores.auditEvents.length).toBeGreaterThan(0);
|
||||
const bootstrap = stores.auditEvents.find((e) => e.eventKind === "auth_bootstrap")!;
|
||||
expect(bootstrap.correlationId).toMatch(/^corr_[a-f0-9]{16}$/);
|
||||
|
||||
// Query /api/events filtered by correlation id
|
||||
const queryResp = await app.inject({
|
||||
method: "GET",
|
||||
url: `/api/events?correlation=${bootstrap.correlationId}`,
|
||||
headers: { authorization: `Bearer ${token}` },
|
||||
});
|
||||
|
||||
expect(queryResp.statusCode).toBe(200);
|
||||
const events = queryResp.json() as Array<{ correlationId: string; eventKind: string }>;
|
||||
expect(events.length).toBe(1);
|
||||
expect(events[0]!.eventKind).toBe("auth_bootstrap");
|
||||
expect(events[0]!.correlationId).toBe(bootstrap.correlationId);
|
||||
});
|
||||
|
||||
it("explicit parent/child correlation chain is preserved across emits", async () => {
|
||||
const correlationId = auditService.createCorrelation();
|
||||
|
||||
auditService.emit({
|
||||
eventKind: "test_parent",
|
||||
source: "test",
|
||||
result: "success",
|
||||
correlationId,
|
||||
});
|
||||
auditService.emit({
|
||||
eventKind: "test_child",
|
||||
source: "test",
|
||||
result: "success",
|
||||
correlationId,
|
||||
parentEventId: "evt-1",
|
||||
});
|
||||
|
||||
await auditService.flushPending();
|
||||
|
||||
const chain = stores.auditEvents.filter((e) => e.correlationId === correlationId);
|
||||
expect(chain).toHaveLength(2);
|
||||
expect(chain.map((e) => e.eventKind).sort()).toEqual(["test_child", "test_parent"]);
|
||||
expect(chain.find((e) => e.eventKind === "test_child")!.parentEventId).toBe("evt-1");
|
||||
});
|
||||
});
|
||||
@@ -1,19 +1,21 @@
|
||||
// Hardening: Pod Security Standards, certificate check, log rotation.
|
||||
// Hardening: Pod Security Standards, certificate check, journald cap, storage.
|
||||
|
||||
import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
|
||||
import { runSequential } from "../utils.js";
|
||||
import { applyPodSecurityStandards } from "../operations/pod-security.js";
|
||||
import { checkCertExpiry } from "../operations/cert-check.js";
|
||||
import { configureLogRotation } from "../operations/log-rotation.js";
|
||||
import { configureJournaldLimits } from "../operations/journald-limits.js";
|
||||
import { configureLonghornDisk } from "../operations/longhorn-disk.js";
|
||||
|
||||
export const hardeningGroup: OperationGroup = {
|
||||
name: "hardening",
|
||||
description: "Pod security, certificate check, log rotation, storage",
|
||||
description: "Pod security, certificate check, journald cap, storage",
|
||||
operations: [
|
||||
{ name: "Apply Pod Security Standards", fn: applyPodSecurityStandards },
|
||||
{ name: "Check certificate expiry", fn: checkCertExpiry },
|
||||
{ name: "Configure log rotation", fn: configureLogRotation },
|
||||
{ name: "Decommission file-based audit logs", fn: configureLogRotation },
|
||||
{ name: "Configure journald disk cap", fn: configureJournaldLimits },
|
||||
{ name: "Configure Longhorn disk", fn: configureLonghornDisk },
|
||||
],
|
||||
};
|
||||
|
||||
@@ -1,21 +1,23 @@
|
||||
// Host preparation: kernel modules, sysctl, swap, firewall, SELinux.
|
||||
// Host preparation: kernel modules, sysctl, swap, storage, firewall, SELinux.
|
||||
|
||||
import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
|
||||
import { runSequential } from "../utils.js";
|
||||
import { loadKernelModules } from "../operations/kernel-modules.js";
|
||||
import { applyCisHardening } from "../operations/sysctl.js";
|
||||
import { disableSwap } from "../operations/swap.js";
|
||||
import { enableSwap } from "../operations/swap.js";
|
||||
import { growRancherLv } from "../operations/rancher-storage.js";
|
||||
import { disableFirewall } from "../operations/firewall.js";
|
||||
import { setSelinuxPermissive } from "../operations/selinux.js";
|
||||
import { enableIscsi } from "../operations/iscsi.js";
|
||||
|
||||
export const hostPrepGroup: OperationGroup = {
|
||||
name: "host-prep",
|
||||
description: "Prepare host for k3s: kernel modules, sysctl, swap, firewall, SELinux, iSCSI",
|
||||
description: "Prepare host for k3s: kernel modules, sysctl, swap, imageFs sizing, firewall, SELinux, iSCSI",
|
||||
operations: [
|
||||
{ name: "Load kernel modules", fn: loadKernelModules },
|
||||
{ name: "Apply CIS sysctl", fn: applyCisHardening },
|
||||
{ name: "Disable swap", fn: disableSwap },
|
||||
{ name: "Enable swap", fn: enableSwap },
|
||||
{ name: "Grow rancher LV", fn: growRancherLv },
|
||||
{ name: "Disable firewall", fn: disableFirewall },
|
||||
{ name: "Set SELinux permissive", fn: setSelinuxPermissive },
|
||||
{ name: "Enable iSCSI", fn: enableIscsi },
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
|
||||
import { runSequential } from "../utils.js";
|
||||
import { installCilium } from "../operations/cilium.js";
|
||||
import { installMultus } from "../operations/multus.js";
|
||||
import { installVlanSetup } from "../operations/vlan-setup.js";
|
||||
import { fixCoreDnsUpstream } from "../operations/dns-fix.js";
|
||||
import { applyDefaultNetworkPolicies } from "../operations/network-policy.js";
|
||||
|
||||
@@ -11,6 +13,11 @@ export const networkingGroup: OperationGroup = {
|
||||
description: "Install Cilium CNI, fix DNS, apply network policies",
|
||||
operations: [
|
||||
{ name: "Install Cilium CNI", fn: installCilium },
|
||||
// Multus + vlan-setup: give pods a second interface on VLAN 10 (macvlan)
|
||||
// for LAN device discovery (Matter/HomeKit mDNS). Must follow Cilium
|
||||
// (needs cni.exclusive=false + bpf.vlanBypass={10} from installCilium).
|
||||
{ name: "Install Multus CNI", fn: installMultus },
|
||||
{ name: "Install vlan-setup (lan10 + CNI plugins)", fn: installVlanSetup },
|
||||
{ name: "Fix CoreDNS upstream", fn: fixCoreDnsUpstream },
|
||||
{ name: "Apply network policies", fn: applyDefaultNetworkPolicies },
|
||||
],
|
||||
|
||||
@@ -76,7 +76,6 @@ sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config 2>/dev/nul
|
||||
# ── 5b. Create k3s config directory ──
|
||||
echo "[5/10] Writing k3s server configuration..."
|
||||
mkdir -p /etc/rancher/k3s
|
||||
mkdir -p /var/log/kubernetes
|
||||
|
||||
cat > /etc/rancher/k3s/config.yaml << 'K3S_CONFIG'
|
||||
# k3s server configuration — CIS hardened
|
||||
@@ -91,13 +90,10 @@ disable:
|
||||
- servicelb
|
||||
- traefik
|
||||
|
||||
# API server hardening
|
||||
# API server hardening (audit-log-path=- routes audit to journald via stdout)
|
||||
kube-apiserver-arg:
|
||||
- "anonymous-auth=false"
|
||||
- "audit-log-path=/var/log/kubernetes/audit.log"
|
||||
- "audit-log-maxage=30"
|
||||
- "audit-log-maxbackup=10"
|
||||
- "audit-log-maxsize=100"
|
||||
- "audit-log-path=-"
|
||||
- "audit-policy-file=/etc/rancher/k3s/audit-policy.yaml"
|
||||
- "enable-admission-plugins=NodeRestriction,PodSecurity"
|
||||
- "request-timeout=300s"
|
||||
@@ -219,7 +215,9 @@ echo " Using network device: \$DEFAULT_DEV"
|
||||
|
||||
KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \\
|
||||
--set kubeProxyReplacement=true \\
|
||||
--set ipam.mode=kubernetes \\
|
||||
--set ipam.mode=cluster-pool \\
|
||||
--set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \\
|
||||
--set ipam.operator.clusterPoolIPv4MaskSize=24 \\
|
||||
--set devices="\$DEFAULT_DEV" \\
|
||||
--set nodePort.directRoutingDevice="\$DEFAULT_DEV"
|
||||
|
||||
|
||||
@@ -78,9 +78,10 @@ export class K3sModule implements Module {
|
||||
return toModuleResult("install", [...prepResults, ...k3sResults], start);
|
||||
}
|
||||
|
||||
// Phase 3: Networking (server only — agents don't install Cilium)
|
||||
// Phase 3: Networking (initial server only — joining servers get Cilium via daemonset)
|
||||
let netResults: OperationResult[] = [];
|
||||
if (isServer) {
|
||||
const isJoiningServer = isServer && !!opCtx.config.k3sServerUrl;
|
||||
if (isServer && !isJoiningServer) {
|
||||
netResults = await runNetworking(opCtx);
|
||||
}
|
||||
|
||||
|
||||
@@ -38,12 +38,22 @@ export const installCilium: Operation = async (ctx): Promise<OperationResult> =>
|
||||
// Install Cilium
|
||||
// - No hardcoded devices: Cilium auto-detects per node (heterogeneous NICs like eno1 vs enP7s7)
|
||||
// - k8sServiceHost/Port: k3s agents proxy the API on 127.0.0.1:6444 (not 6443)
|
||||
// - cni.exclusive=false: required so Multus can install its CNI config alongside
|
||||
// Cilium (Cilium otherwise deletes any non-Cilium CNI conf).
|
||||
// - bpf.vlanBypass={10}: allow VLAN 10 (LoT) tagged traffic through the eBPF
|
||||
// host VLAN filter, so pods on a macvlan/VLAN-10 interface receive multicast
|
||||
// (Matter/mDNS ff02::fb + 224.0.0.251). Without this Cilium drops it
|
||||
// ("VLAN traffic disallowed by VLAN filter", bpf_host.c).
|
||||
const installResult = await ctx.ssh.exec(
|
||||
`KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \
|
||||
--set kubeProxyReplacement=true \
|
||||
--set ipam.mode=kubernetes \
|
||||
--set ipam.mode=cluster-pool \
|
||||
--set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \
|
||||
--set ipam.operator.clusterPoolIPv4MaskSize=24 \
|
||||
--set k8sServiceHost=127.0.0.1 \
|
||||
--set k8sServicePort=6444`,
|
||||
--set k8sServicePort=6444 \
|
||||
--set cni.exclusive=false \
|
||||
--set bpf.vlanBypass="{10}"`,
|
||||
{ timeoutMs: 300_000 },
|
||||
);
|
||||
if (installResult.exitCode !== 0) {
|
||||
|
||||
194
bastion/src/modules/modules/k3s/src/operations/etcd-recover.ts
Normal file
194
bastion/src/modules/modules/k3s/src/operations/etcd-recover.ts
Normal file
@@ -0,0 +1,194 @@
|
||||
// Recover a broken etcd member by removing it from the cluster, wiping its
|
||||
// local state, and restarting k3s so it rejoins as a fresh member.
|
||||
//
|
||||
// Use case: a node panics on startup with
|
||||
// "tocommit(N+1) is out of range [lastIndex(N)]. Was the raft log corrupted,
|
||||
// truncated, or lost?"
|
||||
// This means the local raft WAL is missing the last entry the leader thinks
|
||||
// the follower acknowledged (lost write, unclean shutdown, etc). The fix is
|
||||
// always the same and well-documented; this codifies it so we don't fumble
|
||||
// the procedure under pressure.
|
||||
//
|
||||
// Preconditions:
|
||||
// - At least one healthy peer is reachable so the cluster has quorum after
|
||||
// we remove the broken member. (For a 3-node cluster: 2 healthy. For a
|
||||
// 5-node: 3 healthy.) If quorum would be lost, this function refuses.
|
||||
// - SSH access to both the broken node and a healthy peer.
|
||||
// - etcdctl available on the healthy peer (k3s does not bundle it; the
|
||||
// procedure installs it on demand on Fedora).
|
||||
|
||||
import type { SshClient } from "../types.js";
|
||||
|
||||
const ETCD_TLS = {
|
||||
ca: "/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt",
|
||||
cert: "/var/lib/rancher/k3s/server/tls/etcd/server-client.crt",
|
||||
key: "/var/lib/rancher/k3s/server/tls/etcd/server-client.key",
|
||||
} as const;
|
||||
|
||||
const SSH_TIMEOUT = 60_000;
|
||||
|
||||
export interface RecoverEtcdMemberOptions {
|
||||
/** SSH client for the broken node (the one panicking). */
|
||||
broken: SshClient;
|
||||
/** SSH client for any healthy server peer in the same cluster. */
|
||||
peer: SshClient;
|
||||
/** Hostname (k8s node name) of the broken node. Used to find its etcd member id. */
|
||||
brokenHostname: string;
|
||||
/** Logger for progress output. */
|
||||
log?: (msg: string) => void;
|
||||
}
|
||||
|
||||
export interface RecoverEtcdMemberResult {
|
||||
success: boolean;
|
||||
changed: boolean;
|
||||
message: string;
|
||||
/** New etcd member id assigned after rejoin (when known). */
|
||||
newMemberId?: string;
|
||||
/** Old etcd member id that was removed. */
|
||||
removedMemberId?: string;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
function etcdctl(subcmd: string): string {
|
||||
return [
|
||||
"ETCDCTL_API=3 etcdctl",
|
||||
`--cacert=${ETCD_TLS.ca}`,
|
||||
`--cert=${ETCD_TLS.cert}`,
|
||||
`--key=${ETCD_TLS.key}`,
|
||||
"--endpoints=https://127.0.0.1:2379",
|
||||
"--command-timeout=10s",
|
||||
subcmd,
|
||||
].join(" ");
|
||||
}
|
||||
|
||||
async function ensureEtcdctl(peer: SshClient): Promise<void> {
|
||||
const probe = await peer.exec("command -v etcdctl 2>/dev/null", { timeoutMs: 5_000 });
|
||||
if (probe.exitCode === 0 && probe.stdout.trim()) return;
|
||||
// Best-effort install on Fedora. If the host isn't dnf-based, surface the
|
||||
// error to the caller via the next etcdctl invocation.
|
||||
await peer.exec("dnf install -y etcd 2>&1", { timeoutMs: 120_000 });
|
||||
}
|
||||
|
||||
async function getMemberList(peer: SshClient): Promise<Array<{ id: string; name: string }>> {
|
||||
const result = await peer.exec(etcdctl("member list"), { timeoutMs: SSH_TIMEOUT });
|
||||
if (result.exitCode !== 0) {
|
||||
throw new Error(`etcdctl member list failed: ${result.stderr || result.stdout}`);
|
||||
}
|
||||
// Format: <hex-id>, started, <name>, <peer-urls>, <client-urls>, <isLearner>
|
||||
return result.stdout
|
||||
.split("\n")
|
||||
.map((line) => line.trim())
|
||||
.filter(Boolean)
|
||||
.map((line) => {
|
||||
const [id, , name] = line.split(",").map((p) => p.trim());
|
||||
return { id: id ?? "", name: name ?? "" };
|
||||
})
|
||||
.filter((m) => m.id);
|
||||
}
|
||||
|
||||
export async function recoverEtcdMember(
|
||||
opts: RecoverEtcdMemberOptions,
|
||||
): Promise<RecoverEtcdMemberResult> {
|
||||
const log = opts.log ?? (() => {});
|
||||
|
||||
try {
|
||||
log(`Looking up etcd member id for ${opts.brokenHostname} via peer...`);
|
||||
await ensureEtcdctl(opts.peer);
|
||||
|
||||
const members = await getMemberList(opts.peer);
|
||||
if (members.length < 3) {
|
||||
return {
|
||||
success: false,
|
||||
changed: false,
|
||||
message: "Refusing to remove a member from a cluster with <3 members (quorum would be lost)",
|
||||
error: `member count = ${members.length}`,
|
||||
};
|
||||
}
|
||||
|
||||
// Member names are <hostname>-<random-suffix>; match by hostname prefix.
|
||||
const broken = members.find((m) => m.name.startsWith(opts.brokenHostname));
|
||||
if (!broken) {
|
||||
return {
|
||||
success: false,
|
||||
changed: false,
|
||||
message: `No etcd member found matching hostname ${opts.brokenHostname}`,
|
||||
error: `members: ${members.map((m) => m.name).join(", ")}`,
|
||||
};
|
||||
}
|
||||
log(`Broken member: ${broken.id} (${broken.name})`);
|
||||
|
||||
log("Step 1/4: stopping k3s on broken node");
|
||||
await opts.broken.exec("systemctl stop k3s 2>&1", { timeoutMs: SSH_TIMEOUT });
|
||||
|
||||
log("Step 2/4: removing broken etcd member from cluster");
|
||||
const remove = await opts.peer.exec(
|
||||
etcdctl(`member remove ${broken.id}`),
|
||||
{ timeoutMs: SSH_TIMEOUT },
|
||||
);
|
||||
if (remove.exitCode !== 0) {
|
||||
return {
|
||||
success: false,
|
||||
changed: false,
|
||||
message: "etcdctl member remove failed",
|
||||
error: remove.stderr || remove.stdout,
|
||||
removedMemberId: broken.id,
|
||||
};
|
||||
}
|
||||
|
||||
log("Step 3/4: archiving corrupt etcd state and stale TLS/cred dirs on broken node");
|
||||
const ts = Math.floor(Date.now() / 1000);
|
||||
await opts.broken.exec(
|
||||
[
|
||||
`mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.corrupt-${ts} 2>/dev/null || true`,
|
||||
"rm -rf /var/lib/rancher/k3s/server/tls /var/lib/rancher/k3s/server/cred",
|
||||
].join(" && "),
|
||||
{ timeoutMs: SSH_TIMEOUT },
|
||||
);
|
||||
|
||||
log("Step 4/4: starting k3s on broken node — it will rejoin");
|
||||
await opts.broken.exec("systemctl start k3s 2>&1", { timeoutMs: SSH_TIMEOUT });
|
||||
|
||||
// Poll for rejoin. The new member-id is what the cluster assigns on join.
|
||||
let newMemberId: string | undefined;
|
||||
for (let i = 0; i < 60; i++) {
|
||||
await new Promise((r) => setTimeout(r, 5_000));
|
||||
try {
|
||||
const after = await getMemberList(opts.peer);
|
||||
const rejoined = after.find(
|
||||
(m) => m.name.startsWith(opts.brokenHostname) && m.id !== broken.id,
|
||||
);
|
||||
if (rejoined) {
|
||||
newMemberId = rejoined.id;
|
||||
break;
|
||||
}
|
||||
} catch {
|
||||
// peer may briefly be unreachable mid-rejoin — keep polling
|
||||
}
|
||||
}
|
||||
|
||||
if (!newMemberId) {
|
||||
return {
|
||||
success: false,
|
||||
changed: true,
|
||||
message: "k3s started but new member did not appear in cluster within 5 minutes",
|
||||
removedMemberId: broken.id,
|
||||
};
|
||||
}
|
||||
|
||||
log(`Rejoined as ${newMemberId}`);
|
||||
return {
|
||||
success: true,
|
||||
changed: true,
|
||||
message: `Recovered: removed ${broken.id}, rejoined as ${newMemberId}`,
|
||||
removedMemberId: broken.id,
|
||||
newMemberId,
|
||||
};
|
||||
} catch (err) {
|
||||
return {
|
||||
success: false,
|
||||
changed: false,
|
||||
message: "Recovery failed",
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
export { loadKernelModules } from "./kernel-modules.js";
|
||||
export { applyCisHardening } from "./sysctl.js";
|
||||
export { disableSwap } from "./swap.js";
|
||||
export { enableSwap } from "./swap.js";
|
||||
export { growRancherLv } from "./rancher-storage.js";
|
||||
export { enableIscsi } from "./iscsi.js";
|
||||
export { disableFirewall } from "./firewall.js";
|
||||
export { setSelinuxPermissive } from "./selinux.js";
|
||||
@@ -9,9 +10,17 @@ export { writeAuditPolicy } from "./audit-policy.js";
|
||||
export { cleanupStaleCni } from "./cni-cleanup.js";
|
||||
export { installK3sBinary } from "./k3s-install.js";
|
||||
export { installCilium } from "./cilium.js";
|
||||
export { installMultus } from "./multus.js";
|
||||
export { installVlanSetup } from "./vlan-setup.js";
|
||||
export { fixCoreDnsUpstream } from "./dns-fix.js";
|
||||
export { configureLogRotation } from "./log-rotation.js";
|
||||
export { configureJournaldLimits } from "./journald-limits.js";
|
||||
export { applyDefaultNetworkPolicies } from "./network-policy.js";
|
||||
export { applyPodSecurityStandards } from "./pod-security.js";
|
||||
export { checkCertExpiry } from "./cert-check.js";
|
||||
export { configureLonghornDisk } from "./longhorn-disk.js";
|
||||
export { recoverEtcdMember } from "./etcd-recover.js";
|
||||
export type {
|
||||
RecoverEtcdMemberOptions,
|
||||
RecoverEtcdMemberResult,
|
||||
} from "./etcd-recover.js";
|
||||
|
||||
@@ -13,10 +13,11 @@ export const enableIscsi: Operation = async (ctx): Promise<OperationResult> => {
|
||||
|
||||
// Install the package (detect distro)
|
||||
const osRelease = await ctx.ssh.exec("cat /etc/os-release", sshOpts(ctx));
|
||||
const isFedora = osRelease.stdout.includes("fedora") || osRelease.stdout.includes("rhel") || osRelease.stdout.includes("centos");
|
||||
const osLower = osRelease.stdout.toLowerCase();
|
||||
const isFedora = osLower.includes("fedora") || osLower.includes("rhel") || osLower.includes("centos");
|
||||
|
||||
const pkg = isFedora ? "iscsi-initiator-utils" : "open-iscsi";
|
||||
const installCmd = isFedora ? `dnf install -y ${pkg}` : `apt-get install -y ${pkg}`;
|
||||
const installCmd = isFedora ? `sudo dnf install -y ${pkg}` : `sudo apt-get install -y ${pkg}`;
|
||||
|
||||
const install = await ctx.ssh.exec(installCmd, { timeoutMs: 120_000 });
|
||||
if (install.exitCode !== 0) {
|
||||
@@ -24,7 +25,7 @@ export const enableIscsi: Operation = async (ctx): Promise<OperationResult> => {
|
||||
}
|
||||
|
||||
// Enable and start
|
||||
await ctx.ssh.exec("systemctl enable --now iscsid", sshOpts(ctx));
|
||||
await ctx.ssh.exec("sudo systemctl enable --now iscsid", sshOpts(ctx));
|
||||
|
||||
return { success: true, changed: true, message: `Installed ${pkg} and enabled iscsid` };
|
||||
};
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
// Cap journald disk usage so audit logs (which now flow through journald via
|
||||
// kube-apiserver's stdout) cannot fill /var/log. Default journald uses up to
|
||||
// 10% of the filesystem, capped at 4 GB. In a /var/log of ~10 GB shared with
|
||||
// other services, that's still room for audit volume to evict useful logs.
|
||||
// 2 GB / 200 MB-per-file is a comfortable middle.
|
||||
|
||||
import type { Operation, OperationResult } from "../types.js";
|
||||
import { sshOpts, writeRemoteFile } from "../utils.js";
|
||||
|
||||
const DROPIN_CONTENT = `[Journal]
|
||||
SystemMaxUse=2G
|
||||
SystemKeepFree=1G
|
||||
SystemMaxFileSize=200M
|
||||
`;
|
||||
|
||||
const DROPIN_PATH = "/etc/systemd/journald.conf.d/10-k3s-audit-cap.conf";
|
||||
|
||||
export const configureJournaldLimits: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const changed = await writeRemoteFile(ctx, DROPIN_PATH, DROPIN_CONTENT);
|
||||
if (changed) {
|
||||
// Reload journald so the new limit applies without a reboot.
|
||||
await ctx.ssh.exec(
|
||||
"systemctl kill --signal=SIGUSR2 systemd-journald 2>/dev/null; " +
|
||||
"systemctl restart systemd-journald 2>&1 || true",
|
||||
sshOpts(ctx),
|
||||
);
|
||||
}
|
||||
return {
|
||||
success: true,
|
||||
changed,
|
||||
message: changed ? "journald limits configured (2 GB cap)" : "journald limits already configured",
|
||||
};
|
||||
};
|
||||
@@ -9,7 +9,18 @@ function isServerRole(role: string): boolean {
|
||||
|
||||
function generateServerConfig(config: K3sConfig): string {
|
||||
const tlsSans = [config.hostname, config.ip, ...(config.tlsSans ?? [])];
|
||||
return `# k3s server configuration — CIS hardened
|
||||
const isJoining = !!config.k3sServerUrl;
|
||||
const clusterLines = isJoining
|
||||
? `server: "${config.k3sServerUrl}"\ntoken: "${config.k3sToken}"`
|
||||
: "cluster-init: true";
|
||||
// audit-log-path=- routes audit events to k3s.service's stdout, which systemd
|
||||
// forwards to journald. journald enforces its own size caps (see
|
||||
// configureJournaldLimits) so audit volume cannot fill the disk. File-based
|
||||
// audit logs led to /var/log/kubernetes growing to 7+ GB because apiserver's
|
||||
// own rotation produced files that any logrotate glob would double-rotate
|
||||
// and never expire.
|
||||
return `# k3s server configuration — CIS hardened, etcd HA
|
||||
${clusterLines}
|
||||
protect-kernel-defaults: true
|
||||
secrets-encryption: true
|
||||
write-kubeconfig-mode: "0640"
|
||||
@@ -25,10 +36,7 @@ node-label:
|
||||
|
||||
kube-apiserver-arg:
|
||||
- "anonymous-auth=false"
|
||||
- "audit-log-path=/var/log/kubernetes/audit.log"
|
||||
- "audit-log-maxage=30"
|
||||
- "audit-log-maxbackup=10"
|
||||
- "audit-log-maxsize=100"
|
||||
- "audit-log-path=-"
|
||||
- "audit-policy-file=/etc/rancher/k3s/audit-policy.yaml"
|
||||
- "enable-admission-plugins=NodeRestriction,PodSecurity"
|
||||
- "request-timeout=300s"
|
||||
@@ -56,7 +64,7 @@ kubelet-arg:
|
||||
}
|
||||
|
||||
export const writeK3sConfig: Operation = async (ctx): Promise<OperationResult> => {
|
||||
await ctx.ssh.exec("mkdir -p /etc/rancher/k3s /var/log/kubernetes", sshOpts(ctx));
|
||||
await ctx.ssh.exec("mkdir -p /etc/rancher/k3s", sshOpts(ctx));
|
||||
|
||||
const content = isServerRole(ctx.config.role)
|
||||
? generateServerConfig(ctx.config)
|
||||
|
||||
@@ -15,8 +15,21 @@ export const installK3sBinary: Operation = async (ctx): Promise<OperationResult>
|
||||
const alreadyInstalled = version.exitCode === 0;
|
||||
|
||||
if (isServer) {
|
||||
// Clean stale server state when joining an existing cluster
|
||||
// (TLS certs from a previous run cause "newer than datastore" fatal error)
|
||||
if (ctx.config.k3sServerUrl && ctx.config.k3sToken) {
|
||||
await ctx.ssh.exec(
|
||||
"rm -rf /var/lib/rancher/k3s/server/tls /var/lib/rancher/k3s/server/cred /var/lib/rancher/k3s/server/db",
|
||||
sshOpts(ctx),
|
||||
);
|
||||
}
|
||||
|
||||
// If joining an existing cluster, pass K3S_URL and K3S_TOKEN
|
||||
const joinEnv = ctx.config.k3sServerUrl && ctx.config.k3sToken
|
||||
? `K3S_URL="${ctx.config.k3sServerUrl}" K3S_TOKEN="${ctx.config.k3sToken}"`
|
||||
: "";
|
||||
const result = await ctx.ssh.exec(
|
||||
'curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" INSTALL_K3S_SKIP_SELINUX_RPM=true sh -',
|
||||
`curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" INSTALL_K3S_SKIP_SELINUX_RPM=true ${joinEnv} sh -`,
|
||||
{ timeoutMs: 300_000 },
|
||||
);
|
||||
if (result.exitCode !== 0) {
|
||||
|
||||
@@ -1,25 +1,44 @@
|
||||
// Configure log rotation for k3s.
|
||||
// Decommission file-based k8s audit logging in favor of journald.
|
||||
//
|
||||
// Earlier versions wrote audit events to /var/log/kubernetes/audit.log and
|
||||
// rotated them with a logrotate rule. Two failure modes followed: kube-apiserver
|
||||
// rotated internally (audit-{ts}.log), the *.log glob in logrotate
|
||||
// double-rotated those (-{date}), and the resulting filename matched no
|
||||
// retention policy, so the directory grew unbounded (we observed 7+ GB).
|
||||
//
|
||||
// k3s now sets audit-log-path=- so audit goes to stdout → journald, which
|
||||
// enforces SystemMaxUse caps. This operation removes the obsolete logrotate
|
||||
// rule and reaps any audit files left behind by the old setup. Idempotent: on
|
||||
// fresh installs everything is already absent and the operation is a no-op.
|
||||
|
||||
import type { Operation, OperationResult } from "../types.js";
|
||||
import { writeRemoteFile } from "../utils.js";
|
||||
import { sshOpts } from "../utils.js";
|
||||
|
||||
const LOGROTATE_CONFIG = `/var/log/kubernetes/*.log {
|
||||
daily
|
||||
rotate 14
|
||||
compress
|
||||
delaycompress
|
||||
missingok
|
||||
notifempty
|
||||
copytruncate
|
||||
maxsize 100M
|
||||
}`;
|
||||
const REMOVE_LOGROTATE = "rm -f /etc/logrotate.d/k3s";
|
||||
|
||||
// Bounded by a max-depth and explicit name pattern so we never reach outside
|
||||
// the deprecated audit-log directory.
|
||||
const REAP_OLD_AUDIT_FILES =
|
||||
"find /var/log/kubernetes -maxdepth 1 -type f " +
|
||||
"\\( -name 'audit*.log*' -o -name 'audit-*.log' \\) " +
|
||||
"-delete 2>/dev/null; " +
|
||||
"rmdir /var/log/kubernetes 2>/dev/null; true";
|
||||
|
||||
export const configureLogRotation: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const changed = await writeRemoteFile(ctx, "/etc/logrotate.d/k3s", LOGROTATE_CONFIG);
|
||||
const before = await ctx.ssh.exec(
|
||||
"test -e /etc/logrotate.d/k3s -o -d /var/log/kubernetes && echo present || echo absent",
|
||||
sshOpts(ctx),
|
||||
);
|
||||
const wasPresent = before.stdout.trim() === "present";
|
||||
|
||||
await ctx.ssh.exec(REMOVE_LOGROTATE, sshOpts(ctx));
|
||||
await ctx.ssh.exec(REAP_OLD_AUDIT_FILES, sshOpts(ctx));
|
||||
|
||||
return {
|
||||
success: true,
|
||||
changed,
|
||||
message: changed ? "Log rotation configured" : "Log rotation already configured",
|
||||
changed: wasPresent,
|
||||
message: wasPresent
|
||||
? "Removed legacy file-based audit logging (now via journald)"
|
||||
: "No legacy audit log artifacts present",
|
||||
};
|
||||
};
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
|
||||
import type { Operation, OperationResult } from "../types.js";
|
||||
import { sshOpts } from "../utils.js";
|
||||
import { sshExec as remoteSshExec } from "../../../../src/ssh.js";
|
||||
|
||||
export const configureLonghornDisk: Operation = async (ctx): Promise<OperationResult> => {
|
||||
// Check if /var/lib/longhorn exists on this node
|
||||
@@ -15,12 +16,11 @@ export const configureLonghornDisk: Operation = async (ctx): Promise<OperationRe
|
||||
const nodeNameResult = await ctx.ssh.exec("hostname -f 2>/dev/null || hostname", sshOpts(ctx));
|
||||
const nodeName = nodeNameResult.stdout.trim();
|
||||
|
||||
// Apply the annotation via kubectl (works on server nodes, or via KUBECONFIG on agents)
|
||||
const kubectlPrefix = "k3s kubectl";
|
||||
const annotation = JSON.stringify([{ path: "/var/lib/longhorn", allowScheduling: true }]);
|
||||
|
||||
// Try kubectl locally first (works on server nodes)
|
||||
const result = await ctx.ssh.exec(
|
||||
`${kubectlPrefix} annotate node "${nodeName}" "node.longhorn.io/default-disks-config=${annotation}" --overwrite 2>&1 || true`,
|
||||
`k3s kubectl annotate node "${nodeName}" "node.longhorn.io/default-disks-config=${annotation}" --overwrite 2>&1 || true`,
|
||||
sshOpts(ctx),
|
||||
);
|
||||
|
||||
@@ -28,7 +28,23 @@ export const configureLonghornDisk: Operation = async (ctx): Promise<OperationRe
|
||||
return { success: true, changed: true, message: `Longhorn disk annotation applied to ${nodeName}` };
|
||||
}
|
||||
|
||||
// If kubectl isn't available (agent node without server access), that's OK —
|
||||
// the label is set, annotation can be applied from the server later
|
||||
// For worker/agent nodes without local kubectl: apply via the server
|
||||
if (ctx.config.k3sServerUrl) {
|
||||
// The CLI has SSH access to the server — use sshExec from there
|
||||
const serverHost = new URL(ctx.config.k3sServerUrl).hostname;
|
||||
try {
|
||||
const remoteResult = await remoteSshExec(
|
||||
serverHost, "root",
|
||||
`k3s kubectl annotate node "${nodeName}" "node.longhorn.io/default-disks-config=${annotation}" --overwrite`,
|
||||
{ ...(ctx.ssh.keyPath ? { keyPath: ctx.ssh.keyPath } : {}), timeoutMs: 15_000 },
|
||||
);
|
||||
if (remoteResult.stdout.includes("annotated") || remoteResult.stdout.includes("unchanged")) {
|
||||
return { success: true, changed: true, message: `Longhorn disk annotation applied to ${nodeName} (via server)` };
|
||||
}
|
||||
} catch {
|
||||
// Fall through to manual instruction
|
||||
}
|
||||
}
|
||||
|
||||
return { success: true, changed: false, message: "Longhorn disk label set (annotation requires server kubectl)" };
|
||||
};
|
||||
|
||||
34
bastion/src/modules/modules/k3s/src/operations/multus.ts
Normal file
34
bastion/src/modules/modules/k3s/src/operations/multus.ts
Normal file
@@ -0,0 +1,34 @@
|
||||
// Install Multus CNI (thick plugin) — the meta-CNI that lets pods attach an
|
||||
// extra interface (macvlan on VLAN 10) alongside Cilium, via a
|
||||
// NetworkAttachmentDefinition. Required for Home Assistant's LAN presence
|
||||
// (Matter/HomeKit mDNS discovery). Cilium must be installed with
|
||||
// cni.exclusive=false first (see cilium.ts) or it deletes Multus's CNI conf.
|
||||
|
||||
import type { Operation, OperationResult } from "../types.js";
|
||||
import { sshOpts } from "../utils.js";
|
||||
|
||||
const MULTUS_VERSION = "v4.1.4";
|
||||
const MULTUS_MANIFEST = `https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/${MULTUS_VERSION}/deployments/multus-daemonset-thick.yml`;
|
||||
|
||||
export const installMultus: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const K = "KUBECONFIG=/etc/rancher/k3s/k3s.yaml";
|
||||
|
||||
// Idempotent: skip if the Multus DaemonSet is already present.
|
||||
const check = await ctx.ssh.exec(
|
||||
`${K} kubectl -n kube-system get ds kube-multus-ds -o name 2>/dev/null`,
|
||||
sshOpts(ctx),
|
||||
);
|
||||
if (check.exitCode === 0 && check.stdout.includes("kube-multus-ds")) {
|
||||
return { success: true, changed: false, message: `Multus already installed (${MULTUS_VERSION})` };
|
||||
}
|
||||
|
||||
const apply = await ctx.ssh.exec(
|
||||
`${K} kubectl apply -f ${MULTUS_MANIFEST}`,
|
||||
{ ...sshOpts(ctx), timeoutMs: 120_000 },
|
||||
);
|
||||
if (apply.exitCode !== 0) {
|
||||
return { success: false, changed: false, message: "Failed to apply Multus manifest", error: apply.stderr };
|
||||
}
|
||||
|
||||
return { success: true, changed: true, message: `Installed Multus ${MULTUS_VERSION} (thick)` };
|
||||
};
|
||||
@@ -0,0 +1,48 @@
|
||||
// Grow the labvg/rancher LV (k3s image store / imageFs) to 120G.
|
||||
// 2026-08 incident: the original 20G LV sat at 85% used from steady-state
|
||||
// images alone, so one ~5G image pull tripped imagefs eviction and evicted
|
||||
// unrelated pods. Fresh installs are sized at 120G by the kickstart; this op
|
||||
// covers nodes installed before that change and vanilla nodes converted to
|
||||
// k8s later. Never removes or shrinks anything — if the VG lacks free space
|
||||
// (e.g. a longhorn --grow LV consumed it), it reports and moves on.
|
||||
|
||||
import type { Operation, OperationResult } from "../types.js";
|
||||
import { sshOpts } from "../utils.js";
|
||||
|
||||
const RANCHER_LV = "labvg/rancher";
|
||||
const TARGET_MIB = 122880; // 120G
|
||||
|
||||
export const growRancherLv: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const lv = await ctx.ssh.exec(
|
||||
`lvs --noheadings --units m --nosuffix -o lv_size ${RANCHER_LV} 2>/dev/null || true`,
|
||||
sshOpts(ctx),
|
||||
);
|
||||
const sizeMib = Number.parseFloat(lv.stdout.trim());
|
||||
if (Number.isNaN(sizeMib)) {
|
||||
return { success: true, changed: false, message: "No labvg/rancher LV — imageFs shares /var, skipping" };
|
||||
}
|
||||
if (sizeMib >= TARGET_MIB) {
|
||||
return { success: true, changed: false, message: `rancher LV already ${Math.round(sizeMib / 1024)}G` };
|
||||
}
|
||||
|
||||
const vg = await ctx.ssh.exec(`vgs --noheadings --units m --nosuffix -o vg_free labvg`, sshOpts(ctx));
|
||||
const freeMib = Number.parseFloat(vg.stdout.trim());
|
||||
const neededMib = TARGET_MIB - sizeMib;
|
||||
if (Number.isNaN(freeMib) || freeMib < neededMib) {
|
||||
return {
|
||||
success: true,
|
||||
changed: false,
|
||||
message: `VG labvg has ${Math.floor((Number.isNaN(freeMib) ? 0 : freeMib) / 1024)}G free — ` +
|
||||
`need ${Math.ceil(neededMib / 1024)}G to grow rancher LV to 120G (manual LV rebuild required)`,
|
||||
};
|
||||
}
|
||||
|
||||
await ctx.ssh.exec(`lvextend -L ${TARGET_MIB}m /dev/${RANCHER_LV}`, sshOpts(ctx));
|
||||
await ctx.ssh.exec(`xfs_growfs /var/lib/rancher`, sshOpts(ctx));
|
||||
|
||||
return {
|
||||
success: true,
|
||||
changed: true,
|
||||
message: `rancher LV grown ${Math.round(sizeMib / 1024)}G → 120G`,
|
||||
};
|
||||
};
|
||||
@@ -1,22 +1,40 @@
|
||||
// Disable swap (CIS requirement for k3s).
|
||||
// Enable swap so memory pressure spills to disk instead of OOM-killing.
|
||||
// kubelet runs with failSwapOn=false (k3s default); zram stays the fast tier,
|
||||
// the labvg-swap LV is the overflow tier. Replaces the old CIS-style
|
||||
// disableSwap op — a kernel OOM kill of a node daemon is worse than slow swap.
|
||||
|
||||
import type { Operation, OperationResult } from "../types.js";
|
||||
import { sshOpts } from "../utils.js";
|
||||
|
||||
export const disableSwap: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const check = await ctx.ssh.exec("swapon --show --noheadings", sshOpts(ctx));
|
||||
const active = check.stdout.trim().length > 0;
|
||||
const SWAP_DEV = "/dev/mapper/labvg-swap";
|
||||
|
||||
if (active) {
|
||||
await ctx.ssh.exec("swapoff -a", sshOpts(ctx));
|
||||
export const enableSwap: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const lv = await ctx.ssh.exec(`test -b ${SWAP_DEV} && echo yes || echo no`, sshOpts(ctx));
|
||||
if (lv.stdout.trim() !== "yes") {
|
||||
return { success: true, changed: false, message: "No labvg-swap LV — skipping swap enable" };
|
||||
}
|
||||
|
||||
// Remove swap entries from fstab permanently
|
||||
await ctx.ssh.exec("sed -i '/\\sswap\\s/d' /etc/fstab", sshOpts(ctx));
|
||||
const active = await ctx.ssh.exec(
|
||||
`grep -q "^$(readlink -f ${SWAP_DEV}) " /proc/swaps && echo on || echo off`,
|
||||
sshOpts(ctx),
|
||||
);
|
||||
const wasOff = active.stdout.trim() !== "on";
|
||||
|
||||
if (wasOff) {
|
||||
// Format if the LV was never (or wrongly) initialised, then activate
|
||||
await ctx.ssh.exec(`blkid ${SWAP_DEV} | grep -q 'TYPE="swap"' || mkswap ${SWAP_DEV}`, sshOpts(ctx));
|
||||
await ctx.ssh.exec(`swapon ${SWAP_DEV}`, sshOpts(ctx));
|
||||
}
|
||||
|
||||
// Persist across reboots (idempotent)
|
||||
await ctx.ssh.exec(
|
||||
`grep -q "labvg-swap" /etc/fstab || echo "${SWAP_DEV} none swap defaults 0 0" >> /etc/fstab`,
|
||||
sshOpts(ctx),
|
||||
);
|
||||
|
||||
return {
|
||||
success: true,
|
||||
changed: active,
|
||||
message: active ? "Swap disabled" : "Swap already disabled",
|
||||
changed: wasOff,
|
||||
message: wasOff ? "LV swap enabled" : "LV swap already active",
|
||||
};
|
||||
};
|
||||
|
||||
96
bastion/src/modules/modules/k3s/src/operations/vlan-setup.ts
Normal file
96
bastion/src/modules/modules/k3s/src/operations/vlan-setup.ts
Normal file
@@ -0,0 +1,96 @@
|
||||
// vlan-setup DaemonSet — the node-level half of the macvlan/VLAN-10 story.
|
||||
// On every node it (1) installs the reference CNI plugins (macvlan/ipvlan/
|
||||
// static/host-local/vlan/tuning) into /opt/cni/bin if missing, and (2) creates
|
||||
// a `lan10` VLAN-10 sub-interface on the primary NIC that macvlan
|
||||
// NetworkAttachmentDefinitions use as their master. Idempotent + self-healing
|
||||
// (re-creates lan10 if it disappears). Paired with Multus (multus.ts) + Cilium
|
||||
// bpf.vlanBypass={10} (cilium.ts).
|
||||
|
||||
import type { Operation, OperationResult } from "../types.js";
|
||||
import { sshOpts } from "../utils.js";
|
||||
|
||||
const MANIFEST = `apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: macvlan-sys
|
||||
labels:
|
||||
pod-security.kubernetes.io/enforce: privileged
|
||||
pod-security.kubernetes.io/audit: privileged
|
||||
pod-security.kubernetes.io/warn: privileged
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: vlan-setup
|
||||
namespace: macvlan-sys
|
||||
spec:
|
||||
selector:
|
||||
matchLabels: { app: vlan-setup }
|
||||
template:
|
||||
metadata:
|
||||
labels: { app: vlan-setup }
|
||||
spec:
|
||||
hostNetwork: true
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
containers:
|
||||
- name: vlan
|
||||
image: nicolaka/netshoot
|
||||
securityContext:
|
||||
privileged: true
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -x
|
||||
# install reference CNI plugins (macvlan/ipvlan/static/host-local) if missing
|
||||
if [ ! -f /host/opt/cni/bin/macvlan ] || [ ! -f /host/opt/cni/bin/ipvlan ]; then
|
||||
case "$(uname -m)" in x86_64) A=amd64;; aarch64) A=arm64;; *) A=amd64;; esac
|
||||
curl -sSL "https://github.com/containernetworking/plugins/releases/download/v1.5.1/cni-plugins-linux-$A-v1.5.1.tgz" -o /tmp/cni.tgz
|
||||
tar -xzf /tmp/cni.tgz -C /host/opt/cni/bin ./macvlan ./ipvlan ./static ./host-local ./vlan ./tuning
|
||||
fi
|
||||
# detect the primary NIC (default route dev, else the one holding 192.168.8.x)
|
||||
NIC="$(ip -o -4 route show default 2>/dev/null | awk '{print $5; exit}')"
|
||||
[ -z "$NIC" ] && NIC="$(ip -o -4 addr show 2>/dev/null | awk '/192\\.168\\.8\\./{print $2; exit}')"
|
||||
echo "primary NIC = $NIC"
|
||||
while true; do
|
||||
if [ -n "$NIC" ]; then
|
||||
ip link show lan10 >/dev/null 2>&1 || ip link add link "$NIC" name lan10 type vlan id 10
|
||||
ip link set lan10 up
|
||||
# NIC-driver workarounds for VLAN multicast RX
|
||||
ip link set "$NIC" allmulticast on 2>/dev/null
|
||||
ethtool -K "$NIC" rxvlan off rx-vlan-filter off 2>/dev/null
|
||||
fi
|
||||
sleep 30
|
||||
done
|
||||
volumeMounts:
|
||||
- name: cnibin
|
||||
mountPath: /host/opt/cni/bin
|
||||
volumes:
|
||||
- name: cnibin
|
||||
hostPath:
|
||||
path: /opt/cni/bin
|
||||
`;
|
||||
|
||||
export const installVlanSetup: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const K = "KUBECONFIG=/etc/rancher/k3s/k3s.yaml";
|
||||
|
||||
const check = await ctx.ssh.exec(
|
||||
`${K} kubectl -n macvlan-sys get ds vlan-setup -o name 2>/dev/null`,
|
||||
sshOpts(ctx),
|
||||
);
|
||||
if (check.exitCode === 0 && check.stdout.includes("vlan-setup")) {
|
||||
return { success: true, changed: false, message: "vlan-setup DaemonSet already installed" };
|
||||
}
|
||||
|
||||
const b64 = Buffer.from(MANIFEST).toString("base64");
|
||||
const apply = await ctx.ssh.exec(
|
||||
`echo ${b64} | base64 -d | ${K} kubectl apply -f -`,
|
||||
{ ...sshOpts(ctx), timeoutMs: 60_000 },
|
||||
);
|
||||
if (apply.exitCode !== 0) {
|
||||
return { success: false, changed: false, message: "Failed to apply vlan-setup DaemonSet", error: apply.stderr };
|
||||
}
|
||||
|
||||
return { success: true, changed: true, message: "Installed vlan-setup DaemonSet (lan10 + CNI plugins)" };
|
||||
};
|
||||
@@ -71,9 +71,14 @@ describe("k3s install script — server role", () => {
|
||||
expect(script).toContain("enable-admission-plugins=NodeRestriction,PodSecurity");
|
||||
});
|
||||
|
||||
it("configures audit logging", () => {
|
||||
expect(script).toContain("audit-log-path=/var/log/kubernetes/audit.log");
|
||||
expect(script).toContain("audit-log-maxage=30");
|
||||
it("configures audit logging via journald (stdout)", () => {
|
||||
expect(script).toContain("audit-log-path=-");
|
||||
// file-based fields and the now-obsolete log directory must be gone
|
||||
expect(script).not.toContain("/var/log/kubernetes/audit.log");
|
||||
expect(script).not.toContain("audit-log-maxage");
|
||||
expect(script).not.toContain("audit-log-maxbackup");
|
||||
expect(script).not.toContain("audit-log-maxsize");
|
||||
expect(script).not.toContain("mkdir -p /var/log/kubernetes");
|
||||
});
|
||||
|
||||
it("cleans stale flannel vxlan before Cilium install", () => {
|
||||
|
||||
@@ -72,31 +72,97 @@ describe("applyCisHardening", () => {
|
||||
|
||||
// --- Swap ---
|
||||
|
||||
import { disableSwap } from "../src/operations/swap.js";
|
||||
import { enableSwap } from "../src/operations/swap.js";
|
||||
|
||||
describe("disableSwap", () => {
|
||||
it("disables active swap", async () => {
|
||||
describe("enableSwap", () => {
|
||||
it("activates LV swap when present but off", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec
|
||||
.mockResolvedValueOnce(stdout("/dev/sda2 partition 2G")) // swap active
|
||||
.mockResolvedValueOnce(OK) // swapoff
|
||||
.mockResolvedValueOnce(OK); // sed fstab
|
||||
.mockResolvedValueOnce(stdout("yes")) // LV exists
|
||||
.mockResolvedValueOnce(stdout("off")) // not in /proc/swaps
|
||||
.mockResolvedValueOnce(OK) // blkid || mkswap
|
||||
.mockResolvedValueOnce(OK) // swapon
|
||||
.mockResolvedValueOnce(OK); // fstab entry
|
||||
|
||||
const result = await disableSwap(ctx);
|
||||
const result = await enableSwap(ctx);
|
||||
expect(result.success).toBe(true);
|
||||
expect(result.changed).toBe(true);
|
||||
expectCommand(ctx.ssh, "swapoff -a");
|
||||
expectCommand(ctx.ssh, "swapon /dev/mapper/labvg-swap");
|
||||
});
|
||||
|
||||
it("is idempotent when swap already off", async () => {
|
||||
it("is idempotent when LV swap already active", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec
|
||||
.mockResolvedValueOnce(stdout("")) // no swap
|
||||
.mockResolvedValueOnce(OK); // sed fstab (always runs)
|
||||
.mockResolvedValueOnce(stdout("yes")) // LV exists
|
||||
.mockResolvedValueOnce(stdout("on")) // already in /proc/swaps
|
||||
.mockResolvedValueOnce(OK); // fstab entry (always ensured)
|
||||
|
||||
const result = await disableSwap(ctx);
|
||||
const result = await enableSwap(ctx);
|
||||
expect(result.changed).toBe(false);
|
||||
expectNoCommand(ctx.ssh, "swapoff");
|
||||
expectNoCommand(ctx.ssh, "swapon /dev/mapper/labvg-swap");
|
||||
});
|
||||
|
||||
it("skips when no labvg-swap LV exists", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(stdout("no")); // LV missing
|
||||
|
||||
const result = await enableSwap(ctx);
|
||||
expect(result.success).toBe(true);
|
||||
expect(result.changed).toBe(false);
|
||||
expectNoCommand(ctx.ssh, "swapon");
|
||||
});
|
||||
});
|
||||
|
||||
// --- Rancher LV (imageFs sizing) ---
|
||||
|
||||
import { growRancherLv } from "../src/operations/rancher-storage.js";
|
||||
|
||||
describe("growRancherLv", () => {
|
||||
it("grows a 20G LV to 120G when the VG has space", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec
|
||||
.mockResolvedValueOnce(stdout(" 20480.00")) // lv_size
|
||||
.mockResolvedValueOnce(stdout(" 747807.00")) // vg_free
|
||||
.mockResolvedValueOnce(OK) // lvextend
|
||||
.mockResolvedValueOnce(OK); // xfs_growfs
|
||||
|
||||
const result = await growRancherLv(ctx);
|
||||
expect(result.success).toBe(true);
|
||||
expect(result.changed).toBe(true);
|
||||
expectCommand(ctx.ssh, "lvextend -L 122880m /dev/labvg/rancher");
|
||||
expectCommand(ctx.ssh, "xfs_growfs /var/lib/rancher");
|
||||
});
|
||||
|
||||
it("is idempotent when the LV is already 120G", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(stdout(" 122880.00")); // lv_size
|
||||
|
||||
const result = await growRancherLv(ctx);
|
||||
expect(result.changed).toBe(false);
|
||||
expectNoCommand(ctx.ssh, "lvextend");
|
||||
});
|
||||
|
||||
it("reports without failing when the VG has no free space", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec
|
||||
.mockResolvedValueOnce(stdout(" 20480.00")) // lv_size
|
||||
.mockResolvedValueOnce(stdout(" 0.00")); // vg_free
|
||||
|
||||
const result = await growRancherLv(ctx);
|
||||
expect(result.success).toBe(true);
|
||||
expect(result.changed).toBe(false);
|
||||
expect(result.message).toContain("free");
|
||||
expectNoCommand(ctx.ssh, "lvextend");
|
||||
});
|
||||
|
||||
it("skips when there is no rancher LV", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(stdout("")); // lvs empty
|
||||
|
||||
const result = await growRancherLv(ctx);
|
||||
expect(result.success).toBe(true);
|
||||
expect(result.changed).toBe(false);
|
||||
expectNoCommand(ctx.ssh, "lvextend");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -348,3 +414,143 @@ describe("applyPodSecurityStandards", () => {
|
||||
expectCommand(ctx.ssh, "pod-security.kubernetes.io/audit=restricted");
|
||||
});
|
||||
});
|
||||
|
||||
// --- Audit Logging Decommission (file-based → journald) ---
|
||||
|
||||
import { configureLogRotation } from "../src/operations/log-rotation.js";
|
||||
import { configureJournaldLimits } from "../src/operations/journald-limits.js";
|
||||
|
||||
describe("configureLogRotation (decommission file-based audit logs)", () => {
|
||||
it("removes the legacy logrotate rule and reaps obsolete audit files", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(stdout("present")); // probe: legacy artifacts exist
|
||||
ctx.ssh.exec.mockResolvedValue(OK);
|
||||
|
||||
const result = await configureLogRotation(ctx);
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
expect(result.changed).toBe(true);
|
||||
expectCommand(ctx.ssh, "rm -f /etc/logrotate.d/k3s");
|
||||
expectCommand(ctx.ssh, /find \/var\/log\/kubernetes.*audit.*-delete/);
|
||||
expectCommand(ctx.ssh, "rmdir /var/log/kubernetes");
|
||||
});
|
||||
|
||||
it("is a no-op when nothing legacy is present", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(stdout("absent"));
|
||||
ctx.ssh.exec.mockResolvedValue(OK);
|
||||
|
||||
const result = await configureLogRotation(ctx);
|
||||
expect(result.success).toBe(true);
|
||||
expect(result.changed).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("configureJournaldLimits", () => {
|
||||
it("writes a 2 GB SystemMaxUse drop-in and reloads journald when changed", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(stdout("__LABCTL_NOT_FOUND__")); // no existing drop-in
|
||||
ctx.ssh.exec.mockResolvedValue(OK);
|
||||
|
||||
const result = await configureJournaldLimits(ctx);
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
expect(result.changed).toBe(true);
|
||||
const writeCall = ctx.ssh.exec.mock.calls.find((c) => {
|
||||
const cmd = c[0] as string;
|
||||
return cmd.includes("10-k3s-audit-cap.conf") && cmd.includes("LABCTL_EOF");
|
||||
});
|
||||
expect(writeCall).toBeTruthy();
|
||||
const written = writeCall?.[0] as string;
|
||||
expect(written).toContain("SystemMaxUse=2G");
|
||||
expect(written).toContain("SystemKeepFree=1G");
|
||||
expectCommand(ctx.ssh, "systemctl restart systemd-journald");
|
||||
});
|
||||
|
||||
it("does not restart journald when the drop-in is already correct", async () => {
|
||||
const ctx = mockCtx();
|
||||
const existing =
|
||||
"[Journal]\nSystemMaxUse=2G\nSystemKeepFree=1G\nSystemMaxFileSize=200M\n";
|
||||
ctx.ssh.exec.mockResolvedValueOnce(stdout(existing));
|
||||
ctx.ssh.exec.mockResolvedValue(OK);
|
||||
|
||||
const result = await configureJournaldLimits(ctx);
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
expect(result.changed).toBe(false);
|
||||
expectNoCommand(ctx.ssh, "systemctl restart systemd-journald");
|
||||
});
|
||||
});
|
||||
|
||||
// --- Etcd Recovery ---
|
||||
|
||||
import { recoverEtcdMember } from "../src/operations/etcd-recover.js";
|
||||
import { mockSsh } from "./helpers.js";
|
||||
|
||||
describe("recoverEtcdMember", () => {
|
||||
it("refuses to operate when cluster is below 3 members (quorum risk)", async () => {
|
||||
const broken = mockSsh();
|
||||
const peer = mockSsh();
|
||||
peer.exec.mockResolvedValueOnce(stdout("/usr/bin/etcdctl")); // etcdctl present
|
||||
peer.exec.mockResolvedValueOnce(stdout(
|
||||
"111, started, host-a-aaa, https://10.0.0.1:2380, https://10.0.0.1:2379, false\n" +
|
||||
"222, started, host-b-bbb, https://10.0.0.2:2380, https://10.0.0.2:2379, false",
|
||||
));
|
||||
|
||||
const result = await recoverEtcdMember({ broken, peer, brokenHostname: "host-b" });
|
||||
|
||||
expect(result.success).toBe(false);
|
||||
expect(result.message).toMatch(/quorum/i);
|
||||
// Critically: must NOT have stopped k3s or removed anything
|
||||
expect(broken.exec).not.toHaveBeenCalledWith(expect.stringContaining("systemctl stop k3s"), expect.anything());
|
||||
});
|
||||
|
||||
it("performs full procedure when quorum is preserved", async () => {
|
||||
const broken = mockSsh();
|
||||
const peer = mockSsh();
|
||||
// ensureEtcdctl: present
|
||||
peer.exec.mockResolvedValueOnce(stdout("/usr/bin/etcdctl"));
|
||||
// member list (3 members, target = host-b)
|
||||
peer.exec.mockResolvedValueOnce(stdout(
|
||||
"111, started, host-a-aaa, https://10.0.0.1:2380, https://10.0.0.1:2379, false\n" +
|
||||
"222, started, host-b-bbb, https://10.0.0.2:2380, https://10.0.0.2:2379, false\n" +
|
||||
"333, started, host-c-ccc, https://10.0.0.3:2380, https://10.0.0.3:2379, false",
|
||||
));
|
||||
// member remove
|
||||
peer.exec.mockResolvedValueOnce(stdout("Member 222 removed"));
|
||||
// post-rejoin member list — new id 444 for host-b
|
||||
peer.exec.mockResolvedValueOnce(stdout(
|
||||
"111, started, host-a-aaa, https://10.0.0.1:2380, https://10.0.0.1:2379, false\n" +
|
||||
"333, started, host-c-ccc, https://10.0.0.3:2380, https://10.0.0.3:2379, false\n" +
|
||||
"444, started, host-b-zzz, https://10.0.0.2:2380, https://10.0.0.2:2379, false",
|
||||
));
|
||||
|
||||
const result = await recoverEtcdMember({ broken, peer, brokenHostname: "host-b" });
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
expect(result.removedMemberId).toBe("222");
|
||||
expect(result.newMemberId).toBe("444");
|
||||
expectCommand(broken,"systemctl stop k3s");
|
||||
expectCommand(peer,"member remove 222");
|
||||
expectCommand(broken,/db\.corrupt-/);
|
||||
expectCommand(broken,/rm -rf .*\/server\/tls/);
|
||||
expectCommand(broken,"systemctl start k3s");
|
||||
});
|
||||
|
||||
it("fails clearly when no member matches the broken hostname", async () => {
|
||||
const broken = mockSsh();
|
||||
const peer = mockSsh();
|
||||
peer.exec.mockResolvedValueOnce(stdout("/usr/bin/etcdctl"));
|
||||
peer.exec.mockResolvedValueOnce(stdout(
|
||||
"111, started, host-a-aaa, https://10.0.0.1:2380, https://10.0.0.1:2379, false\n" +
|
||||
"222, started, host-b-bbb, https://10.0.0.2:2380, https://10.0.0.2:2379, false\n" +
|
||||
"333, started, host-c-ccc, https://10.0.0.3:2380, https://10.0.0.3:2379, false",
|
||||
));
|
||||
|
||||
const result = await recoverEtcdMember({ broken, peer, brokenHostname: "host-d" });
|
||||
|
||||
expect(result.success).toBe(false);
|
||||
expect(result.message).toMatch(/No etcd member found/);
|
||||
expect(broken.exec).not.toHaveBeenCalledWith(expect.stringContaining("systemctl stop k3s"), expect.anything());
|
||||
});
|
||||
});
|
||||
|
||||
@@ -16,7 +16,7 @@ describe("smoke: full server install pipeline", () => {
|
||||
const pipeline: NamedOperation[] = [
|
||||
{ name: "Kernel modules", fn: ops.loadKernelModules },
|
||||
{ name: "Sysctl hardening", fn: ops.applyCisHardening },
|
||||
{ name: "Disable swap", fn: ops.disableSwap },
|
||||
{ name: "Enable swap", fn: ops.enableSwap },
|
||||
{ name: "Disable firewall", fn: ops.disableFirewall },
|
||||
{ name: "SELinux permissive", fn: ops.setSelinuxPermissive },
|
||||
{ name: "Write k3s config", fn: ops.writeK3sConfig },
|
||||
@@ -73,7 +73,7 @@ describe("smoke: pipeline stops on failure", () => {
|
||||
};
|
||||
|
||||
const results = await runSequential(ctx, [
|
||||
{ name: "OK op", fn: ops.disableSwap },
|
||||
{ name: "OK op", fn: ops.enableSwap },
|
||||
{ name: "Failing op", fn: failingOp },
|
||||
{ name: "Never called", fn: neverCalled },
|
||||
]);
|
||||
@@ -98,11 +98,12 @@ describe("smoke: agent install rejects missing config", () => {
|
||||
});
|
||||
|
||||
describe("smoke: all operations are exported", () => {
|
||||
it("exports all 15 operations", () => {
|
||||
it("exports all 16 operations", () => {
|
||||
const exported = [
|
||||
ops.loadKernelModules,
|
||||
ops.applyCisHardening,
|
||||
ops.disableSwap,
|
||||
ops.enableSwap,
|
||||
ops.growRancherLv,
|
||||
ops.disableFirewall,
|
||||
ops.setSelinuxPermissive,
|
||||
ops.writeK3sConfig,
|
||||
@@ -117,7 +118,7 @@ describe("smoke: all operations are exported", () => {
|
||||
ops.checkCertExpiry,
|
||||
];
|
||||
|
||||
expect(exported).toHaveLength(15);
|
||||
expect(exported).toHaveLength(16);
|
||||
for (const op of exported) {
|
||||
expect(typeof op).toBe("function");
|
||||
}
|
||||
|
||||
@@ -8,6 +8,10 @@ export type {
|
||||
DebugConfig,
|
||||
BastionState,
|
||||
BastionConfig,
|
||||
VyosVlanSpec,
|
||||
VyosInstallSpec,
|
||||
VyosBundle,
|
||||
VyosBundleSetOp,
|
||||
} from "./types/index.js";
|
||||
|
||||
export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./types/index.js";
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
// Protocol types for agent-labd WebSocket communication.
|
||||
|
||||
import { randomUUID } from "node:crypto";
|
||||
import type { VyosInstallSpec } from "../types/state.js";
|
||||
|
||||
// --- Agent -> labd messages ---
|
||||
|
||||
@@ -108,11 +109,12 @@ export type BastionMessage =
|
||||
export type LabdBastionMessage =
|
||||
| { type: "bastion-enrolled"; bastionId: string }
|
||||
| { type: "bastion-heartbeat-ack"; serverTime: string }
|
||||
| { type: "command-install"; requestId: string; mac: string; hostname: string; disk?: string; role: string; os: string }
|
||||
| { type: "command-install"; requestId: string; mac: string; hostname: string; disk?: string; role: string; os: string; vyos?: VyosInstallSpec }
|
||||
| { type: "command-forget"; requestId: string; mac: string }
|
||||
| { type: "command-role-update"; requestId: string; mac: string; role: string }
|
||||
| { type: "command-debug"; requestId: string; mac: string; pxeBoot?: boolean }
|
||||
| { type: "command-register"; requestId: string; mac: string; hostname: string; role: string; ip: string }
|
||||
| { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }> }
|
||||
| { type: "server-shutdown"; reconnectAfter: number };
|
||||
|
||||
export type BastionMessageType = BastionMessage["type"];
|
||||
@@ -127,7 +129,7 @@ const BASTION_MESSAGE_TYPES = new Set<string>([
|
||||
|
||||
const LABD_BASTION_MESSAGE_TYPES = new Set<string>([
|
||||
"bastion-enrolled", "bastion-heartbeat-ack", "command-install",
|
||||
"command-forget", "command-role-update", "command-debug", "command-register", "server-shutdown",
|
||||
"command-forget", "command-role-update", "command-debug", "command-register", "command-discover", "server-shutdown",
|
||||
]);
|
||||
|
||||
export function isBastionMessage(msg: unknown): msg is BastionMessage {
|
||||
|
||||
@@ -14,6 +14,10 @@ export interface BastionConfig {
|
||||
// Ubuntu support
|
||||
ubuntuVersion: string;
|
||||
ubuntuMirror: string;
|
||||
// VyOS support — netboot artifacts are extracted from the ISO at startup.
|
||||
// LTS ISOs are subscription-only, so this defaults to a rolling release.
|
||||
vyosIsoUrl: string;
|
||||
vyosDefaultPassword: string;
|
||||
// Syslog listener for install logs (Anaconda logging --host)
|
||||
syslogPort: number;
|
||||
// Flags
|
||||
|
||||
@@ -7,6 +7,10 @@ export type {
|
||||
InstalledInfo,
|
||||
DebugConfig,
|
||||
BastionState,
|
||||
VyosVlanSpec,
|
||||
VyosInstallSpec,
|
||||
VyosBundle,
|
||||
VyosBundleSetOp,
|
||||
} from "./state.js";
|
||||
|
||||
export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./state.js";
|
||||
|
||||
@@ -2,10 +2,10 @@
|
||||
|
||||
export type ProvisionStackType = "dhcpproxy" | "iso" | "cloud-init";
|
||||
|
||||
export type OsId = "fedora-43" | "ubuntu-26.04";
|
||||
export type OsId = "fedora-43" | "ubuntu-26.04" | "vyos-rolling";
|
||||
export type Arch = "x86_64" | "aarch64";
|
||||
|
||||
export const SUPPORTED_OS: readonly OsId[] = ["fedora-43", "ubuntu-26.04"] as const;
|
||||
export const SUPPORTED_OS: readonly OsId[] = ["fedora-43", "ubuntu-26.04", "vyos-rolling"] as const;
|
||||
|
||||
export function isValidOsId(value: string): value is OsId {
|
||||
return (SUPPORTED_OS as readonly string[]).includes(value);
|
||||
@@ -75,13 +75,141 @@ export interface ProgressLogEntry {
|
||||
timestamp: string;
|
||||
}
|
||||
|
||||
/** A tagged VLAN sub-interface on the bond (or on the mgmt NIC when unbonded). */
|
||||
export interface VyosVlanSpec {
|
||||
id: number;
|
||||
address: string; // CIDR, e.g. "10.0.10.1/24"
|
||||
description?: string;
|
||||
/**
|
||||
* VRRP virtual address (CIDR) floated on this VLAN. Emitted as a
|
||||
* high-availability vrrp group with vrid = VLAN id, so the same spec on both
|
||||
* HA peers (with different priorities) produces a matching group pair.
|
||||
*/
|
||||
vrrp?: string;
|
||||
}
|
||||
|
||||
/** One config node in a rendered bundle. Mirrors VyosSetOp on the bastion side. */
|
||||
export interface VyosBundleSetOp {
|
||||
path: string[];
|
||||
value?: string;
|
||||
/** false appends to a multi-value node (e.g. bond members) instead of replacing. */
|
||||
replace?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* A router's complete desired config, rendered from the Pulumi model.
|
||||
*
|
||||
* Produced by `kubernetes-deployment/scripts/vyos-render-bundle.ts` from the
|
||||
* same subtree model `pulumi up` applies. The point is that labctl never
|
||||
* authors VyOS config: bring-up replays what Pulumi already declares, so a
|
||||
* freshly installed router and a `pulumi up` cannot disagree.
|
||||
*
|
||||
* Secret values arrive as `@secret:<key>` sentinels and are DROPPED at install
|
||||
* time -- the bundle is committed to git and must stay safe to read. The router
|
||||
* comes up on the LAN without its PPPoE credential; the first `pulumi up`
|
||||
* supplies it. That handoff is deliberate.
|
||||
*/
|
||||
export interface VyosBundle {
|
||||
sets: VyosBundleSetOp[];
|
||||
/** Paths that are VyOS tag nodes — the installer's ConfigTree needs them marked. */
|
||||
tags: string[][];
|
||||
}
|
||||
|
||||
/**
|
||||
* VyOS-specific install parameters. Rendered into the config.boot that the
|
||||
* installer adopts, so the router comes up already configured.
|
||||
*
|
||||
* NOTE: bondMembers must NOT include the interface PXE booted from. Firmware
|
||||
* PXE cannot run over LACP, so the install-time NIC has to stay unbonded.
|
||||
*/
|
||||
export interface VyosInstallSpec {
|
||||
/**
|
||||
* A complete rendered config for this router. When present it REPLACES the
|
||||
* derived interface/VLAN/VRRP config below -- the bundle already describes
|
||||
* all of it, and deriving a second opinion is exactly the drift this exists
|
||||
* to prevent. The remaining install parameters (password, disk, console) are
|
||||
* still honoured because they are installer inputs, not router config.
|
||||
*/
|
||||
bundle?: VyosBundle;
|
||||
/**
|
||||
* Key for the VyOS HTTP API, enabled at install so the router is manageable
|
||||
* from the moment it boots.
|
||||
*
|
||||
* Without this the box comes up reachable only over SSH, and enabling the API
|
||||
* later is a hand-run config change on a live firewall -- which is exactly the
|
||||
* gap that left vyos001/vyos002 unmanageable by Pulumi after their cutover.
|
||||
* The API is deliberately NOT part of the Pulumi model: a provider that
|
||||
* manages its own transport can revoke its own access.
|
||||
*/
|
||||
apiKey?: string;
|
||||
/**
|
||||
* Address the API listens on. Defaults to the management address when static.
|
||||
* Never left unbound: an unrestricted listener puts a config-write endpoint on
|
||||
* every segment the router touches, including the WAN.
|
||||
*/
|
||||
apiListenAddress?: string;
|
||||
/** Interfaces aggregated into bond0 with LACP (802.3ad). Omit for no bond. */
|
||||
bondMembers?: string[];
|
||||
/** CIDR address on bond0 itself — the switch trunk's native/untagged VLAN. */
|
||||
bondAddress?: string;
|
||||
/** VRRP virtual address (CIDR) floated on the untagged bond (vrid 1). */
|
||||
bondVrrp?: string;
|
||||
/**
|
||||
* VRRP priority for every group on this box. Higher wins mastership.
|
||||
* The HA pair differs ONLY here (e.g. 200 on the primary, 100 on the
|
||||
* standby) — addresses differ per box, VIPs and vrids match.
|
||||
*/
|
||||
vrrpPriority?: number;
|
||||
/** Tagged VLAN sub-interfaces, created on bond0 when bonded, else on mgmtInterface. */
|
||||
vlans?: VyosVlanSpec[];
|
||||
/** Untagged interface the machine PXE booted from. Defaults to "eth0". */
|
||||
mgmtInterface?: string;
|
||||
/** CIDR address for mgmtInterface, or "dhcp". Defaults to "dhcp". */
|
||||
mgmtAddress?: string;
|
||||
/**
|
||||
* Tagged management VLAN on mgmtInterface, separate from the routed VLANs
|
||||
* carried by the bond.
|
||||
*
|
||||
* Needed when the PXE port is a trunk: it boots untagged on the VLAN the
|
||||
* bastion's proxy DHCP serves, and carries the management VLAN tagged so the
|
||||
* router stays reachable there without giving up reinstallability.
|
||||
*/
|
||||
mgmtVlan?: VyosVlanSpec;
|
||||
/** Password for the "vyos" user. Falls back to the bastion default. */
|
||||
password?: string;
|
||||
/**
|
||||
* On reinstall the VyOS installer carries the previous on-disk config (and
|
||||
* SSH host keys) forward -- the "reinstall without losing data" default.
|
||||
* Set true to make the bastion-generated config win instead: after install
|
||||
* the driver overwrites the installed image's config.boot.
|
||||
*/
|
||||
freshConfig?: boolean;
|
||||
/**
|
||||
* VyOS interface name -> MAC, emitted as `hw-id` so names bind deterministically.
|
||||
*
|
||||
* Discovery runs under Fedora and reports predictable names (enp2s0,
|
||||
* enp1s0f0np0), but VyOS enumerates its own eth<N> names, so a name observed
|
||||
* during discovery cannot be used directly. Pinning by MAC removes the guess
|
||||
* about which physical port a given eth<N> is.
|
||||
*/
|
||||
hwIds?: Record<string, string>;
|
||||
}
|
||||
|
||||
export interface InstallConfig {
|
||||
hostname: string;
|
||||
disk: string;
|
||||
role: Role;
|
||||
os?: OsId; // defaults to "fedora-43" for backward compat
|
||||
vyos?: VyosInstallSpec; // only consulted when os is "vyos-rolling"
|
||||
arch?: Arch; // detected from HardwareInfo or overridden
|
||||
queued_at: string;
|
||||
/**
|
||||
* When dispatch last served this machine an install boot script. Progress
|
||||
* callbacks only start once the installer environment is up, so a machine
|
||||
* dispatched long ago with no progress is wedged before that point (bad
|
||||
* kernel/initrd, no network in the initramfs, wrong NIC picked...).
|
||||
*/
|
||||
dispatched_at?: string;
|
||||
progress?: string;
|
||||
progress_at?: string;
|
||||
progress_detail?: string;
|
||||
@@ -96,6 +224,13 @@ export interface InstalledInfo {
|
||||
ip: string;
|
||||
installed_at: string;
|
||||
bastionId?: string; // set when aggregated through labd
|
||||
// Hardware info (copied from discovered on install completion)
|
||||
product?: string;
|
||||
manufacturer?: string;
|
||||
cpu_model?: string;
|
||||
cpu_cores?: number;
|
||||
memory_gb?: number;
|
||||
arch?: string;
|
||||
}
|
||||
|
||||
export interface DebugConfig {
|
||||
|
||||
@@ -29,6 +29,17 @@ export interface PxeVmConfig {
|
||||
diskSize: number; // GB
|
||||
network: string; // libvirt network name
|
||||
arch?: "x86_64" | "aarch64";
|
||||
/**
|
||||
* Extra NICs enumerated BEFORE the PXE NIC, on a network with no route to
|
||||
* the bastion (defaults to libvirt's "default").
|
||||
*
|
||||
* Real multi-NIC boxes expose a class of bug a single-NIC VM cannot: an
|
||||
* initramfs that picks "the first connected interface" grabs one of these
|
||||
* instead of the NIC that PXE booted, and then cannot reach the bastion.
|
||||
* Defaults to 0 (single NIC).
|
||||
*/
|
||||
decoyNics?: number;
|
||||
decoyNetwork?: string;
|
||||
}
|
||||
|
||||
/** Create a blank UEFI VM that PXE boots from the network. */
|
||||
@@ -61,6 +72,10 @@ export function createPxeVm(config: PxeVmConfig): void {
|
||||
`--memory=${config.memory}`,
|
||||
`--vcpus=${config.vcpus}`,
|
||||
`--disk=path=${diskPath},format=qcow2,bus=virtio`,
|
||||
// Decoys first so they enumerate ahead of the PXE NIC. They are up and
|
||||
// carry a lease, but have no route to the bastion.
|
||||
...Array.from({ length: config.decoyNics ?? 0 }, () =>
|
||||
`--network=network=${config.decoyNetwork ?? "default"},model=virtio`),
|
||||
`--network=network=${config.network},model=virtio`,
|
||||
// UEFI firmware — required for PXE boot in modern mode
|
||||
`--boot=uefi,network,hd`,
|
||||
@@ -95,12 +110,21 @@ export function destroyPxeVm(name: string): void {
|
||||
}
|
||||
|
||||
/** Get the MAC address of a VM's first NIC. */
|
||||
export function getVmMac(name: string): string | null {
|
||||
export function getVmMac(name: string, network?: string): string | null {
|
||||
const result = virsh("domiflist", name);
|
||||
if (result.status !== 0) return null;
|
||||
// Output format: Interface Type Source Model MAC
|
||||
const match = result.stdout.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
|
||||
return match ? match[1].toLowerCase() : null;
|
||||
// With decoy NICs present, match the line for the PXE network so we return
|
||||
// the NIC that actually boots rather than whichever is listed first.
|
||||
const lines = result.stdout.split("\n");
|
||||
const candidates = network === undefined
|
||||
? lines
|
||||
: lines.filter((l) => l.split(/\s+/).includes(network));
|
||||
for (const line of candidates.length > 0 ? candidates : lines) {
|
||||
const m = line.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
|
||||
if (m) return m[1].toLowerCase();
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Reboot a VM (force off + start). */
|
||||
|
||||
@@ -161,7 +161,7 @@ EOF'
|
||||
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
|
||||
curl -L --fail --silent "https://github.com/cilium/cilium-cli/releases/download/\${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz" | sudo tar xz -C /usr/local/bin
|
||||
DEFAULT_DEV=$(ip -4 route show default | awk '{print $5}' | head -1)
|
||||
sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=kubernetes --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
|
||||
sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=cluster-pool --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' --set ipam.operator.clusterPoolIPv4MaskSize=24 --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
|
||||
`.trim(), "cilium install", { keyPath: sshKeyPath, timeout: 120_000 });
|
||||
|
||||
log("Waiting for Cilium to be ready...");
|
||||
|
||||
387
bastion/tests/integration/vyos-provision.test.ts
Normal file
387
bastion/tests/integration/vyos-provision.test.ts
Normal file
@@ -0,0 +1,387 @@
|
||||
// Integration test: full VyOS unattended provisioning flow.
|
||||
//
|
||||
// Validates the VyOS install path end-to-end, at the same depth as the Fedora
|
||||
// pxe-provision test:
|
||||
// 1. Bastion (HTTP + dnsmasq) on the isolated libvirt PXE network
|
||||
// 2. Blank UEFI VM PXE boots -> Fedora-based discovery (OS-neutral)
|
||||
// 3. Queue os=vyos-rolling -> live boot + live-config hook + pty driver
|
||||
// 4. Fresh-install asserts: installed.ip, streamed logs, applied config,
|
||||
// /config/lab-provisioned, boot-order handling
|
||||
// 5. REINSTALL round: previous config + /config data carried forward
|
||||
// ("reinstall without losing data", VyOS-flavored)
|
||||
// 6. freshConfig round: bastion-generated config wins, /config data kept
|
||||
//
|
||||
// Prerequisites: libvirtd, OVMF, ipxe-bootimgs-x86, sudo, internet
|
||||
// (first run downloads the ~600MB VyOS nightly ISO; artifacts are cached).
|
||||
// Run: sudo pnpm run test:integration:vyos
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll } from "vitest";
|
||||
import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { execSync } from "node:child_process";
|
||||
import { join } from "node:path";
|
||||
import { homedir, tmpdir } from "node:os";
|
||||
import { log, waitForSsh } from "./helpers/libvirt.js";
|
||||
import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
|
||||
import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm } from "./helpers/pxe-vm.js";
|
||||
import { sshExec } from "./helpers/ssh.js";
|
||||
|
||||
const VM_NAME = "lab-vyos-test";
|
||||
const VM_MEMORY = 4096;
|
||||
const VM_VCPUS = 4;
|
||||
const VM_DISK_GB = 10; // VyOS image install needs ~2GB minimum
|
||||
const HTTP_PORT = 8099;
|
||||
const SSH_USER = "vyos"; // the only VyOS login user
|
||||
const BASTION_IP = PXE_GATEWAY;
|
||||
const DHCP_RANGE_START = `${PXE_SUBNET}.100`;
|
||||
const DHCP_RANGE_END = `${PXE_SUBNET}.200`;
|
||||
|
||||
const DISCOVERY_TIMEOUT_MS = 5 * 60_000;
|
||||
const INSTALL_TIMEOUT_MS = 15 * 60_000; // squashfs fetch + copy; much faster than Anaconda
|
||||
const SSH_TIMEOUT_MS = 8 * 60_000;
|
||||
|
||||
const HOSTNAME_R1 = "vyos-r1";
|
||||
const HOSTNAME_R2 = "vyos-r2";
|
||||
const HOSTNAME_R3 = "vyos-r3";
|
||||
|
||||
function findSshKey(): { pubKey: string; keyPath: string } {
|
||||
const homes = [homedir()];
|
||||
const sudoUser = process.env["SUDO_USER"];
|
||||
if (sudoUser) homes.push(join("/home", sudoUser));
|
||||
if (process.env["SSH_KEY_PATH"]) {
|
||||
const keyPath = process.env["SSH_KEY_PATH"];
|
||||
const pubPath = `${keyPath}.pub`;
|
||||
if (existsSync(keyPath) && existsSync(pubPath)) {
|
||||
return { pubKey: readFileSync(pubPath, "utf-8").trim(), keyPath };
|
||||
}
|
||||
}
|
||||
for (const home of homes) {
|
||||
for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
|
||||
const keyPath = join(home, ".ssh", name);
|
||||
const pubPath = `${keyPath}.pub`;
|
||||
if (existsSync(keyPath) && existsSync(pubPath)) {
|
||||
return { pubKey: readFileSync(pubPath, "utf-8").trim(), keyPath };
|
||||
}
|
||||
}
|
||||
}
|
||||
throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
|
||||
}
|
||||
|
||||
function sleep(ms: number): Promise<void> {
|
||||
return new Promise((r) => setTimeout(r, ms));
|
||||
}
|
||||
|
||||
async function pollApi<T>(
|
||||
url: string,
|
||||
check: (data: T) => boolean,
|
||||
timeoutMs: number,
|
||||
intervalMs = 5000,
|
||||
): Promise<T> {
|
||||
const start = Date.now();
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
try {
|
||||
const res = await fetch(url);
|
||||
if (res.ok) {
|
||||
const data = (await res.json()) as T;
|
||||
if (check(data)) return data;
|
||||
}
|
||||
} catch { /* not ready yet */ }
|
||||
await sleep(intervalMs);
|
||||
}
|
||||
throw new Error(`Timeout after ${timeoutMs}ms polling ${url}`);
|
||||
}
|
||||
|
||||
type LogsResponse = {
|
||||
status: string;
|
||||
progress: string;
|
||||
progress_detail?: string;
|
||||
ip?: string;
|
||||
log_total?: number;
|
||||
log_lines?: Array<{ line: string }>;
|
||||
};
|
||||
|
||||
/** Queue a VyOS install, reboot the VM into PXE, wait for completion + SSH. */
|
||||
async function installRound(opts: {
|
||||
mac: string;
|
||||
hostname: string;
|
||||
freshConfig?: boolean;
|
||||
}): Promise<string> {
|
||||
const body = {
|
||||
mac: opts.mac,
|
||||
hostname: opts.hostname,
|
||||
disk: "/dev/vda",
|
||||
role: "vanilla",
|
||||
os: "vyos-rolling",
|
||||
vyos: {
|
||||
mgmtInterface: "eth0",
|
||||
mgmtAddress: "dhcp",
|
||||
hwIds: { eth0: opts.mac },
|
||||
...(opts.freshConfig ? { freshConfig: true } : {}),
|
||||
},
|
||||
};
|
||||
const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/install`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
log(`Install queued (${opts.hostname}): ${JSON.stringify(await res.json())}`);
|
||||
|
||||
await sleep(5_000);
|
||||
rebootPxeVm(VM_NAME);
|
||||
await sleep(3_000);
|
||||
deleteNftablesRejectRules();
|
||||
|
||||
const finalState = await pollApi<LogsResponse>(
|
||||
`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(opts.mac)}`,
|
||||
(data) => data.status === "installed" || data.progress === "error",
|
||||
INSTALL_TIMEOUT_MS,
|
||||
10_000,
|
||||
);
|
||||
if (finalState.progress === "error") {
|
||||
log(`INSTALL FAILED: ${JSON.stringify(finalState.progress_detail ?? finalState, null, 2)}`);
|
||||
throw new Error(`VyOS install failed for ${opts.hostname}`);
|
||||
}
|
||||
const ip = finalState.ip ?? "";
|
||||
log(`Install complete (${opts.hostname}). IP: ${ip}`);
|
||||
|
||||
// The driver force-reboots; the VM PXE boots, dispatch says installed ->
|
||||
// localboot exit -> GRUB -> VyOS. nftables reject rules do not reappear
|
||||
// (guest reboot, not a libvirt restart), but clearing is harmless.
|
||||
deleteNftablesRejectRules();
|
||||
await waitForSsh(ip, SSH_USER, SSH_TIMEOUT_MS, sshKeyPathGlobal);
|
||||
return ip;
|
||||
}
|
||||
|
||||
let sshKeyPathGlobal = "";
|
||||
|
||||
describe("VyOS provisioning", () => {
|
||||
let bastionApp: { close: () => Promise<void> };
|
||||
let testDir: string;
|
||||
let vmMac: string;
|
||||
let vmIp: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
const { pubKey, keyPath } = findSshKey();
|
||||
sshKeyPathGlobal = keyPath;
|
||||
|
||||
log("Setting up PXE test network...");
|
||||
ensurePxeNetwork();
|
||||
|
||||
testDir = join(tmpdir(), `lab-vyos-test-${Date.now()}`);
|
||||
mkdirSync(join(testDir, "tftp"), { recursive: true });
|
||||
mkdirSync(join(testDir, "http"), { recursive: true });
|
||||
mkdirSync(join(testDir, "logs"), { recursive: true });
|
||||
|
||||
log("Starting bastion...");
|
||||
const { createApp } = await import("../../src/bastion/src/server.js");
|
||||
const { loadConfig } = await import("../../src/bastion/src/config.js");
|
||||
const { generateDnsmasqConf, startDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
|
||||
const { generateDiscoverKickstart } = await import("../../src/bastion/src/services/kickstart-generator.js");
|
||||
const { renderBootIpxe } = await import("../../src/bastion/src/templates/boot.ipxe.js");
|
||||
const { prepareVyosArtifacts } = await import("../../src/bastion/src/main.js");
|
||||
|
||||
const config = loadConfig({
|
||||
bastionDir: testDir,
|
||||
httpPort: HTTP_PORT,
|
||||
iface: "virbr-pxe",
|
||||
serverIp: BASTION_IP,
|
||||
network: `${PXE_SUBNET}.0`,
|
||||
gateway: BASTION_IP,
|
||||
dhcpMode: "full",
|
||||
dhcpRangeStart: DHCP_RANGE_START,
|
||||
dhcpRangeEnd: DHCP_RANGE_END,
|
||||
domain: "pxe-test.local",
|
||||
sshKeys: [pubKey],
|
||||
adminUser: "lab",
|
||||
});
|
||||
|
||||
// iPXE binary
|
||||
const ipxeSrc = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
|
||||
if (!existsSync(ipxeSrc)) {
|
||||
throw new Error(`iPXE not found: ${ipxeSrc}. Install: sudo dnf install ipxe-bootimgs-x86`);
|
||||
}
|
||||
copyFileSync(ipxeSrc, join(config.tftpDir, "ipxe.efi"));
|
||||
try { symlinkSync(join(config.tftpDir, "ipxe.efi"), join(config.httpDir, "ipxe.efi")); } catch { /* exists */ }
|
||||
|
||||
const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
|
||||
execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
|
||||
|
||||
// Fedora kernel+initrd for DISCOVERY (OS-neutral, same as pxe test)
|
||||
const kernel = join(cacheDir, `vmlinuz-${config.fedoraVersion}`);
|
||||
const initrd = join(cacheDir, `initrd-${config.fedoraVersion}.img`);
|
||||
if (!existsSync(kernel)) {
|
||||
log(`Downloading Fedora ${config.fedoraVersion} kernel (discovery)...`);
|
||||
execSync(`curl -# -L -f -o "${kernel}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 300_000 });
|
||||
}
|
||||
if (!existsSync(initrd)) {
|
||||
log(`Downloading Fedora ${config.fedoraVersion} initrd (discovery)...`);
|
||||
execSync(`curl -# -L -f -o "${initrd}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 300_000 });
|
||||
}
|
||||
copyFileSync(kernel, join(config.httpDir, "vmlinuz"));
|
||||
copyFileSync(initrd, join(config.httpDir, "initrd.img"));
|
||||
|
||||
// VyOS netboot artifacts — cache the three extracted files across runs
|
||||
const vyosCache = {
|
||||
kernel: join(cacheDir, "vyos-vmlinuz"),
|
||||
initrd: join(cacheDir, "vyos-initrd"),
|
||||
squashfs: join(cacheDir, "vyos-filesystem.squashfs"),
|
||||
};
|
||||
if (Object.values(vyosCache).every((p) => existsSync(p))) {
|
||||
log("VyOS netboot artifacts cached");
|
||||
copyFileSync(vyosCache.kernel, join(config.httpDir, "vyos-vmlinuz"));
|
||||
copyFileSync(vyosCache.initrd, join(config.httpDir, "vyos-initrd"));
|
||||
copyFileSync(vyosCache.squashfs, join(config.httpDir, "vyos-filesystem.squashfs"));
|
||||
} else {
|
||||
log("Extracting VyOS artifacts from ISO (downloads ~600MB on first run)...");
|
||||
prepareVyosArtifacts(config);
|
||||
copyFileSync(join(config.httpDir, "vyos-vmlinuz"), vyosCache.kernel);
|
||||
copyFileSync(join(config.httpDir, "vyos-initrd"), vyosCache.initrd);
|
||||
copyFileSync(join(config.httpDir, "vyos-filesystem.squashfs"), vyosCache.squashfs);
|
||||
}
|
||||
|
||||
writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config));
|
||||
writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }));
|
||||
generateDnsmasqConf(config);
|
||||
|
||||
const { app, syslog } = createApp(config);
|
||||
bastionApp = app;
|
||||
await app.listen({ port: config.httpPort, host: "0.0.0.0" });
|
||||
syslog.start();
|
||||
log(`Bastion listening on :${HTTP_PORT}`);
|
||||
|
||||
log("Starting dnsmasq...");
|
||||
startDnsmasq(config).catch((err) => {
|
||||
log(`dnsmasq failed (expected without root): ${err instanceof Error ? err.message : String(err)}`);
|
||||
});
|
||||
await sleep(1000);
|
||||
|
||||
log("Creating PXE VM...");
|
||||
// Two decoy NICs ahead of the PXE NIC, on a network with no route to the
|
||||
// bastion. This reproduces the real VP2440 topology: live-boot scans for
|
||||
// "the first connected interface", and without BOOTIF it picks a decoy,
|
||||
// times out on DHCP/fetch, and dies with "Unable to find a live file
|
||||
// system on the network". A single-NIC VM cannot catch that.
|
||||
createPxeVm({
|
||||
name: VM_NAME,
|
||||
memory: VM_MEMORY,
|
||||
vcpus: VM_VCPUS,
|
||||
diskSize: VM_DISK_GB,
|
||||
network: PXE_NETWORK_NAME,
|
||||
decoyNics: 2,
|
||||
});
|
||||
const mac = getVmMac(VM_NAME, PXE_NETWORK_NAME);
|
||||
if (!mac) throw new Error("Could not determine VM MAC address");
|
||||
vmMac = mac;
|
||||
log(`VM MAC: ${vmMac}`);
|
||||
|
||||
log("Waiting for discovery...");
|
||||
type MachinesResponse = { discovered: Record<string, unknown> };
|
||||
await pollApi<MachinesResponse>(
|
||||
`http://${BASTION_IP}:${HTTP_PORT}/api/machines`,
|
||||
(data) => vmMac in data.discovered,
|
||||
DISCOVERY_TIMEOUT_MS,
|
||||
);
|
||||
log("VM discovered. Running fresh VyOS install (round 1)...");
|
||||
|
||||
await sleep(15_000); // discovery reboot cycle
|
||||
vmIp = await installRound({ mac: vmMac, hostname: HOSTNAME_R1 });
|
||||
log("Round 1 (fresh install) complete.");
|
||||
}, DISCOVERY_TIMEOUT_MS + INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
|
||||
|
||||
afterAll(async () => {
|
||||
log("Cleaning up...");
|
||||
if (bastionApp) await bastionApp.close().catch(() => {});
|
||||
const { stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
|
||||
stopDnsmasq();
|
||||
destroyPxeVm(VM_NAME);
|
||||
destroyPxeNetwork();
|
||||
if (testDir) rmSync(testDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("machine is installed with a real IP (WI-1: ready-at parsing)", async () => {
|
||||
const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
|
||||
const data = (await res.json()) as { installed: Record<string, { ip: string; os?: string }> };
|
||||
const machine = data.installed[vmMac];
|
||||
expect(machine).toBeDefined();
|
||||
expect(machine.ip).toMatch(/^\d+\.\d+\.\d+\.\d+$/);
|
||||
expect(machine.os).toBe("vyos-rolling");
|
||||
});
|
||||
|
||||
it("install logs were streamed live (WI-2)", async () => {
|
||||
const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`);
|
||||
const data = (await res.json()) as LogsResponse;
|
||||
expect(data.log_total).toBeGreaterThan(0);
|
||||
const lines = (data.log_lines ?? []).map((l) => l.line).join("\n");
|
||||
// Installer transcript lines and driver messages both flow through /api/log
|
||||
expect(lines).toMatch(/Welcome to VyOS installation|>>> answered|base config:/);
|
||||
});
|
||||
|
||||
it("SSH works as the vyos user with the injected key", () => {
|
||||
const result = sshExec(vmIp, SSH_USER, "whoami", { keyPath: sshKeyPathGlobal });
|
||||
expect(result.exitCode).toBe(0);
|
||||
expect(result.stdout.trim()).toBe("vyos");
|
||||
});
|
||||
|
||||
it("generated config was adopted (hostname + ssh key)", () => {
|
||||
const result = sshExec(vmIp, SSH_USER, "cat /opt/vyatta/etc/config/config.boot", { keyPath: sshKeyPathGlobal });
|
||||
expect(result.exitCode).toBe(0);
|
||||
expect(result.stdout).toContain(`host-name "${HOSTNAME_R1}"`);
|
||||
expect(result.stdout).toContain("public-keys");
|
||||
});
|
||||
|
||||
it("boot-order step ran and reported (WI-3)", async () => {
|
||||
const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`);
|
||||
const data = (await res.json()) as LogsResponse;
|
||||
const lines = (data.log_lines ?? []).map((l) => l.line).join("\n");
|
||||
expect(lines).toContain("boot order:");
|
||||
});
|
||||
|
||||
it("provisioning metadata persisted to /config (WI-4)", () => {
|
||||
const result = sshExec(vmIp, SSH_USER, "cat /config/lab-provisioned 2>/dev/null || cat /opt/vyatta/etc/config/lab-provisioned", { keyPath: sshKeyPathGlobal });
|
||||
expect(result.exitCode).toBe(0);
|
||||
expect(result.stdout).toContain(`hostname=${HOSTNAME_R1}`);
|
||||
expect(result.stdout).toContain("role=vanilla");
|
||||
expect(result.stdout).toContain(`bastion=http://${BASTION_IP}:${HTTP_PORT}`);
|
||||
});
|
||||
|
||||
it("reinstall preserves config and /config data (round 2)", async () => {
|
||||
// Drop a marker in /config — the installer's previous-installation copy
|
||||
// must carry it (and the whole old config) into the new image.
|
||||
// `sync` is REQUIRED: rebootPxeVm uses `virsh destroy` (a hard power-cut),
|
||||
// so an unsynced write never reaches the disk and the marker vanishes for
|
||||
// reasons that have nothing to do with the installer.
|
||||
const marker = sshExec(vmIp, SSH_USER, "echo LAB-MARKER-R2 > /config/lab-marker && sync && cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
|
||||
expect(marker.exitCode).toBe(0);
|
||||
expect(marker.stdout).toContain("LAB-MARKER-R2");
|
||||
|
||||
// Queue with a DIFFERENT hostname: with preserve semantics the previous
|
||||
// config must win, so the hostname must NOT change.
|
||||
vmIp = await installRound({ mac: vmMac, hostname: HOSTNAME_R2 });
|
||||
|
||||
// Assert the config carry-forward first — it is the primary preservation
|
||||
// signal and does not depend on the marker mechanism above.
|
||||
const cfg = sshExec(vmIp, SSH_USER, "cat /opt/vyatta/etc/config/config.boot", { keyPath: sshKeyPathGlobal });
|
||||
expect(cfg.stdout).toContain(`host-name "${HOSTNAME_R1}"`); // old config carried
|
||||
expect(cfg.stdout).not.toContain(`host-name "${HOSTNAME_R2}"`);
|
||||
|
||||
const markerAfter = sshExec(vmIp, SSH_USER, "cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
|
||||
expect(markerAfter.exitCode).toBe(0);
|
||||
expect(markerAfter.stdout).toContain("LAB-MARKER-R2");
|
||||
}, INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 60_000);
|
||||
|
||||
it("freshConfig makes the generated config win, data still kept (round 3)", async () => {
|
||||
// Re-assert the marker is on disk and synced before the next power-cut.
|
||||
const pre = sshExec(vmIp, SSH_USER, "sync && cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
|
||||
expect(pre.stdout).toContain("LAB-MARKER-R2");
|
||||
|
||||
vmIp = await installRound({ mac: vmMac, hostname: HOSTNAME_R3, freshConfig: true });
|
||||
|
||||
const cfg = sshExec(vmIp, SSH_USER, "cat /opt/vyatta/etc/config/config.boot", { keyPath: sshKeyPathGlobal });
|
||||
expect(cfg.stdout).toContain(`host-name "${HOSTNAME_R3}"`); // generated config won
|
||||
|
||||
// The marker file (non-config data under /config) still survives —
|
||||
// freshConfig replaces only config.boot, not the carried data.
|
||||
const markerAfter = sshExec(vmIp, SSH_USER, "cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
|
||||
expect(markerAfter.exitCode).toBe(0);
|
||||
expect(markerAfter.stdout).toContain("LAB-MARKER-R2");
|
||||
}, INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 60_000);
|
||||
});
|
||||
@@ -1,6 +1,7 @@
|
||||
{
|
||||
"files": [],
|
||||
"references": [
|
||||
{ "path": "src/core" },
|
||||
{ "path": "src/shared" },
|
||||
{ "path": "src/bastion" },
|
||||
{ "path": "src/cli" },
|
||||
|
||||
8
labsim/.gitignore
vendored
Normal file
8
labsim/.gitignore
vendored
Normal file
@@ -0,0 +1,8 @@
|
||||
# runtime artifacts, not source
|
||||
*.log
|
||||
labsim_matrix_lib.py
|
||||
__pycache__/
|
||||
|
||||
# Cluster-admin credentials for the rehearsal cluster, written by
|
||||
# `k8s-up.sh --kubeconfig`. Regenerate it rather than commit it.
|
||||
*.kubeconfig
|
||||
276
labsim/README.md
Normal file
276
labsim/README.md
Normal file
@@ -0,0 +1,276 @@
|
||||
# labsim — libvirt replica of the lab network
|
||||
|
||||
A throwaway copy of the production VLAN topology for testing routing, firewall
|
||||
rules and failover **without touching the real network**. Same VLAN IDs and
|
||||
roles as UniFi, deliberately different IP ranges so nothing can be confused for
|
||||
production.
|
||||
|
||||
## Topology
|
||||
|
||||
Each VLAN is its own isolated libvirt network with one tiny Alpine VM on it.
|
||||
|
||||
| VLAN | Name | Sim subnet | VM address | Mirrors production |
|
||||
|-----:|------|------------|-----------|--------------------|
|
||||
| 1 | management | 172.31.1.0/24 | 172.31.1.10 | 192.168.1.0/24 |
|
||||
| 2 | k8s | 172.31.2.0/24 | 172.31.2.10 | 192.168.8.0/23 |
|
||||
| 3 | kvm | 172.31.3.0/24 | 172.31.3.10 | 192.168.3.0/24 |
|
||||
| 9 | private | 172.31.9.0/24 | 172.31.9.10 | 10.0.9.0/23 |
|
||||
| 10 | lot | **172.31.10.0/23** | 172.31.10.10 | 10.0.0.0/23 |
|
||||
| 200 | roomates | 172.31.200.0/24 | 172.31.200.10 | 192.168.2.0/24 |
|
||||
|
||||
The sim subnet encodes the VLAN id: `172.31.<vlan>.0/24`, with one exception.
|
||||
**VLAN 10 is a `/23`** because every UniFi DHCP reservation lives in LoT and LoT
|
||||
spans `10.0.0.x` *and* `10.0.1.x`, which a `/24` cannot hold. The mapping stays
|
||||
readable — `10.0.0.46 → 172.31.10.46`, `10.0.1.67 → 172.31.11.67`.
|
||||
|
||||
LoT's host leg is `.3`, not `.2`, because `10.0.0.2` is a real reservation
|
||||
(Hubitat) that maps onto `172.31.10.2`. `.3` is unreserved and sits below the
|
||||
DHCP pool, so it can never be handed out.
|
||||
|
||||
`vlans.conf` therefore takes two optional trailing fields:
|
||||
|
||||
```
|
||||
vlan_id:name:sim_prefix:real_subnet[:masklen][:host_octet]
|
||||
```
|
||||
|
||||
defaulting to `24` and `2`. k8s and Private are also `/23` in production but
|
||||
hold no reservations, so they keep their `/24` and their DHCP range is clamped
|
||||
— reported at generation time, never silently.
|
||||
|
||||
Address plan, identical on every VLAN:
|
||||
|
||||
| Address | Role |
|
||||
|---------|------|
|
||||
| `.1` | gateway under test — a router VM you add (not created by default) |
|
||||
| `.2` | host bridge — how you reach the VMs from this workstation |
|
||||
| `.10` | the VLAN's micro VM |
|
||||
| `.254` | reserved for a VRRP VIP, mirroring production |
|
||||
|
||||
The host sits at `.2` purely so you can SSH in. It is deliberately **not** the
|
||||
VMs' default route — that is `.1` — so inter-VLAN tests fail loudly when no
|
||||
router is present instead of being silently served by the host's own routing
|
||||
table. libvirt also installs reject rules that stop these networks forwarding
|
||||
to each other, so traffic between VLANs only works once a router VM bridges
|
||||
them.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
./labsim-up.sh # bring up every VLAN (idempotent)
|
||||
./labsim-up.sh 2 3 # only VLANs 2 and 3
|
||||
./labsim-down.sh # destroy VMs + networks, keep the base image
|
||||
./labsim-down.sh --purge # also delete the downloaded Alpine image
|
||||
```
|
||||
|
||||
Each VM: 256 MB, 1 vCPU, a copy-on-write overlay on one shared 176 MB Alpine
|
||||
image (so six VMs cost a few MB of disk, not 1 GB).
|
||||
|
||||
## Access
|
||||
|
||||
```bash
|
||||
ssh alpine@172.31.2.10 # normal user (password: labsim)
|
||||
ssh root@172.31.2.10 # privileged — this image has no sudo
|
||||
curl http://172.31.2.10/ # hello-world page naming the VLAN
|
||||
```
|
||||
|
||||
Console, when the network is the thing that is broken:
|
||||
|
||||
```bash
|
||||
sudo virsh console labsim-2-k8s # root / labsim
|
||||
```
|
||||
|
||||
## Watching it
|
||||
|
||||
```bash
|
||||
./labsim-matrix.py --watch 2 # terminal grid, changed cells highlighted
|
||||
./monitoring-up.sh # topology page + Prometheus + Grafana
|
||||
```
|
||||
|
||||
## Testing the DHCP migration
|
||||
|
||||
`./labsim-dhcp-test.sh` boots throwaway VMs whose MACs are **real production
|
||||
MACs** and checks each gets the address UniFi reserved for it. MACs are the one
|
||||
piece of production config that transplants verbatim, which is what makes this a
|
||||
test rather than a rehearsal. It is safe because `ovs-labsim` has no physical
|
||||
NIC — verified with `ovs-vsctl show` — so a production MAC cannot reach the real
|
||||
LAN.
|
||||
|
||||
Apply the config first, from `../migration`:
|
||||
|
||||
```bash
|
||||
python3 unifi-to-vyos.py --mode sim -o /tmp/sim.conf # 6 subnets, 31 mappings
|
||||
# load onto labsim-vyos, then:
|
||||
./labsim-dhcp-test.sh
|
||||
```
|
||||
|
||||
**Result on VyOS 2026.08 (kea): all four cases pass.** The one that mattered:
|
||||
most UniFi reservations sit *inside* the DHCP pool, and **kea honours in-pool
|
||||
host reservations** — `printer1` received `172.31.10.46` from within the
|
||||
`.10.11–.11.254` pool. That was the open question blocking the cutover.
|
||||
|
||||
### The lease database will lie to you
|
||||
|
||||
The script wipes `/config/dhcp/dhcp4-leases.csv*` before every run, and both
|
||||
halves of that matter:
|
||||
|
||||
- **Stale leases defeat reservations.** Re-running against yesterday's leases,
|
||||
kea handed dynamic addresses to three devices that have reservations. The
|
||||
reservation was present and correct in `/run/kea/kea-dhcp4.conf` the whole
|
||||
time. Kea saw the reserved address as already leased to "another client" —
|
||||
same MAC, but a different client-id from the earlier boot — and allocated
|
||||
elsewhere. The cutover itself starts with an empty lease database, so this is
|
||||
a *testing* artifact, but it is worth knowing that a reservation is not an
|
||||
unconditional guarantee once leases exist.
|
||||
- **The `*` is load-bearing.** Kea's memfile backend keeps lease-file-cleanup
|
||||
rotations (`dhcp4-leases.csv.2`) and restores from them on start, so
|
||||
truncating only the primary file changes nothing.
|
||||
|
||||
Both of those first appeared as a *passing* test. The verdict logic now refuses
|
||||
to score a MAC with more than one lease, because taking the first match had
|
||||
reported an hours-old lease as the current answer and turned three failures
|
||||
into apparent passes.
|
||||
|
||||
Still open: whether kea will hand a *reserved* address to a *different* client
|
||||
while the reserved device is offline. The negative case here only proves an
|
||||
unreserved MAC gets an unreserved address.
|
||||
|
||||
- **http://localhost:9101/** — live mesh: a node per VLAN, the router in the
|
||||
middle, one line per pair coloured green/red with the ICMP RTT on it. Hover a
|
||||
line for per-direction detail. Refreshes every 5s. This is the one to watch
|
||||
while changing firewall rules.
|
||||
- **http://localhost:3000/d/labsim-matrix** — Grafana (anonymous, no login) for
|
||||
*history*: when did a path flip, and how has latency moved.
|
||||
- **http://localhost:9101/metrics** — `labsim_reachable{src,dst,proto}` and
|
||||
`labsim_rtt_ms{src,dst}`.
|
||||
|
||||
## Routing: BGP, dual WAN, and the ISP VMs
|
||||
|
||||
`sim-ha-config.py` covers the LAN side of the routers. `sim-net-config.py`
|
||||
covers everything that makes this a rehearsal for production *routing*:
|
||||
|
||||
| role | VM | what it generates |
|
||||
|---|---|---|
|
||||
| `primary` | `labsim-vyos` | BGP + dual WAN + health-checked failover |
|
||||
| `secondary` | `labsim-vyos2` | BGP only |
|
||||
| `isp-dhcp` | `labsim-isp-dhcp` | 10gig-equivalent ISP on VLAN 53 |
|
||||
| `isp-pppoe` | `labsim-isp-pppoe` | Vodafone-equivalent PPPoE ISP on VLAN 51 |
|
||||
|
||||
Both ISP VMs are VyOS with two NICs: one on the OVS trunk facing the sim
|
||||
router, one on libvirt's `default` network, NATing customers to the real
|
||||
internet. They use RFC 5737 documentation ranges (`203.0.113.0/24`,
|
||||
`198.51.100.0/24`) so a leaked sim route cannot blackhole anything real.
|
||||
|
||||
```sh
|
||||
./sim-net-apply.sh check # VM state vs what the code says — run this first
|
||||
./sim-net-apply.sh apply # push generated config over the serial console
|
||||
```
|
||||
|
||||
`check` is the important one. All of this previously existed only as running
|
||||
state, applied by hand over SSH; rebuilding a VM lost it, and nothing recorded
|
||||
why any of it was shaped the way it was.
|
||||
|
||||
### Known gaps vs production
|
||||
|
||||
- **WAN is on the primary router only.** Production has WAN on both. Two PPPoE
|
||||
clients sharing one credential against a single access concentrator is a
|
||||
failure mode production does not have, so the sim does not model it. VRRP and
|
||||
conntrack failover are still exercised.
|
||||
- **ISP VM interface names are not stable across a rebuild** — `isp-dhcp` came
|
||||
up as `eth0`/`eth1` and `isp-pppoe` as `eth2`/`eth3` from identical XML.
|
||||
Check `show interfaces` and pass `--wan-if` / `--uplink-if` rather than
|
||||
trusting the defaults.
|
||||
- **`eth2` on the primary router** is a libvirt-NAT uplink predating the ISP
|
||||
VMs: a third default route with no production equivalent that masks real WAN
|
||||
failures during a failover test. `--drop-scaffold` removes it.
|
||||
- **Committing on `isp-pppoe` drops the router's PPPoE session**, and the
|
||||
client does not redial promptly. After any change there, check `pppoe0` on
|
||||
the router and `sudo systemctl restart ppp@pppoe0` if it is missing.
|
||||
|
||||
## The trunk carries every VLAN tagged, including Management
|
||||
|
||||
There is deliberately **no native/untagged VLAN** on the trunks to the routers,
|
||||
and Management lives on `bond0.1`, not on the bare `bond0`.
|
||||
|
||||
A native VLAN is what puts a subnet on the bond **parent** while every other
|
||||
VLAN sits on a sub-interface of it. With `dhcp-socket-type: raw`, kea receives
|
||||
each tagged frame *twice* — once on the sub-interface and once on the parent —
|
||||
and answers from the parent's pool as well (ISC Kea
|
||||
[#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117)). A client on
|
||||
VLAN 3 gets two OFFERs and keeps whichever arrives first:
|
||||
|
||||
```
|
||||
bond0.3 : 172.31.3.252 → 172.31.3.11 correct
|
||||
bond0 : 172.31.1.252 → 172.31.1.8 UNTAGGED, Management pool, wrong
|
||||
```
|
||||
|
||||
`./labsim-vlan-leak-test.sh` makes one client on a tagged VLAN send a DISCOVER
|
||||
and captures on the parent and the sub-interface at once. The verdict is how
|
||||
many OFFERs the **server** emitted and from which subnets — deliberately not
|
||||
"did the client get the right address", because a client picking correctly is
|
||||
exactly how this hid. Both orderings were observed across runs, so a passing
|
||||
client proves nothing.
|
||||
|
||||
```sh
|
||||
./labsim-vlan-leak-test.sh --vlan 3 # PASS on the current shape
|
||||
LABSIM_NATIVE_VLAN=1 ./router-up.sh # restore the old shape...
|
||||
./labsim-vlan-leak-test.sh --vlan 3 # ...and it FAILs again
|
||||
```
|
||||
|
||||
Three things this cost, all of which apply to production:
|
||||
|
||||
- **Kea must be restarted after the address moves.** VyOS does not restart it
|
||||
for an interface address change, so it keeps a raw socket bound with the old
|
||||
address and the bug survives the fix. In the sim kea had been running since
|
||||
16 Aug; the first post-fix test failed for this reason alone and looked like
|
||||
the fix simply not working.
|
||||
- **The firewall interface-group must move too.** `interface-group LAN` named
|
||||
the bare `bond0`; with a default-deny ruleset, moving the address without
|
||||
moving the group drops every management session and all VLAN 1 routing.
|
||||
- **Duplicate delivery does not stop.** #1117 says only that there is no longer
|
||||
a subnet on the parent to match, and that is exactly what happens: two replies
|
||||
per DISCOVER, both now from the correct pool. Harmless, but do not read a
|
||||
duplicate as a failure.
|
||||
|
||||
### Tagged and untagged Management coexist
|
||||
|
||||
Verified directly, and it is what makes the production cutover a rolling change
|
||||
rather than an outage: with the primary still untagged on `bond0` and the
|
||||
secondary already tagged on `bond0.1`, both routers were reachable, the VIP
|
||||
stayed up and a VLAN 1 client kept its gateway. One VLAN is one broadcast
|
||||
domain regardless of how each port tags it, so the two firewalls can be
|
||||
converted one at a time. See `migration/MANAGEMENT-VLAN-TAGGED.md`.
|
||||
|
||||
`./vlan1-move-monitor.sh` logs VIP/router liveness once a second during the
|
||||
change, because VRRP reconverges and leaves no trace of who held the VIP.
|
||||
|
||||
## Notes for whoever extends this
|
||||
|
||||
Things that cost time the first time round, all verified on this image:
|
||||
|
||||
- **No `sudo`.** Alpine ships `doas`; cloud-init's `sudo:` directive is inert
|
||||
here. Use `root@` for privileged work.
|
||||
- **cloud-init leaves users locked** (`!*` in `/etc/shadow`) unless
|
||||
`lock_passwd: false`, and sshd then refuses key auth for that user.
|
||||
- **One failing `runcmd` aborts every command after it.** Each entry is
|
||||
`|| true` for that reason.
|
||||
- **busybox here has no `httpd` applet**, and the VMs have no internet to
|
||||
`apk add` one — so the hello-world server is `python3 -m http.server`
|
||||
(python3 is already present because cloud-init depends on it).
|
||||
- **`start-stop-daemon --exec /usr/bin/python3` matches cloud-init's own
|
||||
python3** at boot and refuses to start anything.
|
||||
- **busybox `pgrep -f PATTERN` matches its own argv**, so a "skip if already
|
||||
running" guard always fires. Verified: `guard_exit=0` with nothing listening.
|
||||
|
||||
## Not modelled (yet)
|
||||
|
||||
- **The secondary's bond was fiction until 2026-09-02.** `ovs_bond_router`'s
|
||||
"already bonded, nothing to do" check compared only the trunk VLAN list, not
|
||||
the membership. Restarting a VM recreates its taps under new names, so the
|
||||
bond sat there holding two interfaces that no longer existed while the router's
|
||||
real taps ran in the bridge as two *independent* ports — no LACP, and carrying
|
||||
libvirt's own portgroup VLAN config rather than the bond's. It reconciles
|
||||
membership now, but the lesson generalises: a sim that reports success is not
|
||||
the same as a sim that models the thing.
|
||||
- **`labsim-vyos` has a third NIC** on libvirt's `default` network (the scaffold
|
||||
uplink, see `--drop-scaffold`). The tap count is filtered to `$OVS_NET` for
|
||||
that reason; an unfiltered count is 3 and silently skipped the primary's bond.
|
||||
144
labsim/cilium-ipam-switch.sh
Executable file
144
labsim/cilium-ipam-switch.sh
Executable file
@@ -0,0 +1,144 @@
|
||||
#!/usr/bin/env bash
|
||||
# Procedure around a Cilium IPAM mode change. Works against any cluster, so the
|
||||
# rehearsal in labsim and the real thing in production run the SAME steps.
|
||||
#
|
||||
# It deliberately does NOT change the mode itself. In labsim that is `helm
|
||||
# upgrade`; in production Pulumi owns the release and a script racing it would
|
||||
# just reintroduce drift. What this owns is everything around the apply -- the
|
||||
# evidence, the deadlock, and the verdict.
|
||||
#
|
||||
# ./cilium-ipam-switch.sh preflight record what the cluster looks like now
|
||||
# ./cilium-ipam-switch.sh unstick break the agent-not-ready taint deadlock
|
||||
# ./cilium-ipam-switch.sh verify compare against preflight, report renumbering
|
||||
#
|
||||
# KUBECONFIG=... ./cilium-ipam-switch.sh preflight
|
||||
#
|
||||
# Whether a recycle is needed is CONDITIONAL, and `verify` is what decides it.
|
||||
#
|
||||
# The operator does not preserve which node held which /24 -- it adopts whatever
|
||||
# CiliumNode.spec.ipam.podCIDRs already says. So:
|
||||
#
|
||||
# * If CiliumNode already agrees with node.spec.podCIDRs on every node -- which
|
||||
# is the case for any cluster that has only ever run ipam=kubernetes, because
|
||||
# the operator syncs one from the other -- the pool adopts the existing
|
||||
# allocation, no node is renumbered, and NO pod recycle is needed. Verified
|
||||
# on the 3-node labsim cluster: CIDRs unchanged, nothing stranded, the only
|
||||
# blip was the cilium DaemonSet restarting itself.
|
||||
#
|
||||
# * If the two sources DISAGREE, nodes can swap /24s. Their running pods keep
|
||||
# addresses that no longer fall inside the node's range, every other node
|
||||
# routes that prefix to the wrong node, and those pods go unreachable
|
||||
# cross-node while still showing Running. Then a full recycle is mandatory.
|
||||
#
|
||||
# Do not skip `verify` on the assumption of the good case. Run it and read it.
|
||||
set -uo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
STATE="${STATE:-$SCRIPT_DIR/.ipam-switch-state}"
|
||||
K="kubectl"
|
||||
|
||||
say() { printf '\033[0;36m[ipam]\033[0m %s\n' "$*"; }
|
||||
warn() { printf '\033[1;33m[ipam]\033[0m %s\n' "$*" >&2; }
|
||||
|
||||
snapshot() {
|
||||
echo "## nodes"
|
||||
$K get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDRs}{"\n"}{end}' 2>/dev/null
|
||||
echo "## ciliumnodes"
|
||||
$K get ciliumnode -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipam.podCIDRs}{"\n"}{end}' 2>/dev/null
|
||||
echo "## pods"
|
||||
$K get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}' 2>/dev/null \
|
||||
| grep -vP '\t$' | sort
|
||||
echo "## ipam"
|
||||
$K -n kube-system get cm cilium-config -o jsonpath='{.data.ipam}' 2>/dev/null; echo
|
||||
}
|
||||
|
||||
cmd_preflight() {
|
||||
mkdir -p "$STATE"
|
||||
snapshot > "$STATE/before.txt"
|
||||
say "recorded $(grep -c . "$STATE/before.txt") lines -> $STATE/before.txt"
|
||||
say "mode now: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1)"
|
||||
# The pod inventory is the rollback reference: if the switch renumbers, this
|
||||
# is the only record of what an address USED to be.
|
||||
say "pods on the pod network: $(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -c '10\.')"
|
||||
}
|
||||
|
||||
# The deadlock, in one place because it WILL happen and doing it by hand under
|
||||
# time pressure is how the wrong node gets untainted:
|
||||
# agent has no pod CIDR -> agent not ready -> node keeps
|
||||
# node.cilium.io/agent-not-ready:NoSchedule -> the operator that would assign
|
||||
# the CIDR cannot schedule -> agent still has no pod CIDR.
|
||||
# Removing the taint is safe: it exists to keep normal workloads off a node
|
||||
# without working networking, and the operator is precisely the thing that fixes
|
||||
# that. Kubernetes re-adds it on the next agent restart.
|
||||
cmd_unstick() {
|
||||
local stuck=0
|
||||
for n in $($K get nodes -o name 2>/dev/null); do
|
||||
$K get "$n" -o jsonpath='{.spec.taints[*].key}' 2>/dev/null | grep -q 'agent-not-ready' || continue
|
||||
warn "${n#node/} carries agent-not-ready; removing so the operator can schedule"
|
||||
$K taint "$n" node.cilium.io/agent-not-ready- >/dev/null 2>&1 && stuck=$((stuck+1))
|
||||
done
|
||||
[ "$stuck" -eq 0 ] && say "no node was stuck" || say "cleared $stuck node(s)"
|
||||
local pend
|
||||
pend="$($K -n kube-system get pods -l io.cilium/app=operator --no-headers 2>/dev/null | grep -c Pending)"
|
||||
[ "${pend:-0}" -gt 0 ] && warn "$pend operator pod(s) still Pending — check tolerations, not just taints"
|
||||
return 0
|
||||
}
|
||||
|
||||
cmd_verify() {
|
||||
[ -f "$STATE/before.txt" ] || { warn "no preflight snapshot; nothing to compare"; return 1; }
|
||||
snapshot > "$STATE/after.txt"
|
||||
echo
|
||||
say "mode: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1) -> $(sed -n '/^## ipam/,$p' "$STATE/after.txt" | tail -1)"
|
||||
|
||||
# The question that decides the size of the maintenance window: did per-node
|
||||
# CIDRs survive, or was every node renumbered (and every pod with it)?
|
||||
local moved=0
|
||||
while IFS=$'\t' read -r node cidr; do
|
||||
[ -z "${node:-}" ] && continue
|
||||
local now; now="$(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/after.txt" | awk -F'\t' -v n="$node" '$1==n{print $2}')"
|
||||
if [ -n "$now" ] && [ "$now" != "$cidr" ]; then
|
||||
printf ' %-16s %s -> %s\n' "$node" "$cidr" "$now"; moved=$((moved+1))
|
||||
fi
|
||||
done < <(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/before.txt" | grep -P '\t')
|
||||
if [ "$moved" -eq 0 ]; then
|
||||
say "per-node CIDRs UNCHANGED — the pool adopted the existing allocation"
|
||||
else
|
||||
warn "$moved node(s) renumbered — every pod on them must be recycled"
|
||||
fi
|
||||
|
||||
local before after same
|
||||
before="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | wc -l)"
|
||||
after="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt" | grep -P '\t10\.' | wc -l)"
|
||||
same="$(comm -12 <(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | sort) \
|
||||
<(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt" | grep -P '\t10\.' | sort) | wc -l)"
|
||||
say "pods: $before before, $after after, $same kept the SAME address"
|
||||
# Keeping the address is NOT the good outcome. If a node's CIDR moved, its
|
||||
# existing pods keep IPs that no longer fall inside it, every other node routes
|
||||
# that prefix to the WRONG node, and those pods go unreachable cross-node while
|
||||
# looking perfectly healthy. Observed in labsim: two nodes swapped CIDRs and
|
||||
# cross-node ping to their pods dropped 100%, with every pod still Running.
|
||||
# This is the check that decides whether a recycle is optional or mandatory.
|
||||
local stranded=0
|
||||
while read -r ns name ip node; do
|
||||
[ -z "${node:-}" ] && continue
|
||||
local cidr; cidr="$($K get ciliumnode "$node" -o jsonpath='{.spec.ipam.podCIDRs[0]}' 2>/dev/null)"
|
||||
[ -z "$cidr" ] && continue
|
||||
case "$ip" in
|
||||
"${cidr%.*/*}".*) ;;
|
||||
*) printf ' STRANDED %-40s %-15s on %s (now %s)\n' "$ns/$name" "$ip" "$node" "$cidr"; stranded=$((stranded+1)) ;;
|
||||
esac
|
||||
done < <($K get pods -A -o jsonpath='{range .items[?(@.status.podIP)]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.podIP}{" "}{.spec.nodeName}{"\n"}{end}' 2>/dev/null | grep -E ' 10\.')
|
||||
if [ "$stranded" -gt 0 ]; then
|
||||
warn "$stranded pod(s) sit OUTSIDE their node CIDR — unreachable cross-node until recycled"
|
||||
warn "recycle: for ns in $(kubectl get ns -o name | cut -d/ -f2); do kubectl -n $ns rollout restart deploy,ds,sts 2>/dev/null; done"
|
||||
else
|
||||
say "every pod is inside its node CIDR — no recycle needed"
|
||||
fi
|
||||
say "not-Running pods: $($K get pods -A --no-headers 2>/dev/null | grep -vcE 'Running|Completed')"
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
preflight) cmd_preflight ;;
|
||||
unstick) cmd_unstick ;;
|
||||
verify) cmd_verify ;;
|
||||
*) sed -n '2,16p' "$0"; exit 1 ;;
|
||||
esac
|
||||
109
labsim/console-apply.py
Executable file
109
labsim/console-apply.py
Executable file
@@ -0,0 +1,109 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Apply VyOS config to a labsim VM over its serial console.
|
||||
|
||||
Needed because a freshly installed VyOS comes up holding the same addresses as
|
||||
its peer, so there is a window where it cannot safely be reached over the
|
||||
network at all. The console does not care.
|
||||
|
||||
./console-apply.py --vm labsim-vyos2 --config r2.conf
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import time
|
||||
|
||||
import pexpect
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--vm", required=True)
|
||||
ap.add_argument("--config", required=True)
|
||||
ap.add_argument("--user", default="vyos")
|
||||
ap.add_argument("--password", default="vyos")
|
||||
args = ap.parse_args()
|
||||
|
||||
cmds = [l.rstrip() for l in open(args.config)
|
||||
if l.strip() and not l.lstrip().startswith("#")]
|
||||
print(f"{len(cmds)} commands to apply to {args.vm}", file=sys.stderr)
|
||||
|
||||
c = pexpect.spawn(f"virsh --connect qemu:///system console {args.vm}",
|
||||
timeout=90, encoding="utf-8")
|
||||
c.logfile_read = None
|
||||
c.sendline("")
|
||||
time.sleep(2)
|
||||
c.sendline("")
|
||||
|
||||
# Log in. A freshly booted box may still be starting services, so allow a
|
||||
# generous window and re-prod the console rather than failing on the first
|
||||
# miss.
|
||||
#
|
||||
# `# ` matters as much as `$ `: a previous run that died mid-config leaves
|
||||
# the console sitting in configuration mode, and waiting only for the
|
||||
# operational prompt then hangs forever against a perfectly healthy VM.
|
||||
in_config = False
|
||||
for _ in range(40):
|
||||
i = c.expect([r"login:", r"\$ ", r"# ", pexpect.TIMEOUT], timeout=15)
|
||||
if i == 0:
|
||||
c.sendline(args.user)
|
||||
c.expect("Password:", timeout=30)
|
||||
c.sendline(args.password)
|
||||
c.expect([r"\$ ", r"# "], timeout=60)
|
||||
break
|
||||
if i == 1:
|
||||
break
|
||||
if i == 2:
|
||||
in_config = True
|
||||
break
|
||||
c.sendline("")
|
||||
else:
|
||||
print("never reached a prompt", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if in_config:
|
||||
# Drop whatever the previous run left half-built rather than committing
|
||||
# a candidate nobody has seen.
|
||||
print("console was left in config mode; discarding stale candidate",
|
||||
file=sys.stderr)
|
||||
c.sendline("discard")
|
||||
c.expect(r"# ", timeout=60)
|
||||
else:
|
||||
c.sendline("configure")
|
||||
c.expect(r"# ", timeout=60)
|
||||
|
||||
for cmd in cmds:
|
||||
c.sendline(cmd)
|
||||
c.expect(r"# ", timeout=60)
|
||||
out = c.before or ""
|
||||
if "Set failed" in out or "not valid" in out or "Invalid" in out:
|
||||
print(f"FAILED: {cmd}\n {out.strip()[:200]}", file=sys.stderr)
|
||||
|
||||
print("committing...", file=sys.stderr)
|
||||
c.sendline("commit")
|
||||
c.expect(r"# ", timeout=300)
|
||||
commit_out = c.before or ""
|
||||
c.sendline("save")
|
||||
c.expect(r"# ", timeout=120)
|
||||
# Accept either prompt on the way out. Insisting on `$ ` here hangs against
|
||||
# a healthy box -- and worse, leaves the console parked in config mode, so
|
||||
# the NEXT run finds a `# ` it was not expecting either. One strict expect
|
||||
# turned into two failures.
|
||||
c.sendline("exit")
|
||||
c.expect([r"\$ ", r"# ", pexpect.TIMEOUT], timeout=60)
|
||||
c.sendline("exit")
|
||||
c.close(force=True)
|
||||
|
||||
bad = [l for l in commit_out.splitlines()
|
||||
if "failed" in l.lower() or "error" in l.lower()]
|
||||
if bad:
|
||||
print("commit reported:", file=sys.stderr)
|
||||
for l in bad[:10]:
|
||||
print(f" {l.strip()}", file=sys.stderr)
|
||||
return 1
|
||||
print("committed and saved", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user