user@homelab:~/docs$ cat troubleshooting.md

Troubleshooting

Technical problems hit while building the homelab: symptom, cause and fix

# contents
#SymptomCauseFix
1Windows installer requires network and a Microsoft accountdefault Win11 Home behaviourShift+F10 → start ms-cxh:localonly, Ventoy VTOY_WIN11_BYPASS_NRO
2After installing Windows, USB no longer bootsWindows Boot Manager first in UEFINovo → Boot Menu → EFI USB Device
3Ubuntu: Probing for devices to install to failed, Invalid dep_id (disk-dm-1)Ventoy maps the ISO via device-mapperVentoy → Boot in grub2 mode
4Ubuntu: no Add GPT Partition and no free spacein grub2 mode the installer did not scan gaps between partitionssgdisk -n 0:0:0 … in the installer shell, then reboot
5Wi-Fi does not work (not even a hotspot)rfkill soft=1 + empty wifis: {} in netplanunblock via /sys/class/rfkill/*/soft + 50-wifi.yaml
6No rfkill in the installerminimal environmentinstall without network, fix in the installed system
7GRUB menu not shownUbuntu Server has GRUB_TIMEOUT=0GRUB_TIMEOUT_STYLE=menu, GRUB_TIMEOUT=5
8Windows listed twice in GRUBos-prober + a manual entry in 40_customremove the manual entry
9Reservation on the ISP router has no effectaddresses are handed out by the own Wi-Fi router behind itDHCP reservation on the router that is the gateway (route -n get default)
10Flux: apps reports dependency ‘flux-system/infra’ is not readyafter a new commit infra syncs first (dependsOn)nothing, it resolves shortly; to speed up: flux reconcile kustomization apps

Useful diagnostic commands

shell @ server
lsblk -o NAME,SIZE,MODEL,MOUNTPOINTS          # disks and partitions
sudo parted /dev/sda unit GiB print free      # including free space
grep . /sys/class/rfkill/*/{name,soft,hard}   # radio blocks
ip -br addr ; ip route                        # addresses and gateway
networkctl status wlp3s0 --no-pager           # Wi-Fi state
sudo journalctl -u ssh -n 20 --no-pager       # why SSH rejected a login
sudo sshd -T | grep -i auth                   # effective sshd config
sudo ufw status verbose                       # firewall
k get pods -A ; k get events -A --sort-by=.lastTimestamp | tail

Lessons learned

  • No network? Scripts on a USB disk: diagnostics and configuration as a script, with output saved to a file instead of retyping.
  • Test key login before disabling passwords, and keep a second SSH session open as a safety net.
  • Fixed IP before k3s: k3s writes the address into its certificates and kubeconfig.