initramfs debugging

After getting a bit excited by the release of Debian Trixie, I upgraded a couple of machines and it all went fine. Then I upgraded another and it didn't, due to me diligently reading the Release Notes and then promptly forgetting about the known Intel nic issue.

That's fine, I thought, this is why I set up serial port with a getty on it! Alas, I also hit a poorly understood dropbear/initrams issue which totally fucked the console on startup.

Pas de problème, I thought, I can just boot in to init=/bin/bash and fix it. Hm, well, that doesn't work super well, since that doesn't do anything until after the initramfs has run, which is currently spewing command not found errors at 100Hz.

No worries, I thought, this is why I left a USB stick with a bootable Debian Live CD on it, physically plugged in to the machine! Alas, once I booted in to it, I remembered that my root filesystem was on ZFS, and I'd upgraded to ZFS 2.3, and the Live CD was Bookworm, and the machine had quite...elaborate network connectivity, which mostly required bird2 to be up and speaking BGP to the neighbours.

Some worries, I thought, given the machine is not in the same country as me.

These days¹, though, we have an initramfs, and they're fucking massive, like:

-rw------- 1 root root 40M Aug  9 18:31 /boot/initrd.img-6.1.0-37-amd64
-rw------- 1 root root 44M Aug  9 21:15 /boot/initrd.img-6.12.38+deb13-amd64

that's bigger than my first LFS install was! Surely I can screw around with the boot parameters to get a shell. I eventually found the Debian wiki page for InitramsDebug, which explains that the devs were way ahead of me - there's a break= kernel parameter that the initramfs machinery consumes, which initramfs-tools(7) helpfully elaborates on:

break spawns a shell in the initramfs image at the chosen phase
(top, modules, premount, mount, mountroot, bottom, init) before
actually executing the corresponding scripts

So, I just needed break=bottom and I'd get dropped in to a root shell in the initramfs, with everything up to that point done, whereby I could just fix up the interface naming in /etc so I'd get network on boot, and then slightly more complicatedly, bind mount /dev, /sys and /proc in to /root, chroot in to the rootfs, dpkg -P initramfs-dropbear and update-initramfs -k all -c to remove the broken initramfs script from dropbear.

Et voila.

Some unused fallback plans:

• boot in to GRML and see if could use it to build recent enough kernel modules without a network
• boot in to USB, then ask someone on the same L2 if I could configure a sneaky static default route to them to get ZFS tools on to the live cd and fix things
• try to build a custom Debian Live ISO with more tooling and kernel modules, and then post it on a usb key to $city
• cry
• go to $city