NVMe Boot Failure — Debugging Report

Problem

Boot fails with: ERROR: Failed to mount 'UUID=bbbe5f36-54e5-4799-841f-168977356f2b' on real root
→ Drops into emergency shell.

System Info

  • OS: Artix Linux (base-openrc), kernel 7.1.4-artix1-1, mkinitcpio 41
  • Bootloader: syslinux (MBR boot, NOT UEFI)
  • NVMe: /dev/nvme0n1p1 — ext4, UUID bbbe5f36-54e5-4799-841f-168977356f2b, 476.9G
  • Disk layout: sda = Live USB, sdb = 3.6T disk (sdb1=EFI empty, sdb2=ext4 "ssd"), nvme0n1 = target
  • Live env: Artix Linux ISO booted from sda

Root Causes Found (4 issues)

1. Wrong MODULES in mkinitcpio.conf

MODULES=(nvme_core nvme ahci ext4 xhci_pci virtio_net r8169)
  • ahci does NOT exist as .ko in kernel 7.1.4 (only ahci_platform.ko)
  • ext4 is built-in (CONFIG_EXT4_FS=y) — no module needed
  • Fix: MODULES=(nvme_core nvme nvme_auth nvme_keyring ext4 xhci_pci virtio_net r8169)
  • Added nvme_auth and nvme_keyring (dependencies of nvme-core)

2. autodetect hook stripped NVMe modules

The autodetect hook analyzed only the Live USB hardware (no NVMe present), so it stripped all NVMe-related modules from the initramfs. Only 32 modules were included instead of ~1053.
- Fix: Removed autodetect from HOOKS

3. MODULES_DECOMPRESS="yes" caused path mismatch in module dependency databases

  • install_modules() adds .ko.zst files, then decompresses them to .ko in the initramfs
  • But modules.dep.bin still referenced the original .ko.zst paths
  • When udev triggers modprobe, it reads modules.dep.bin, tries to load nvme-core.ko.zst → file not found (only nvme-core.ko exists)
  • The depmod regeneration line in mkinitcpio functions is commented out (line 1339 of /usr/lib/initcpio/functions)
  • Fix: MODULES_DECOMPRESS="no" — keeps modules as .ko.zst, paths match

4. Module dependency metadata not included in initramfs

The install_modules() function only copies modules.{builtin,builtin.modinfo,order} (line 1335-1337 of /usr/lib/initcpio/functions). It does NOT copy:
- modules.alias / modules.alias.bin (needed by udev to match PCI device IDs → module names)
- modules.dep / modules.dep.bin (module dependency resolution)
- modules.softdep (soft dependency info)

Without modules.alias.bin, udev cannot resolve PCI vendor/device IDs to kernel module names → nvme module never gets auto-loaded.

  • Fix: Custom moduledeps hook at /usr/lib/initcpio/install/moduledeps:
#!/usr/bin/env bash
build() {
    local kver="$KERNELVERSION"
    local moddir="/usr/lib/modules/$kver"
    for f in modules.alias modules.dep modules.softdep modules.alias.bin modules.dep.bin modules.symbols.bin modules.weakdep; do
        add_file "$moddir/$f" "$moddir/$f" 0444
    done
}
help() {
    cat <<HELPEOF
This hook adds module dependency metadata (modules.alias, modules.dep, etc.)
required for udev to auto-load modules based on PCI/device aliases.
HELPEOF
}

Added moduledeps to HOOKS after block.

Changes Made

/etc/mkinitcpio.conf

MODULES=(nvme_core nvme nvme_auth nvme_keyring ext4 xhci_pci virtio_net r8169)
MODULES_DECOMPRESS="no"
HOOKS=(base udev microcode modconf kms keyboard keymap consolefont block moduledeps filesystems fsck)

/boot/syslinux/syslinux.cfg

APPEND root=UUID=bbbe5f36-54e5-4799-841f-168977356f2b rw
  • Removed debug break=premount (was added for diagnostics)

/usr/lib/initcpio/install/moduledeps (NEW)

Custom hook — see above.

/boot/initramfs-linux.img

Rebuilt with mkinitcpio -p linux after all changes.

Verification

QEMU test passed: NVMe module auto-loaded via udev, root mount successful.

:: mounting 'UUID=bbbe5f36-54e5-4799-841f-168977356f2b' on real root
=== INITRAMFS TEST: Minimal init reached! ===
Root mount SUCCESSFUL!

Debugging Commands Used

# Check what's in the initramfs
lsinitcpio /boot/initramfs-linux.img

# Check modules.dep for path references
grep nvme /usr/lib/modules/7.1.4-artix1-1/modules.dep

# Check if module files exist on disk
ls /usr/lib/modules/7.1.4-artix1-1/modules.alias.bin
ls /usr/lib/modules/7.1.4-artix1-1/modules.dep.bin

# QEMU test with real kernel/initramfs
qemu-system-x86_64 -m 2G \
  -kernel /boot/vmlinuz-linux \
  -initrd /boot/initramfs-linux.img \
  -append "root=UUID=bbbe5f36-54e5-4799-841f-168977356f2b rw console=ttyS0" \
  -drive if=none,id=nvme0,file=/tmp/nvme-test.raw,format=raw \
  -device nvme,serial=deadbeef,drive=nvme0 \
  -nographic -no-reboot

# Rebuild initramfs from chroot
mount --bind /proc /mnt/proc && mount --bind /sys /mnt/sys && \
mount --bind /dev /mnt/dev && mount --bind /run /mnt/run && \
chroot /mnt /bin/sh -c "mkinitcpio -p linux"

Key Files for Reference

  • /mnt/usr/lib/initcpio/functions — mkinitcpio core: install_modules() at line 1300, module file copy at 1335-1337, commented-out depmod at 1339
  • /mnt/usr/lib/initcpio/install/block — block hook: calls add_checked_modules '/(block|ata|scsi|fusion|nvme)/'
  • /mnt/usr/lib/initcpio/install/modconf — only copies /etc/modprobe.d/*.conf and /usr/lib/modprobe.d/*.conf
  • /mnt/boot/config-7.1.4-artix1-1 — does NOT exist on disk
  • Backup configs: /mnt/boot/syslinux/syslinux.cfg.bak.20260709_183835, /mnt/etc/mkinitcpio.conf.bak.20260709_174658