I just installed NixOS on a machine that uses ZFS for the filesystem. The install was clean (after lots of fiddling around), but I’m getting an error when I boot. The error says:
starting decive mapper and LVM...
importing root ZFS pool "zroot"...
Enter passphrase for 'zroot':
1 / 1 key(s) sucessfully loaded
mounting zroot/root on /...
mounting zroot/nix on /nix...
mounting zroot/var on /var...
stage 2 init script (/mnt-root//nix/store/...<lots of chars>.../init) not found
An error occurred in stage 1 of the boot process, which must mount the root filesystem on`/mnt-root' and then start stage 2.
Here’s a pic:
There’s also some error messages about invalid report_size. No idea what that means.
My first guess is that something isn’t getting mounted in the right place, but I’m not sure what. Does anyone have any idea what might be going on? Happy to share configuration files, if needed.
Thanks in advance.
Pretty much impossible to know what’s going wrong without seeing the configs.
Roger that. Here are my configuration files.
configuration.nix
# Edit this configuration file to define what should be installed on
# your system. Help is available in the configuration.nix(5) man page, on
# https://search.nixos.org/options and in the NixOS manual (`nixos-help`).
{ config, lib, pkgs, ... }:
{
imports =
[ # Include the results of the hardware scan.
./hardware-configuration.nix
];
# Allow unfree
nixpkgs.config.allowUnfree = true;
# Use the GRUB 2 boot loader.
boot.loader.efi.canTouchEfiVariables = true;
boot.loader.grub = {
enable = true;
zfsSupport = true;
efiSupport = true;
device = "nodev";
mirroredBoots = [
{
devices = [ "/dev/disk/by-id/nvme-Samsung_SSD_970_PRO_1TB_S462NF0K607654A-part1" ];
path = "/boot";
}
{
devices = [ "/dev/disk/by-id/nvme-Samsung_SSD_970_PRO_1TB_S462NF0K609877B-part1" ];
path = "/boot-fallback";
}
];
};
# ZFS
boot.supportedFilesystems = [ "zfs" ];
boot.zfs.extraPools = [ "zroot" ];
services.zfs.autoScrub.enable = true;
# Nvidia
hardware.opengl = {
enable = true;
};
services.xserver.videoDrivers = [ "nvidia" ];
hardware.nvidia = {
modesetting.enable = true;
powerManagement.enable = false;
powerManagement.finegrained = false;
open = false;
nvidiaSettings = true;
package = config.boot.kernelPackages.nvidiaPackages.stable;
};
# Networking
networking.hostName = "workhorse"; # Define your hostname.
networking.hostId = "4B10C5DD";
# Pick only one of the below networking options.
# networking.wireless.enable = true; # Enables wireless support via wpa_supplicant.
networking.networkmanager.enable = true; # Easiest to use and most distros use this by default.
# Set your time zone.
time.timeZone = "America/New_York";
# Configure network proxy if necessary
# networking.proxy.default = "http://user:password@proxy:port/";
# networking.proxy.noProxy = "127.0.0.1,localhost,internal.domain";
# Select internationalisation properties.
# i18n.defaultLocale = "en_US.UTF-8";
# console = {
# font = "Lat2-Terminus16";
# keyMap = "us";
# useXkbConfig = true; # use xkb.options in tty.
# };
# Enable the X11 windowing system.
# services.xserver.enable = true;
# Configure keymap in X11
# services.xserver.xkb.layout = "us";
# services.xserver.xkb.options = "eurosign:e,caps:escape";
# Enable CUPS to print documents.
# services.printing.enable = true;
# Enable sound.
# hardware.pulseaudio.enable = true;
# OR
# services.pipewire = {
# enable = true;
# pulse.enable = true;
# };
# Enable touchpad support (enabled default in most desktopManager).
# services.libinput.enable = true;
# Define a user account. Don't forget to set a password with ‘passwd’.
# users.users.alice = {
# isNormalUser = true;
# extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
# packages = with pkgs; [
# firefox
# tree
# ];
# };
# List packages installed in system profile. To search, run:
# $ nix search wget
# environment.systemPackages = with pkgs; [
# vim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
# wget
# ];
# Some programs need SUID wrappers, can be configured further or are
# started in user sessions.
# programs.mtr.enable = true;
# programs.gnupg.agent = {
# enable = true;
# enableSSHSupport = true;
# };
# List services that you want to enable:
# Enable the OpenSSH daemon.
# services.openssh.enable = true;
# Open ports in the firewall.
# networking.firewall.allowedTCPPorts = [ ... ];
# networking.firewall.allowedUDPPorts = [ ... ];
# Or disable the firewall altogether.
# networking.firewall.enable = false;
# Copy the NixOS configuration file and link it from the resulting system
# (/run/current-system/configuration.nix). This is useful in case you
# accidentally delete configuration.nix.
# system.copySystemConfiguration = true;
# This option defines the first version of NixOS you have installed on this particular machine,
# and is used to maintain compatibility with application data (e.g. databases) created on older NixOS versions.
#
# Most users should NEVER change this value after the initial install, for any reason,
# even if you've upgraded your system to a new NixOS release.
#
# This value does NOT affect the Nixpkgs version your packages and OS are pulled from,
# so changing it will NOT upgrade your system - see https://nixos.org/manual/nixos/stable/#sec-upgrading for how
# to actually do that.
#
# This value being lower than the current NixOS release does NOT mean your system is
# out of date, out of support, or vulnerable.
#
# Do NOT change this value unless you have manually inspected all the changes it would make to your configuration,
# and migrated your data accordingly.
#
# For more information, see `man configuration.nix` or https://nixos.org/manual/nixos/stable/options#opt-system.stateVersion .
system.stateVersion = "24.05"; # Did you read the comment?
}
hardware-configuration.nix
# Do not modify this file! It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations. Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, modulesPath, ... }:
{
imports =
[ (modulesPath + "/installer/scan/not-detected.nix")
];
boot.initrd.availableKernelModules = [ "xhci_pci" "ahci" "nvme" "usb_storage" "usbhid" "sd_mod" ];
boot.initrd.kernelModules = [ ];
boot.kernelModules = [ "kvm-amd" ];
boot.extraModulePackages = [ ];
fileSystems."/" =
{ device = "zroot/root";
fsType = "zfs";
mountPoint = "/";
options = [ "zfsutil" ];
};
fileSystems."/nix" =
{ device = "zroot/nix";
fsType = "zfs";
mountPoint = "/nix";
options = [ "zfsutil" ];
};
fileSystems."/var" =
{ device = "zroot/var";
fsType = "zfs";
mountPoint = "/var";
options = [ "zfsutil" ];
};
fileSystems."/home" =
{ device = "zroot/home";
fsType = "zfs";
mountPoint = "/home";
options = [ "zfsutil" ];
};
fileSystems."/boot" =
{ device = "/dev/disk/by-id/nvme-Samsung_SSD_970_PRO_1TB_S462NF0K607654A";
fsType = "vfat";
mountPoint = "/boot";
options = [ "fmask=0022" "dmask=0022" ];
};
fileSystems."/boot-fallback" =
{ device = "/dev/disk/by-id/nvme-Samsung_SSD_970_PRO_1TB_S462NF0K609877B";
fsType = "vfat";
mountPoint = "/boot-fallback";
options = [ "fmask=0022" "dmask=0022" ];
};
swapDevices = [ ];
# Enables DHCP on each ethernet and wireless interface. In case of scripted networking
# (the default) this is the recommended approach. When using systemd-networkd it's
# still possible to use this option, but it's recommended to use it in conjunction
# with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
networking.useDHCP = lib.mkDefault true;
# networking.interfaces.enp4s0.useDHCP = lib.mkDefault true;
# networking.interfaces.enp6s0.useDHCP = lib.mkDefault true;
# networking.interfaces.enp9s0.useDHCP = lib.mkDefault true;
# networking.interfaces.wlp5s0.useDHCP = lib.mkDefault true;
nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
hardware.cpu.amd.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
}
I just ran into this, also with ZFS on root.
Unsurprisingly, @ElvishJerricco was right – my issue was that my ZFS filesystems were not being recursively mounted (even though canmount and mountpoint were appropriate). After importing the pools and loading my encryption key, I would manually zfs mount my-root-pool at /mnt but I hadn’t noticed that my my-root-pool/nix and my-root-pool/var were not being mounted, so things were being installed “underneath”. When I tried to boot the system, these were being mounted appropriately and therefore my init was not being found.
Easy fix:
zpool import -R /mnt my-root-poolzfs load-key my-root-pool/enczfs mount -a ← this was the missing ingredient- mount other filesystems (boot, EFI)
nixos-install
I don’t see why that would fix anything? Unless you re-generated your config too, but now you have a different problem.
Here’s the weird thing about ZFS and NixOS’s hardware-configuration.nix (more accurately, the fileSystems option). For datasets that have a non-legacy mountpoint property, you’re not supposed to have them in fileSystems. That will cause a race condition in stage 2 (so after initrd, and therefore not related to “stage 2 init script not found”) where zfs-mount.service and systemd-fstab-generator units will race to mount your datasets and things do break. You want to either:
- Use
mountpoint=legacy, which requires manually mounting with plain ole mount -t zfs foo/bar /mnt/bar before generating your fileSystems definitions.
- Let them have regular ZFS-style
mountpoint properties and make sure those datasets aren’t in fileSystems.
But there’s an additional twist for root-on-ZFS folks. For the datasets that need to be mounted in initrd, which include these, you need to make a similar but different choice:
- You can still just use
mountpoint=legacy, let nixos-generate-config put those in fileSystems, and everything’s fine.
- But if you use ZFS-style mountpoints for these, you do have to put them in
fileSystems and you need to add "zfsutil" to their options.
Failing to do one of these things can result in failing to boot, possibly in the way you two described.
1 Like
Great response – yes, that’s almost exactly what I’ve found with a fair amount of tinkering and experimentation.
Because nixos-install was installing all my files to /nix on my rpool/enc/nixos/root dataset, but at boot time my rpool/enc/nixos/root/nix dataset was being mounted overtop and leading to my init not existing in the visible nix store.
I only have minimal mountpoints listed in fileSystems – /, /nix, /var/log, /boot, and /boot/efi. /boot is my only legacy mountpoint; /, /nix, and /var/log all specify zfsutil.
I’ve found that /, /nix, and /var/log (all of which are separate datasets to facilitate different snapshotting rules) all have to be specified in fileSystems or my system doesn’t boot. And I think /var/log requires neededForBoot.
