So for full context: I’m trying to experiment with a new system using a multi-disk bcachefs setup (not for root; it’s mounted as /home). However, I’m having trouble getting it to mount on boot. I’m using device = "UUID:<uuid>" in the config generated by nixos-generate-config, but after a reboot, the only mount option that works for me is using /dev/sda1:/dev/sdb1:... (which isn’t great because the letters change on reboot). Looking into the issue, I discovered this explanation, which indicates that util-linux 2.39.2 is bugged, and that util-linux 2.39.3 includes a fix for this issue.
Using the package search on the website, I see that the 23.11 channel has 2.39.2, while unstable has 2.39.3. I’d rather not move the entire system to the unstable channel if I can avoid it, so I’m trying to upgrade just util-linux to the version with the bug fix.
I added let unstable = import <nixos-unstable> { }; to my config based on this thread response, however, I can’t figure out how to get the system to use unstable.util-linux. From what I can tell, boot.kernelPackages = pkgs.linuxPackages_latest; is taking precedence over any additions to environment.systemPackages. However, when I try to find information about overriding the packages in pkgs.linuxPackages_latest, all I’ve found is information on overriding the kernel itself.
Anybody have any advice on how to do this? I’ll also take feedback with alternative ways to work around this issue. I’m new to NixOS, and I feel like I’m over my head with this.
Changing util-linux system-wide is very non-trivial, because too many things depend on it. You can do it in an overlay, but you will end up re-compiling half of nixpkgs. Instead of putting it in an overlay, you can just put it in systemPackages, but then the important things like systemd won’t use it.
nixos-23.11 shipped the patch for 2.39.2 since the reddit post you linked. So you’ve got the fix already.
The problems with multi-device bcachefs go much further than this, and that’s most likely why you’re having trouble mounting it on boot. This is just a limitation in systemd currently . There’s also a nixpkgs issue about it. TL;DR: If you use UUID=... in fstab, systemd wants to wait for only one device and then pass that device alone to the mount CLI, which doesn’t work. If you use /dev/foo:/dev/bar, systemd wants to wait for a single device whose name is /dev/foo:/dev/bar rather than two separate devices. I’m looking into fixes for this problem myself.
Actually, you don’t have to use the lettered names. You can use like /dev/disk/by-id/foo which doesn’t change on reboot. You can’t use /dev/disk/by-uuid though because all the devices share a UUID so only one gets the link in there.
In the meantime, you probably just have to make a custom service that runs the mount command itself. Here’s some more discussion on here about this: How do I mount multiple bcachefs devices on boot?
Changing util-linux system-wide is very non-trivial
Noted. I was thinking it probably wasn’t going to be painless, considering how little I’ve found on it. I’ll try to go with a different solution then.
nixos-23.11 shipped the patch for 2.39.2 since the reddit post you linked. So you’ve got the fix already.
I noticed this right before you left a comment. I was looking through the nixpkgs definitions and saw the patches on the releases-23.11 branch. I’m not sure why it’s not working then.
Well my system disagrees, as mount --version gives mount from util-linux 2.39.2. Also, the 23.11 version still says 2.39.2. I’m guessing that commit was for unstable, as I see that version there on search.nixos.org.
You can use like /dev/disk/by-id/foo which doesn’t change on reboot.
I tried using /dev/disk/by-partuuid/ (since I put them in a GPT header), but couldn’t get it to work for some reason. I can try again, as well as using by-id instead.
In the meantime, you probably just have to make a custom service that runs the mount command itself.
I’ll look into it, then. I would like to know why using the UUID= specifier isn’t working, but I’ll experiment with this to see if I can at least get it working.
Yes, I made an error in my repo browsing and thought I was looking at the nixos-23.11 branch when I wasn’t. Sorry about that.
I already said why:
systemd doesn’t pass the argument in the form UUID=.... to the mount CLI. It decodes it to /dev/disk/by-uuid/.... and passes that as the argument. But that’s only one device. A random member of the bcachefs file system will claim that link, and the bcachefs mount CLI does not allow you to provide only one of the necessary devices; you have to provide them all, or the literal syntax UUID=..., which systemd is not doing.
The exact /dev/sdX device it mentions changes. This happens whether or not I have it in my hardware config (and therefore fstab). I’m guessing it’s still an issue of how the UUID is getting associated with the disks, which might still be in systemd, but I don’t know for sure.
Ahh I didn’t realize you had a problem mounting with a manual mount command.
This is strange. If I strace -f the mount command, it also is decoding UUID=... to a single file path in /dev before passing it to mount.bcachefs. I did not expect that, and that’s definitely why it’s failing. You can use mount.bcachefs directly instead (it’ll spit out “errors” because it looks for bcachefs superblocks on all block devices, regardless of if it makes sense, but that’s harmless).
I was wondering why others have recommended that you can use mount -t bcachefs UUID=..., and I think I figured out why. They’ve been using it in initrd, which by default uses busybox for the mount command, which doesn’t do the UUID= -> /dev decoding that util-linux seems to be doing.
Just for future reference (in case anybody happens to run across this thread with the same problem), this actually provides for a very simple (but terrible) workaround. You can just force the partition to be mounted in initrd with fileSystems."/home".neededForBoot = true; (with "/home" being whatever the mounting point is).