Nix-daemon doesn't start on boot on Ubuntu with multi-user installation and a bind-mounted /nix

1

Hi all,

I installed nix on a server running Ubuntu 20.04, with a separate /nix bind-mount. But nix-daemon doesn’t start on boot and I have to do the following manually on each reboot:

sudo systemctl daemon-reload
sudo systemctl start nix-daemon

The relavent bits in /etc/fstab is the following:

/dev/nvme0n1p5  /mnt/ssd        ext4     defaults                             0       2

# bind mount /nix
/mnt/ssd/nix    /nix            none     bind,x-systemd.requires=/mnt/ssd     0       0

I took a look at journalctl -b, but there’s no mentioning of nix-daemon at all, not even an error message.

One interesting thing is that I have to do systemctl daemon-reload on each reboot, otherwise systemd won’t recognize nix-daemon:

Unit nix-daemon.service could not be found.

Does this mean that the /nix partition was not visible to systemd during the early initialization s.t. the nix-daemon unit doesn’t get loaded at all?

If so, should I try to mount /mnt/data and nix partitions in initramfs? How?

2

I don’t know much about what nix installation does on non-nixos distros. But my best guess would be that some .service file is a symlink to some /nix/store path, and when systemd initializes during boot, this path doesn’t exist yet and therefore the symlink is dead and the service cannot be loaded.

You probably need to replace the service with another one that doesn’t reside in /nix/store. But then the question is if that will break during the next update or not.

3 Likes
3

Thanks for the suggestion. The nix-daemon.service and nix-daemon.sockets are indeed symlinked to /nix/store/<bla> and they probably are not available when systemd initializes.

But NixOS must be able to handle that somehow, given that /etc/systemd/system is a link to /etc/static/systemd/system which then links to /nix/store. How does NixOS handle this w/o problem?

4

It mounts /nix/store in initramfs, so it’s available before systemd even starts.

1 Like
5

I see. So I’ll just do the same for Ubuntu then.

6

@yipengsun It would be interesting to know if you have come to a solution for this.

7

Sorry, I didn’t pursue this further.

8

I encountered this error trying to have a separate /nix partition on a Non-NixOS system. I didn’t find a nice way to fix it, but as a workaround I created a service that gets it started. This is the only mention of the issue I found, so I’m posting here in case anyone else finds it.

/etc/systemd/system/nix-kludge.service:

[Unit]
Description=Fix nix-daemon for separate /nix partition
RequiresMountsFor=/nix

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl daemon-reload
ExecStart=/usr/bin/systemctl start nix-daemon.socket

[Install]
WantedBy=multi-user.target
9

@samh That shouldn’t be necessary. The nix-daemon.socket unit should already have the equivalent of RequiresMountsFor=/nix/var/nix/daemon-socket/socket thanks to the default dependencies of socket units, and the nix-daemon.service unit that nix ships with already has:

RequiresMountsFor=/nix/store
RequiresMountsFor=/nix/var
RequiresMountsFor=/nix/var/nix/db

So you shouldn’t have needed anything extra. Can you identify why what you did actually fixed anything? Because it shouldn’t have been necessary.

10

@ElvishJerricco You are right that it has something like that. As far as I can tell, the problem is that the socket and service files are symbolic links:

[samh@centos9-stream ~]$ ll /etc/systemd/system/nix-daemon.*
lrwxrwxrwx. 1 root root 67 Nov 12 10:52 /etc/systemd/system/nix-daemon.service -> /nix/var/nix/profiles/default/lib/systemd/system/nix-daemon.service
lrwxrwxrwx. 1 root root 66 Nov 12 10:52 /etc/systemd/system/nix-daemon.socket -> /nix/var/nix/profiles/default/lib/systemd/system/nix-daemon.socket

When the system is starting up, I see this in the journal, before systemd mounts my /nix filesystem:

Nov 12 12:00:35 centos9-stream systemd[1]: nix-daemon.socket: Failed to open /etc/systemd/system/nix-daemon.socket: No such file or directory

It says “No such file or directory” because nix-daemon.socket is a link to a filesystem that is not yet mounted.

If I ask for the status of the socket before systemctl daemon-reload, it says:

[samh@centos9-stream ~]$ systemctl status nix-daemon.socket 
Unit nix-daemon.socket could not be found.
11

Ah, right, I had forgotten the context of this thread :stuck_out_tongue: Indeed that would be an issue. You could work around that by mounting /nix in your initramfs. How to do that depends on your distro, but on distros with systemd-in-initrd (e.g. fedora) you should be able to just add x-initrd.mount to the entry in /etc/fstab in your root fs, and the initrd will pick it up after mounting the root fs and before switching to the main system.

2 Likes
12

@ElvishJerricco thank you, that was exactly what I was trying to figure out! (my searches didn’t find any useful information on how to do it)

Adding x-initrd.mount to the /nix mount in /etc/fstab looks like it solved it for me (on CentOS 9, derived from Fedora):

LABEL=centos9-data /nix btrfs noatime,subvol=@nix,compress=zstd:9,x-initrd.mount 0 0

After some more searching I had found this blog post: Metamagical
It describes another way to fix it on Fedora (and derivatives), involving an /etc/fstab.sys file. I would have been satisfied with that, but the x-initrd.mount solution is simpler (and better documented compared to dracut and /etc/fstab.sys).