Strange warnings/errors in rebuilds

Help
1

Hello all, I have had a few strange errors in my nixos configuration for a while, but they don’t seem to hinder the stability of the system or ability to rebuild. I would just like to fix them if possible, however they are a bit strange to look up and I havent found anything too similar.

The initial warnings/errors that would appear were the /usr/bin/env target failing to build due to a Read Only FileSystem, However I thought that was the whole point of nixos?

chmod: changing permissions of '/usr/bin': Read-only file system
ln: failed to create symbolic link '/usr/bin/.env.tmp': Read-only file system
mv: cannot stat '/usr/bin/.env.tmp': No such file or directory
Activation script snippet 'usrbinenv' failed (1)

The second warning/error is the systemd-run command for switching configurations:

Command '['systemd-run', '-E', 'LOCALE_ARCHIVE', '-E', 'NIXOS_INSTALL_BOOTLOADER', '--collect', '--no-ask-password', '--pipe', '--quiet', '--service-type=exec', '--unit=nixos-rebuild-switch-to-configuration', PosixPath('/nix/store/dxz86r7p255fcvm4dxhbgb8w2836y39g-nixos-system-nixos-25.11.20250617.9e83b64/bin/switch-to-configuration'), 'switch']' returned non-zero exit status 2.

Here is the full rebuild command I use and its output:

Summary 
[nathan@nixos:~]$ sudo nixos-rebuild switch --show-trace --flake /etc/nixos#nixos
building the system configuration...
activating the configuration...
setting up /etc...
chmod: changing permissions of '/usr/bin': Read-only file system
ln: failed to create symbolic link '/usr/bin/.env.tmp': Read-only file system
mv: cannot stat '/usr/bin/.env.tmp': No such file or directory
Activation script snippet 'usrbinenv' failed (1)
Failed to run activate script
reloading user units for nathan...
restarting sysinit-reactivation.target
the following new units were started: libvirtd.service, NetworkManager-dispatcher.service
Command '['systemd-run', '-E', 'LOCALE_ARCHIVE', '-E', 'NIXOS_INSTALL_BOOTLOADER', '--collect', '--no-ask-password', '--pipe', '--quiet', '--service-type=exec', '--unit=nixos-rebuild-switch-to-configuration', PosixPath('/nix/store/dxz86r7p255fcvm4dxhbgb8w2836y39g-nixos-system-nixos-25.11.20250617.9e83b64/bin/switch-to-configuration'), 'switch']' returned non-zero exit status 2.

[nathan@nixos:~]$

I am on nixos unstable at the moment too, but its persisted between channels from before I switched to unsable

2

Sounds like you mounted something read-only on /usr. Can we see your mounts?

3

Here is df -h and my mtab, don’t mind the mount names lmao

Summary 
[nathan@nixos:~]$ df -h
Filesystem                                                  Size  Used Avail Use% Mounted on
devtmpfs                                                    1.6G     0  1.6G   0% /dev
tmpfs                                                        16G   30M   16G   1% /dev/shm
tmpfs                                                       7.7G   11M  7.7G   1% /run
/dev/nvme1n1p2                                              907G  567G  295G  66% /
efivarfs                                                    128K   47K   77K  39% /sys/firmware/efi/efivars
tmpfs                                                       1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
tmpfs                                                        16G  1.8M   16G   1% /run/wrappers
/dev/nvme1n1p1                                              511M  319M  193M  63% /boot
tmpfs                                                       1.0M     0  1.0M   0% /run/credentials/getty@tty1.service
tmpfs                                                       3.1G  3.7M  3.1G   1% /run/user/1000
nixstor2@[redacted].NixStorage=/                            668G     0  668G   0% /home/nathan/mnt/NixStorage
10.69.1.26:/Temp-Pool                                       526G  213G  314G  41% /home/nathan/mnt/slimjim
10.69.1.100:/mnt/BiggusDickus/core/SoftwareAndMisc           11T  1.7T  8.7T  16% /home/nathan/mnt/nasMisc
10.69.1.100:/mnt/BiggusDickus/core/Media                    8.7T  1.0M  8.7T   1% /home/nathan/mnt/Media
10.69.1.100:/mnt/BiggusDickus/backups/Store2-Backup          11T  1.5T  8.7T  15% /home/nathan/mnt/Store2
10.69.1.100:/mnt/BiggusDickus/backups/Store4-Backup         9.5T  804G  8.7T   9% /home/nathan/mnt/Store4

[nathan@nixos:~]$ cat /etc/mtab
devtmpfs /dev devtmpfs rw,nosuid,size=1599628k,nr_inodes=3992713,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=3,mode=620,ptmxmode=666 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,size=15996248k 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /run tmpfs rw,nosuid,nodev,size=7998124k,mode=755 0 0
ramfs /run/keys ramfs rw,nosuid,nodev,relatime,mode=750 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/disk/by-uuid/2ee11375-6999-431d-a160-32bd213dbc83 / ext4 rw,relatime 0 0
/dev/disk/by-uuid/2ee11375-6999-431d-a160-32bd213dbc83 /nix/store ext4 ro,relatime 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
none /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,nosuid,nodev,relatime,pagesize=2M 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /run/credentials/systemd-journald.service tmpfs ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,noswap 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /run/wrappers tmpfs rw,nodev,relatime,size=15996248k,mode=755 0 0
/dev/nvme1n1p1 /boot vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
binder /dev/binderfs binder rw,relatime,max=1048576 0 0
tmpfs /run/credentials/getty@tty1.service tmpfs ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,noswap 0 0
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3199248k,nr_inodes=799812,mode=700,uid=1000,gid=100 0 0
gvfsd-fuse /run/user/1000/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=100 0 0
nixstor2@[redacted].NixStorage=/ /home/nathan/mnt/NixStorage ceph rw,relatime,name=nixstor2,secret=<hidden>,acl,mon_addr=10.69.1.23:6789/10.69.1.24:6789 0 0
10.69.1.26:/Temp-Pool /home/nathan/mnt/slimjim nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.69.1.90,local_lock=none,addr=10.69.1.26 0 0
portal /run/user/1000/doc fuse.portal rw,nosuid,nodev,relatime,user_id=1000,group_id=100 0 0
10.69.1.100:/mnt/BiggusDickus/core/SoftwareAndMisc /home/nathan/mnt/nasMisc nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.69.1.100,mountvers=3,mountport=43540,mountproto=udp,local_lock=none,addr=10.69.1.100 0 0
10.69.1.100:/mnt/BiggusDickus/core/Media /home/nathan/mnt/Media nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.69.1.100,mountvers=3,mountport=43540,mountproto=udp,local_lock=none,addr=10.69.1.100 0 0
10.69.1.100:/mnt/BiggusDickus/backups/Store2-Backup /home/nathan/mnt/Store2 nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.69.1.100,mountvers=3,mountport=43540,mountproto=udp,local_lock=none,addr=10.69.1.100 0 0
10.69.1.100:/mnt/BiggusDickus/backups/Store4-Backup /home/nathan/mnt/Store4 nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.69.1.100,mountvers=3,mountport=43540,mountproto=udp,local_lock=none,addr=10.69.1.100 0 0
overlay /var/lib/docker/overlay2/3c207a821379d3ce7c7969f056463c9ad4bee7209fe538766cc640da2ad247bb/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/P6JZW4MJBBUKUZG6567F2PU5CI:/var/lib/docker/overlay2/l/P7UC7NR7EYGXXHHWUYGB72VZPW:/var/lib/docker/overlay2/l/SLNLXYTB76N5EV5KAVHQKU37KX:/var/lib/docker/overlay2/l/GGZARGLDYZFQTPQYKMUXMPZ5DM:/var/lib/docker/overlay2/l/33KW6P7Q2FFLCDJF2VN534BXBH:/var/lib/docker/overlay2/l/6J4FZFSHAGDGLZULWZW27XYIMR:/var/lib/docker/overlay2/l/3EPEZUSOTY253UWAO2OPOFZG4K:/var/lib/docker/overlay2/l/JUZUMACZIPEM4WMIXZDPMEFLMV:/var/lib/docker/overlay2/l/KYMZVR2DN3QN5RCNRMSHW5LQRK:/var/lib/docker/overlay2/l/DEBFFJLSULHKXARVWPMSR6E6Y4:/var/lib/docker/overlay2/l/KOKTJQU3KRDKPR6AD6C2MHPCPF:/var/lib/docker/overlay2/l/YEL5ZXYUZGQT4CBTTYY4I2F7UO:/var/lib/docker/overlay2/l/4UHZKSJKIYC4YVIO7GOV275STN:/var/lib/docker/overlay2/l/J4WVKF5YXNLWW74LPIMKCMYXRC,upperdir=/var/lib/docker/overlay2/3c207a821379d3ce7c7969f056463c9ad4bee7209fe538766cc640da2ad247bb/diff,workdir=/var/lib/docker/overlay2/3c207a821379d3ce7c7969f056463c9ad4bee7209fe538766cc640da2ad247bb/work 0 0
nsfs /run/docker/netns/23a8b197254c nsfs rw 0 0

[nathan@nixos:~]$ cat /etc/mtab | grep /usr

[nathan@nixos:~]$
4

Not a mount then. A symlink? Is /usr a symlink into the nix store for some reason?

5

aha!, its symlinked into /run/opengl-driver/ for some reason

[nathan@nixos:~]$ ls -lah /
total 2.4M
drwxr-xr-x  21 root root 4.0K Apr 26 13:54 .
drwxr-xr-x  21 root root 4.0K Apr 26 13:54 ..
drwxr-xr-x   3 root root 4.0K Apr 26 13:54 absolute
drwxr-xr-x   2 root root 4.0K Jun 30 20:34 bin
drwxr-xr-x   6 root root 4.0K Dec 31  1969 boot
drwxr-xr-x  23 root root 4.4K Jun 30 20:34 dev
drwxr-xr-x  45 root root 4.0K Jun 30 20:34 etc
drwxr-xr-x   3 root root 4.0K Jan  3  2024 home
drwxr-xr-x   2 root root 4.0K May  6 13:07 lib
drwxr-xr-x   2 root root 4.0K May 21 11:00 lib64
drwx------   2 root root  16K Jan  3  2024 lost+found
drwxr-xr-x   4 root root 4.0K Feb 27  2024 mnt
drwxr-xr-x   4 root root 4.0K Jan  3  2024 nix
drwx--x--x   4 root root 4.0K Jan 19  2024 opt
dr-xr-xr-x 521 root root    0 Jun 30 20:34 proc
drwx------  11 root root 4.0K May 18 23:46 root
drwxr-xr-x  34 root root  960 Jun 30 20:54 run
drwxr-xr-x   2 root root 4.0K Jan  3  2024 srv
dr-xr-xr-x  13 root root    0 Jun 30 20:34 sys
drwxrwxrwt  27 root root 2.4M Jun 30 21:23 tmp
lrwxrwxrwx   1 root root   19 Feb  9  2024 usr -> /run/opengl-driver/
drwxr-xr-x   9 root root 4.0K Jan  3  2024 var

[nathan@nixos:~]$
6

That sure is a weird one… no idea how it got that way, but deleting /usr and running a rebuild (or just sudo /run/current-system/bin/switch-to-configuration switch) should fix it, unless it’s somehow part of your declarative config.

7

I just removed it and rebuilt, success on fixing that!
I have a vague idea of how it happened, Might’ve been a mistake in my configuration when I was doing a systemd.tempfiles to fix an issue with a piece of software at the time, but it is no longer in my configuration. phew!

Now I’m not sure exactly how to trigger the systemd-run error as it is somewhat transient, but I am attempting a rebuild with a new piece of software in hopes it will show up. If not It might be only during system changes.

8

Yeah, an errant tempfiles rule could have caused it initially. Perhaps the usrbinenv snippet should handle the case where /usr is a symlink…

9

That would be nice. It did drive me up a wall at the beginning, but after a while nothing broke from it so I just kinda ignored it until the systemd-run error started popping up.

10

The systemd-run thing should just be an effect of the usrbinenv problem, I believe. The reason it only started showing up later is because we changed the default nixos-rebuild implementation to the new one written in python, and it reports errors somewhat differently.

11

ooooo ok that would make sense! (and as to why I cannot replicate it anymore); Though I wish the rebuild impl would be a bit more verbose when non-nix errors happen, but that might be a bit more difficult to do as the python traceback (which I got once and completely forgot about until now) wasn’t super helpful. Just a trace with all the calls seemingly referencing the exact failure I was experiencing. Though maybe if systemd-run had any output besides an exit code that would be of use.

12

Filed an issue: