Nix-shell "error opening lock file ... Read-only filesystem"

On one of my NixOS machines, I stopped being able to run nix-shell without root for some reason. I’ve tried a few things I’ve found googling and searching on the forums here, but nothing has worked so far. The other NixOS machine I don’t have this issue.

Issue

If I try to use nix-shell -p somePackage and somePackage isn’t in the store yet, it fails with "error opening lock file … Read-only filesystem.

If I then go sudo nix-shell -p somePackage it runs fine.

Then after having made a shell with sudo, I’m able then able to open nix-shells with somePackage without needing root.

Example

cat@beppo ~> nix-shell -p xcaddy                                                                                                         
error:                                                                                                                                   
       … while calling the 'derivationStrict' builtin                                                                                    
                                                                                                                                         
         at /builtin/derivation.nix:9:12: (source not available)                                                                         
                                                                                                                                         
       … while evaluating derivation 'shell'                                                                                             
         whose name attribute is located at /nix/store/27gmaqdprq2g8xrgk9jkp691qykb9c4s-nixos-24.05/nixos/pkgs/stdenv/generic/make-deriva
tion.nix:331:7                                                                                                                           
                                                                                                                                         
       … while evaluating attribute 'buildInputs' of derivation 'shell'                                                                  
                                                                                                                                         
         at /nix/store/27gmaqdprq2g8xrgk9jkp691qykb9c4s-nixos-24.05/nixos/pkgs/stdenv/generic/make-derivation.nix:378:7:                 
                                                                                                                                         
          377|       depsHostHost                = elemAt (elemAt dependencies 1) 0;                                                     
          378|       buildInputs                 = elemAt (elemAt dependencies 1) 1;                                                     
             |       ^                                                                                                                   
          379|       depsTargetTarget            = elemAt (elemAt dependencies 2) 0;                                                     
                                                                                                                                         
       (stack trace truncated; use '--show-trace' to show the full trace)                                                                
                                                                                                                                         
       error: opening lock file '/nix/store/dpshf2nsmygnd54df7ncq47c7rmrbjf7-inject_version_info.diff.lock': Read-only file system       

Then with root:

cat@beppo ~> sudo nix-shell -p xcaddy
this path will be fetched (1.11 MiB download, 3.41 MiB unpacked):
  /nix/store/8grfa8lbq1lf0wxnhaxgfqmmcddhh6jy-xcaddy-0.4.2
copying path '/nix/store/8grfa8lbq1lf0wxnhaxgfqmmcddhh6jy-xcaddy-0.4.2' from 'https://cache.nixos.org'...

[nix-shell:/home/cat]# which xcaddy
/nix/store/8grfa8lbq1lf0wxnhaxgfqmmcddhh6jy-xcaddy-0.4.2/bin/xcaddy 

Now again without root, but successfully:

cat@beppo ~> nix-shell -p xcaddy

[nix-shell:~]$ which xcaddy
/nix/store/8grfa8lbq1lf0wxnhaxgfqmmcddhh6jy-xcaddy-0.4.2/bin/xcaddy

Additional context

system info:

cat@beppo ~> nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.9.3, NixOS, 24.05 (Uakari), 24.05.984.0b8e7a1ae5a9`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.2`
 - channels(root): `"nixos-24.05"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

mount info:

cat@beppo ~> sudo findmnt -A -R &| tee mounts.txt
TARGET                          SOURCE                                                                                            FSTYPE     OPTIONS
/                               /dev/sdc:/dev/sda:/dev/sdb:/dev/sdd:/dev/sde:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme0n1p2             bcachefs   rw,noatime,metadata_replicas=2,data_replicas=2,compression=lz4,foreground_target=ssd,background_target=hdd,promote_target=ssd,nojournal_transaction_names
├─/dev                          devtmpfs                                                                                          devtmpfs   rw,nosuid,size=6587852k,nr_inodes=16463976,mode=755
│ ├─/dev/pts                    devpts                                                                                            devpts     rw,nosuid,noexec,relatime,gid=3,mode=620,ptmxmode=666
│ ├─/dev/shm                    tmpfs                                                                                             tmpfs      rw,nosuid,nodev
│ ├─/dev/mqueue                 mqueue                                                                                            mqueue     rw,nosuid,nodev,noexec,relatime
│ └─/dev/hugepages              hugetlbfs                                                                                         hugetlbfs  rw,nosuid,nodev,relatime,pagesize=2M
├─/proc                         proc                                                                                              proc       rw,nosuid,nodev,noexec,relatime
├─/run                          tmpfs                                                                                             tmpfs      rw,nosuid,nodev,size=32939260k,mode=755
│ ├─/run/keys                   ramfs                                                                                             ramfs      rw,nosuid,nodev,relatime,mode=750
│ ├─/run/wrappers               tmpfs                                                                                             tmpfs      rw,nodev,relatime,mode=755
│ └─/run/user/1000              tmpfs                                                                                             tmpfs      rw,nosuid,nodev,relatime,size=13175704k,nr_inodes=3293926,mode=700,uid=1000,gid=999
├─/sys                          sysfs                                                                                             sysfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security        securityfs                                                                                        securityfs rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup              cgroup2                                                                                           cgroup2    rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
│ ├─/sys/fs/pstore              pstore                                                                                            pstore     rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware/efi/efivars   efivarfs                                                                                          efivarfs   rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf                 bpf                                                                                               bpf        rw,nosuid,nodev,noexec,relatime,mode=700
│ ├─/sys/kernel/debug           debugfs                                                                                           debugfs    rw,nosuid,nodev,noexec,relatime
│ │ └─/sys/kernel/debug/tracing tracefs                                                                                           tracefs    rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/fuse/connections    fusectl                                                                                           fusectl    rw,nosuid,nodev,noexec,relatime
│ └─/sys/kernel/config          configfs                                                                                          configfs   rw,nosuid,nodev,noexec,relatime
├─/nix/store                    /dev/sdc:/dev/sda:/dev/sdb:/dev/sdd:/dev/sde:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme0n1p2[/nix/store] bcachefs   ro,noatime,metadata_replicas=2,data_replicas=2,compression=lz4,foreground_target=ssd,background_target=hdd,promote_target=ssd,nojournal_transaction_names
└─/boot                         /dev/nvme0n1p1                                                                                    vfat       rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro

for comparison, here’s the system info and mounts for my machine without the issue:

cat@bippo ~> nix-shell -p xcaddy
 - system: `"x86_64-linux"`
 - host os: `Linux 6.9.3, NixOS, 24.05 (Uakari), 24.05.984.0b8e7a1ae5a9`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.2`
 - channels(root): `"nixos-24.05"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
TARGET                          SOURCE                                                    FSTYPE          OPTIONS
/                               /dev/sda:/dev/sdb:/dev/nvme0n1p2:/dev/nvme1n1             bcachefs        rw,noatime,compression=lz4,foreground_target=ssd,background_target=hdd,promote_target=ssd
├─/dev                          devtmpfs                                                  devtmpfs        rw,nosuid,size=1622720k,nr_inodes=4053346,mode=755
│ ├─/dev/pts                    devpts                                                    devpts          rw,nosuid,noexec,relatime,gid=3,mode=620,ptmxmode=666
│ ├─/dev/shm                    tmpfs                                                     tmpfs           rw,nosuid,nodev,size=16227200k
│ ├─/dev/mqueue                 mqueue                                                    mqueue          rw,nosuid,nodev,noexec,relatime
│ └─/dev/hugepages              hugetlbfs                                                 hugetlbfs       rw,nosuid,nodev,relatime,pagesize=2M
├─/proc                         proc                                                      proc            rw,nosuid,nodev,noexec,relatime
├─/run                          tmpfs                                                     tmpfs           rw,nosuid,nodev,size=8113600k,mode=755
│ ├─/run/keys                   ramfs                                                     ramfs           rw,nosuid,nodev,relatime,mode=750
│ ├─/run/wrappers               tmpfs                                                     tmpfs           rw,nodev,relatime,size=16227200k,mode=755
│ └─/run/user/1000              tmpfs                                                     tmpfs           rw,nosuid,nodev,relatime,size=3245436k,nr_inodes=811359,mode=700,uid=1000,gid=998
│   ├─/run/user/1000/gvfs       gvfsd-fuse                                                fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=998
│   └─/run/user/1000/doc        portal                                                    fuse.portal     rw,nosuid,nodev,relatime,user_id=1000,group_id=998
├─/sys                          sysfs                                                     sysfs           rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security        securityfs                                                securityfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup              cgroup2                                                   cgroup2         rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
│ ├─/sys/fs/pstore              pstore                                                    pstore          rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware/efi/efivars   efivarfs                                                  efivarfs        rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf                 bpf                                                       bpf             rw,nosuid,nodev,noexec,relatime,mode=700
│ ├─/sys/kernel/debug           debugfs                                                   debugfs         rw,nosuid,nodev,noexec,relatime
│ │ └─/sys/kernel/debug/tracing tracefs                                                   tracefs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/fuse/connections    fusectl                                                   fusectl         rw,nosuid,nodev,noexec,relatime
│ └─/sys/kernel/config          configfs                                                  configfs        rw,nosuid,nodev,noexec,relatime
├─/nix/store                    /dev/sda:/dev/sdb:/dev/nvme0n1p2:/dev/nvme1n1[/nix/store] bcachefs        ro,noatime,compression=lz4,foreground_target=ssd,background_target=hdd,promote_target=ssd
└─/boot                         /dev/nvme0n1p1                                            vfat            rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro

Thank you! Please let me know if you have additional questions I can answer or ideas of things I can try!

Is your nix-daemon still active? I wouldn’t be surprised if, with a dead nix-daemon, the unprivileged nix processes try to manage the store themselves, which they wouldn’t be able to do.

Note that the daemon doesn’t necessarily have to be running if the systemd socket is active so the service can be socket activated. Try systemctl status 'nix-daemon.*'

It did look like the nix daemon wasn’t running! I manually started it with sudo systemctl start nix-daemon and now it started running successfully:

sudo systemctl status nix-daemon
● nix-daemon.service - Nix Daemon
     Loaded: loaded (/etc/systemd/system/nix-daemon.service; linked; preset: enabled)
    Drop-In: /nix/store/5j5zlrdqy94q391isb1r00yhdrfadj08-system-units/nix-daemon.service.d
             └─overrides.conf
     Active: active (running) since Sat 2024-06-08 18:55:28 PDT; 1min 14s ago
TriggeredBy: ● nix-daemon.socket
       Docs: man:nix-daemon
             https://nixos.org/manual
   Main PID: 49451 (nix-daemon)
         IP: 0B in, 0B out
         IO: 0B read, 0B written
      Tasks: 2 (limit: 1048576)
     Memory: 3.6M (peak: 4.0M)
        CPU: 16ms
     CGroup: /system.slice/nix-daemon.service
             └─49451 nix-daemon --daemon

Jun 08 18:55:28 beppo systemd[1]: Started Nix Daemon.

Even with the daemon running though, the nix-shell issue still persists:

cat@beppo ~/caddy> nix-shell -p toybox
error:
       … while calling the 'derivationStrict' builtin

         at /builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'shell'
         whose name attribute is located at /nix/store/ypdqfzw041rprrlzckic9lmhp9navbiz-nixos-24.05/nixos/pkgs/stdenv/generic/make-derivation.nix:331:7

       … while evaluating attribute 'buildInputs' of derivation 'shell'

         at /nix/store/ypdqfzw041rprrlzckic9lmhp9navbiz-nixos-24.05/nixos/pkgs/stdenv/generic/make-derivation.nix:378:7:

          377|       depsHostHost                = elemAt (elemAt dependencies 1) 0;
          378|       buildInputs                 = elemAt (elemAt dependencies 1) 1;
             |       ^
          379|       depsTargetTarget            = elemAt (elemAt dependencies 2) 0;

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: opening lock file '/nix/store/n36xck7wx1g0yqysnn96qgqr7kiwbviw-source.drv.lock': Read-only file system

Checking journalctl for nix-daemon, it only has the single line Jun 08 18:55:28 beppo systemd[1]: Started Nix Daemon.

The socket looks fine for it:

cat@beppo ~/caddy [1]> sudo systemctl status nix-daemon.socket
● nix-daemon.socket - Nix Daemon Socket
     Loaded: loaded (/etc/systemd/system/nix-daemon.socket; enabled; preset: enabled)
    Drop-In: /nix/store/5j5zlrdqy94q391isb1r00yhdrfadj08-system-units/nix-daemon.socket.d
             └─overrides.conf
     Active: active (running) since Fri 2024-06-07 12:23:15 PDT; 1 day 6h ago
   Triggers: ● nix-daemon.service
     Listen: /nix/var/nix/daemon-socket/socket (Stream)
     CGroup: /system.slice/nix-daemon.socket

Jun 07 12:23:15 beppo systemd[1]: Listening on Nix Daemon Socket.

stating the socket itself shows it’s owned by root and has 0666 permissions:

cat@beppo ~/caddy> stat /nix/var/nix/daemon-socket/socket
  File: /nix/var/nix/daemon-socket/socket
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 8,32    Inode: 1209091714  Links: 1
Access: (0666/srw-rw-rw-)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-06-07 12:23:15.365999692 -0700
Modify: 2024-06-07 12:23:15.365999692 -0700
Change: 2024-06-07 12:23:15.365999692 -0700
 Birth: 2024-06-07 12:23:15.365999692 -0700

I noticed looking back at the nix-info from the first post, the broken machine isn’t running in multi-user mode. Looking at the documentation for multi-user mode, it suggests that group nix-users be allowed to the nix-daemon socket. I ran the chgrp and chmod commands suggested at the bottom of the documentation (substituting nix-users for users which exists on my machine), restarted nix-daemon, and tried a nix-shell again, but to no avail.

Forgot to update this. Setting NIX_REMOTE = "daemon"; in my environment.variables in /etc/nixos/configuration.nix seemed to clear it up.