How does root filesystem work?

Hi all!

I am new to the nix community. I am using NixOS, and am evangelizing nix in the tech department.

My first question is: I have noticed something curious when using NixOS:

I can see that my root filesystem (unimportantly, it is /dev/mapper/root, since it is LUKS-encrypted) is mounted twice one to / one to /nix/store, but they look absolutely different:

# mount | grep root
/dev/mapper/root on / type ext4 (rw,relatime)
/dev/mapper/root on /nix/store type ext4 (rw,relatime)
$ ls /
bin  boot  data  dev  etc  home  lost+found  nix  proc  root  run  srv  sys  tmp  usr  var
$ ls /nix/store/ | head -n3
0006yk8jxi0nmbz09fq86zl037c1wx9b-automake-1.16.5.tar.xz.drv
001gp43bjqzx60cg345n2slzg7131za8-nix-nss-open-files.patch
001vnhxaswqq5fv88p61m0rypp88knmr-unagi-chan-0.4.1.4.tar.gz.drv

I am super intrigued to learn what kind of sorcery is happening here.

/nix/store is remounted as a read only bind mount usually. Not sure why it is RW for you.

Hmm, okay, it is almost a fresh installation of NixOS, I don’t think I changed much configuration yet.

I added a bit more context in the post. You can see that β€œ/” and β€œ/nix/store” look very different, unsurprisingly, but they are mounted to the same device, which I am very puzzled.

I think it’s because you can mount a directory of a filesystem on Linux, but it seems mount doesn’t show this. Maybe a flag is required.

[root@hellwolf-xps15-nixos:~]# findmnt 
TARGET                        SOURCE           FSTYPE          OPTIONS
/                             /dev/disk/by-uuid/784abb2f-b58f-49ef-bc07-38bf8843498b
β”‚                                              ext4            rw,relatime
β”œβ”€/dev                        devtmpfs         devtmpfs        rw,nosuid,size=1631144k,nr_inodes=4075159,mode=755
β”‚ β”œβ”€/dev/pts                  devpts           devpts          rw,nosuid,noexec,relatime,gid=3,mode=620,ptmxmode=666
β”‚ β”œβ”€/dev/shm                  tmpfs            tmpfs           rw,nosuid,nodev
β”‚ β”œβ”€/dev/hugepages            hugetlbfs        hugetlbfs       rw,relatime,pagesize=2M
β”‚ └─/dev/mqueue               mqueue           mqueue          rw,nosuid,nodev,noexec,relatime
β”œβ”€/proc                       proc             proc            rw,nosuid,nodev,noexec,relatime
β”œβ”€/run                        tmpfs            tmpfs           rw,nosuid,nodev,size=8155716k,mode=755
β”‚ β”œβ”€/run/keys                 none             ramfs           rw,nosuid,nodev,relatime,mode=750
β”‚ β”œβ”€/run/wrappers             tmpfs            tmpfs           rw,nodev,relatime,mode=755
β”‚ └─/run/user/1000            tmpfs            tmpfs           rw,nosuid,nodev,relatime,size=3262284k,nr_inodes=815571,mode=700,uid=1000,gid=1000
β”‚   β”œβ”€/run/user/1000/gvfs     gvfsd-fuse       fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
β”‚   └─/run/user/1000/doc      portal           fuse.portal     rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
β”œβ”€/sys                        sysfs            sysfs           rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/sys/kernel/security      securityfs       securityfs      rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/sys/fs/cgroup            cgroup2          cgroup2         rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
β”‚ β”œβ”€/sys/firmware/efi/efivars efivarfs         efivarfs        rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/sys/fs/bpf               bpf              bpf             rw,nosuid,nodev,noexec,relatime,mode=700
β”‚ β”œβ”€/sys/kernel/debug         debugfs          debugfs         rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/sys/kernel/config        configfs         configfs        rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/sys/fs/fuse/connections  fusectl          fusectl         rw,nosuid,nodev,noexec,relatime
β”‚ └─/sys/fs/pstore            pstore           pstore          rw,nosuid,nodev,noexec,relatime
β”œβ”€/nix/store                  /dev/disk/by-uuid/784abb2f-b58f-49ef-bc07-38bf8843498b[/nix/store]
β”‚                                              ext4            rw,relatime
└─/boot                       /dev/nvme0n1p9   ext4            rw,relatime
  └─/boot/efi                 /dev/nvme0n1p1   vfat            rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro

Yep, findmnt shows a bit more details.

It might seem a bug in NixOS? Perhaps the intention was to rebind /nix/store to β€œread-only”, so that one cannot touch it accidentally?

Might you have set nix.readOnlyStore to false? The default is true.

I am new to this, how can I use nix repl to check that?

As I have said, /nix/store is bindmounted as itself, and the underlying device is /dev/,apper/root. So this is how it gets displayed. This is as expected and not a problem with nixos.

It depends, are you using flakes or regular channels?

I installed through the Graphical ISO image downloaded from nixos.org.

This is what I can see.

# nix-channel --list
nixos https://nixos.org/channels/nixos-22.05

Then nix-shell -p nixos-option --run 'nixos-option nix.readOnlyStore' is probably the easiest way

$ nixos-option nix.readOnlyStore
Value:
true

Default:
true

Type:
"boolean"

Description:
''
  If set, NixOS will enforce the immutability of the Nix store
  by making <filename>/nix/store</filename> a read-only bind
  mount.  Nix will automatically make the store writable when
  needed.
''

Declared by:
[ "/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/services/misc/nix-daemon.nix" ]

Defined by:
[ "/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/services/misc/nix-daemon.nix" ]

:frowning:

Assuming you’re using Gnome since you used the graphical installer, I believe I can comprehensively answer the original topic.

There are two mounts because NixOS uses what’s called a bind mount to make the Nix store readonly by default. mount -o ro --bind /foo /bar results in /bar containing the same things as /foo, but it’s readonly, and it’ll show up in your mount table as being the same file system as whichever contains /foo. So NixOS does mount -o ro --bind /nix/store /nix/store, which basically adds an entry to the mount table where all it does is make a readonly version of /nix/store, which appears to be the same FS as /.

Now the fact that your store is still rw I believe is because of a bug involving Gnome or Pantheon. This bug has been fixed in unstable, but it seems it hasn’t been backported to 22.05. The bug is that Gnome enables PackageKit by default, which has this bad habit of remounting whatever mount the /nix/store is as rw. So it remounts the bind mount as rw, making it totally redundant of course, but the wider problem is that it makes your store writable! The fix is here, but again, it doesn’t seem to be backported to 22.05.

3 Likes

In the meantime, you can fix it yourself by adding services.packagekit.enable = false; to your NixOS config.

2 Likes

That explains it @ElvishJerricco !

Thank you everyone. Nix community is fast.

1 Like