Allowing /nix/store to be a symlink, in order to allow a daemon-less multi-user installation

Development
1

Hi,

I find nix to be really useful, and I started using it in my company. However, it’s currently quite inconvenient. The reason is that we work on servers which host many users, where only IT people have root access. I consulted with the IT guy, and he really didn’t want to install the full nix daemon. He agreed to create an empty directory /nix, which allowed me to use bubblewrap to bind it to a personal directory. However, this means that everything using nix must run under bubblewrap.

I had an idea which would allow multiple users to use nix without having IT being responsible for a daemon which compiles stuff and may use a lot of storage not associated with specific users. I created a very simple read-only FUSE filesystem, which has two files, store and var, which are symlinks to ~/nix/store and ~/nix/var of the user viewing the filesystem. So, it looks like this:

❯ ll /nix
total 1.0K
lrwxrwxrwx 1 noamyr noamyr 22 Jan  1  1970 store -> /home/noamyr/nix/store
lrwxrwxrwx 1 noamyr noamyr 20 Jan  1  1970 var -> /home/noamyr/nix/var

However, I then discovered that nix forbids symlinks. I added the configuration allow-symlinked-store = true, but it still doesn’t work:

❯ nix-shell --packages cowsay
error:
       … while evaluating a branch condition
         at /nix/store/1wirgshrfl96hc8nj397n98llpj30w3m-nixpkgs/nixpkgs/pkgs/stdenv/booter.nix:99:7:
       … in the right operand of the update (//) operator
         at /nix/store/1wirgshrfl96hc8nj397n98llpj30w3m-nixpkgs/nixpkgs/pkgs/stdenv/booter.nix:84:7:
       (stack trace truncated; use '--show-trace' to show the full trace)

       error: path '«unknown»/nix/store' is a symlink

I found here a reason for not allowing symlinks:

The reason is that builders could canonicalize a path (e.g. do realpath $dir) and end up storing the symlink target in derivation outputs. Which is bad if the output is shared with other machines via a binary cache.

Hmm. So I thought that I can add a simple command, say nix-realpath, which will be exactly like realpath, except for that if the path starts with /nix/store or /nix/var, the first two elements would remain as they are. Then, all realpath usages in nixpkgs could be replaced with nix-realpath, and then the home directory would not reach the binary cache.

And now to the question: what is the chance of the Nix project accepting such a change? Specifically, those are the changes I see:

  1. Modify PosixSourceAccessor::assertNoSymlinks() to avoid checking for symlinks at /nix/store or /nix, if allow-symlinked-store = true.
  2. Add a utility nix-realpath (or whatever should be the name, or even adding an argument --nix to the realpath shipped with nix in coreutils).
  3. Modify many realpath usages to use nix-realpath. That would be a large pull request.

The benefit is that it would be much easier for IT people to agree to having this tiny read-only filesystem, than to agree to being responsible for a daemon which could use significant computation and storage.

What do you think?

3 Likes
2

It’s not just the realpath shell utility though. The processes running in a nix build can also call realpath from glibc, musl, or whatever other libc. Some binaries (i.e. go lang) don’t even link with a libc and perform kernel syscalls directly.

Given your options, I think your best bet is still bwrap. Maybe you could hack your shell profile to always enter a bubblewrap namespace with /nix/store mounted on top.

I haven’t tested this at all, but maybe something conceptually similar to this would work:

# bashrc
# Re-exec itself in bwrap if /nix/store does not exist
if [ ! -d /nix/store ]; then
    exec bwrap .. "$0" "$@"
fi
3

Thanks for the suggestion. I’m sure that the easiest thing for me would be to use bubblewrap. I want to check with the nix community if there’s a chance for removing the symlink restriction.

I don’t think it would be a big problem for any builder to use the modified realpath. Any program could just use a subprocess, or implement the logic in the language. I see that realpath is provided by glibc, but I don’t think the logic is quite complicated, so it should be reasonable to implement an alternative in any language.