Nix without (p)root: error getting status of '/nix'

Every so often I have to use some high-performance computing systems which comes with the typical problems: only very few programs are preinstalled and those that are have ancient versions. On top of that I want to build my own programs with Nix, of course!

To this end I decided to build a static Nix from source. Brotli needed a patch for static linking and I patched Nix with some fixes for NFS file systems.

with import <nixpkgs> {
  overlays = [ (final: prev: {
    brotli = prev.brotli.overrideAttrs (_: {
      patches = [ ./brotli.patch ];
    });

    nix = prev.nix.overrideAttrs ({ patches ? [], ...}: {
      patches = patches ++ [ ./nfs.patch ./sqlite.patch ];
    });
  }) ];
};

pkgsStatic.nix

brotli.patch: http://dpaste.com/AH7N4CZ7D (expires in 10 days)
nfs.patch: http://dpaste.com/749RLT3MJ (expires in 10 days)
sqlite.patch: http://dpaste.com/H7WKPPKAT (expires in 10 days)

Then I built the binary, edited the permissions and distributed it to the HPC system:

$ nix build
$ cp -vLR result/ nix
$ chmod -R u+w nix/
$ scp -r nix/* hpc:.local/

On to the HPC system, where I add Nix to the $PATH and try to execute it

hpc $ export PATH="$HOME/.local:$PATH:
hpc $ nix run --store ~/.nix nixpkgs.hello
error: getting status of '/nix': No such file or directory

Here I am stuck. I don’t see what is going on and the error message is not very helpful. Surprisingly there is strace installed on the cluster and I can even run Nix in it, but also the trace is not very insightful. Here are the last 20 lines of the strace:

read(4, "SQLite format 3\0\20\0\1\1\0@  \0\0\0\7\0\0\0\v"..., 4096) = 4096
fcntl(4, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
brk(0x1d41000)                          = 0x1d41000
brk(0x1d42000)                          = 0x1d42000
brk(0x1d43000)                          = 0x1d43000
brk(0x1d44000)                          = 0x1d44000
brk(0x1d45000)                          = 0x1d45000
brk(0x1d46000)                          = 0x1d46000
brk(0x1d47000)                          = 0x1d47000
brk(0x1d48000)                          = 0x1d48000
lstat("/nix", 0x7ffdb1b2ffd8)           = -1 ENOENT (No such file or directory)
)                  = 4
futex(0x7fe51e2dd99c, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1d11c8c, FUTEX_WAKE_PRIVATE, 1) = 1
write(2, "\33[31;1merror:\33[0m getting status"..., 74error: getting status of '/nix': No such file or directory
) = 74
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT PIPE TERM WINCH], 8) = 0
rt_sigprocmask(SIG_SETMASK, [HUP INT PIPE TERM WINCH], NULL, 8) = 0
madvise(0x1cd4000, 192512, MADV_DONTNEED) = 0
exit_group(1)                           = ?
+++ exited with 1 +++

Try choosing the store with these environment variables:
https://github.com/NixOS/nix/blob/387f824cab50682e373ade49dcec4e6f99c10a42/tests/common.sh.in#L5-L13

I think the path to the store can be changed with ./configure options as well.

No, that’s not what I want. I want to make use of user namespaces via the --store option of Nix 2.0, instead of hard-coding the store (see also Nix Installation Guide - NixOS Wiki).

Ah, I’ve never tried that. You might be interested in nix-user-chroot, if you have access to user namespaces.

Just to be sure, have you considered https://github.com/nix-community/nix-user-chroot ?
I use it on HPC clusters, but it requires user namespaces.

I had considered this previously but it was giving me some problems. However, now I have actually returned to it and given up on a static Nix build. To this end I built the Nix release tarball from source on my NixOS machine.

Then I transferred the resulting nix-2.3.7-x86_64-linux.tar.xz to the HPC system and downloaded nix-user-chroot which I deposited in ~/.local/bin, which is in my PATH.

Next I extracted the tarball, entered the chroot and ran the install script.

Finally I added an overlay for Nix in ~/.config/nixpkgs/overlays/ which includes the patches, so they don’t get lost on an update and installed my stuff.

There are some additional goodies that I built around all of this including a wrapper around nix-user-chroot so that I can just run nix-shell without having to enter the chroot first. If anyone is interested in the exact procedures of each step, please ping me and I’ll add details.

Better, setup a little project on github or other, I’ll test it.

I was literally dealing with this today at work. I am demoing Nix as a build tool, and basically needed to get it running on a CentOS 7 box without root, or the ability to make the /nix directory. We are a few minor versions shy of being able to use unshare the way that nix-user-chroot requires, so I had to get creative.

I am building from source and modifying the store paths, which the manual warns is a pain in the ass, but I’m making progress. I am in stage 2 of bootstrapping now, so I have built Nix, and can build programs which depend on the store, but Nix itself still depends on libs outside of the store. Nix is currently building Nix so hopefully tomorrow I will be able to run with an empty environment.

Here is what I had to do:

  1. Build all of Nix’s dependencies from source. ( Time consuming. )
  2. Build Nix from source, and activate ./configure --prefix=${HOME}/.local --with-store-dir=${HOME}/.nix/store --localstatedir=${HOME}/.nix/var
  3. Patch a few spots where pkg-config returned -R flags to g++ which it didn’t like.
  4. make -j then make install and add source ${HOME}/.local/etc/profile.d/nix.sh to .profile.
  5. nix-channel --add ... && nix-channel --update; nix -iA nixpkgs.hello Triggers a build for 2 1/2 hours because changing the store path invalidates all of the pre-compiled binaries.
  6. LD_LIBRARY_PATH=${HOME}/.nix-profile/lib hello works, but if LD_LIBRARY_PATH is unmanaged everything blows up because it’s pulling /lib64/libc.so.6.

So not a bad start, once Nix finishing building itself with store dependencies, I can demo nix by wiping out my usual environment paths.