Building a statically linked nix for hpc environments

Development
#1

After being chewed out by my sysadmin over slack for asking whether we can enable user namespaces on the compute cluster, I am interested in building a statically linked nix executable with custom /nix/store and /nix/var locations that I can transfer over to the cluster with a copy of nixpkgs and rebuild everything I need from scratch.

Here is what I have so far:

with (import <nixpkgs> {});

pkgsStatic.nix.overrideAttrs (oldAttrs: rec {
  configureFlags = [ "--with-store-dir=/foo/.nix/store"
                     "--localstatedir=/foo/.nix/var"
                     "--sysconfdir=/foo/.nix/etc"
                     "--enable-gc"
                     "--disable-shared"
                     "--build=x86_64-unknown-linux-gnu"
                     "--host=x86_64-unknown-linux-musl" ];
    })

but the resulting executable keeps trying to lstat /nix when doing anything non trivial which is undesirable.

Is what I am trying to do even possible? If so, it would be super useful for people who want to use nix on large shared computer systems, and I would be more than happy to put in the work to get it done.

3 Likes
Make statically-linked Nix binaries discoverable
#2

Maybe try using these environment variables at runtime:
https://github.com/NixOS/nix/blob/0df69d96e02ce4c9e17bd33333c5d78313341dd3/tests/common.sh.in#L5-L14

nix uses these to run its test so they should work :wink:

#3

good idea @symphorien, but unfortunately setting those env variables doesn’t fix the problem. Infact, i just notice that if you search for strings in the static executable generated by the above nix expression, it still has strings like /nix/store and /nix/var ect floating around, so im guessing that I am just missing some configure flags or something like that

#4

I remember doing this a couple of years ago. I think this can be compared to bootstrapping a compiler toolchain on a new target. The point being that the nix that you built is only half way. It will “live” in /nix/store but know how to build for /foo. You will need to build a third nix using that second nix that will live in /foo and build for /foo.

#5

@knedlsepp: that makes so much sense! So i built the nix which lives in /nix/store but targets /foo/.nix/store using the above derivation. When I try and use this new nix to build the derivation again i get the following error:

error: attribute 'unsafeDiscardStringContext' missing, at /home/danielbarter/nixpkgs/lib/strings.nix:429:13

which is very strange since unsafeDiscardStringContext appears to be a builtin function

#6

If you can’t use user namespaces, maybe the PRoot variant works for you: Nix Installation Guide - NixOS Wiki

Since you’re in an HPC environment your home directory is probably on NFS, so you need to build Nix with two extra patches, namely the attribute patch and the SQLite patch.

See also this blog post: Local Nix without Root :: Rohit Goswami — Reflections

#7

I tried using proot, but I couldn’t get a static version building on either nixpkgs master or nixpkgs stable (https://github.com/NixOS/nix/issues/4432).

Regardless, i don’t particularly like the proot solution: using qemu to implement a user chroot just feels kind of clunky to me. I would much prefer to build a static version of nix on my local machine and transfer it over.

The home dir is indeed nfs, so i will keep those patches in mind.

I’ve actually tried the approach from Rohit’s blog post before (building nix from source on the foreign machine). I ended up getting stuck because I couldn’t for the life of me get their compiler to detect my downloaded version of libeditline. Avoiding this type of problem is why I like nix so much in the first place!

I am hopeful that I can get @knedlsepp’s double build static nix working. At the very least it is a cool trick!

#8

I guess you could try to get the “normal” dynamic build working first. That one is better supported. And then work your way to building statically. (I think for your application the “static” part isn’t as important as the custom prefix anyway)

#9

@knedlsepp: seems like a good idea, then you can just transfer the store you build over (possibly with some fiddling to get the symlinks to stick).

Here is where I got: GitHub - danielbarter/hpc-nix

but this is giving me the following error:

configure flags: --disable-dependency-tracking --prefix=/global/home/users/dbarter/nix/store/z899n47cy9m793wr4832h1kc85zwjz73-gnum4-1.4.18 --with-syscmd-shell=/global/home/users/dbarter/nix/store/ahvz416s9795rszckcicvghzpd1mkhrw-bootstrap-tools/bin/bash
/global/home/users/dbarter/nix/store/0c1090hab3yb9xzihna7rf7cxhs1ix5i-bootstrap-stage1-stdenv-linux/setup: ./configure: /bin/sh: bad interpreter: No such file or directory
builder for '/global/home/users/dbarter/nix/store/pg6p7xl4dpl86adn0r3r2q7hf3r24lna-gnum4-1.4.18.drv' failed with exit code 126

Not sure what exactly is going wrong here. I’ll try some more tomorrow.

#10

If your home is on NFS you definitely need those patches. Without the attribute patch you won’t even be able to install Nix, but instead receive

error: removing extended attribute ‘system.nfs4_acl‘

and without the SQLite patch, the Nix database will either deadlock or become corrupted sooner or later.

#11

@hmenke: Yah, i added the patches.

#12

I added a PR that I think should fix your problem: https://github.com/danielbarter/hpc-nix/pull/1

#13

thanks @knedlsepp: building it now. Seems to be working. Was there some sort of aliasing issue which gets fixed by using nix, nix-intermediate and nix-final?

#14

I think it was mostly using overrideAttrs instead of override which did change more than you actually aimed for.
The build.sh Change was just because nix complained that it won’t override existing “result” symlinks.

#15

Yep it works perfectly. Thanks for the help everyone. Here is the repo once again for anyone that is trying to figure this out again in the future: GitHub - danielbarter/hpc-nix

#16

Note that there is a single-file statically linked Nix distribution now. You can get it as follows:

# curl -L https://hydra.nixos.org/job/nix/master/buildStatic.x86_64-linux/latest/download-by-type/file/binary-dist > nix
# chmod +x ./nix

and use it with a non-standard Nix store as follows:

# NIX_REMOTE='local?store=/home/eelco/nix/store&state=/home/eelco/nix/var/nix&log=/home/eelco/nix/var/log/nix' /path/to/nix \
  --experimental-features 'nix-command flakes' \
  build nixpkgs#hello
10 Likes
#17 
path '/nix/store/k7053q8cbnyplysc319n1n1fgbkxidzd-flake-registry.json' is not in the Nix store

Not quite sure the right way to add that in manually. Attemped store add-file and store add-path, but the paths end up wrong. Also tried manually placing it where it needs to be, but the db.sqlite probably needs the entry before that will work.

Or I’m missing something simple.

#18

Try rm -rf ~/.cache/nix. It probably has files like ~/.cache/nix/flake-registry.json that point to the wrong store.

2 Likes
#19

In case it helps, I just released nix-portable, which is static and uses bubblewrap/proot under the hood. It should work on all kinds of restricted systems and you can still use the cache.

1 Like
#20

I tried this exact command line and it didn’t work:

[0/78 built, 0/43 copied] error: error: executing '/nix/store/2bw77pxjzs7yj09vlchnw96lk0dkxwx6-nix-2.4pre20210319_3e0e443-x86_64-unknown-linux-musl/libexec/nix/build-remote': No such file or directory
Hosted by Flying Circus.