After being chewed out by my sysadmin over slack for asking whether we can enable user namespaces on the compute cluster, I am interested in building a statically linked nix executable with custom /nix/store and /nix/var locations that I can transfer over to the cluster with a copy of nixpkgs and rebuild everything I need from scratch.
but the resulting executable keeps trying to lstat /nix when doing anything non trivial which is undesirable.
Is what I am trying to do even possible? If so, it would be super useful for people who want to use nix on large shared computer systems, and I would be more than happy to put in the work to get it done.
good idea @symphorien, but unfortunately setting those env variables doesnât fix the problem. Infact, i just notice that if you search for strings in the static executable generated by the above nix expression, it still has strings like /nix/store and /nix/var ect floating around, so im guessing that I am just missing some configure flags or something like that
I remember doing this a couple of years ago. I think this can be compared to bootstrapping a compiler toolchain on a new target. The point being that the nix that you built is only half way. It will âliveâ in /nix/store but know how to build for /foo. You will need to build a third nix using that second nix that will live in /foo and build for /foo.
@knedlsepp: that makes so much sense! So i built the nix which lives in /nix/store but targets /foo/.nix/store using the above derivation. When I try and use this new nix to build the derivation again i get the following error:
error: attribute 'unsafeDiscardStringContext' missing, at /home/danielbarter/nixpkgs/lib/strings.nix:429:13
which is very strange since unsafeDiscardStringContext appears to be a builtin function
Since youâre in an HPC environment your home directory is probably on NFS, so you need to build Nix with two extra patches, namely the attribute patch and the SQLite patch.
Regardless, i donât particularly like the proot solution: using qemu to implement a user chroot just feels kind of clunky to me. I would much prefer to build a static version of nix on my local machine and transfer it over.
The home dir is indeed nfs, so i will keep those patches in mind.
Iâve actually tried the approach from Rohitâs blog post before (building nix from source on the foreign machine). I ended up getting stuck because I couldnât for the life of me get their compiler to detect my downloaded version of libeditline. Avoiding this type of problem is why I like nix so much in the first place!
I am hopeful that I can get @knedlseppâs double build static nix working. At the very least it is a cool trick!
I guess you could try to get the ânormalâ dynamic build working first. That one is better supported. And then work your way to building statically. (I think for your application the âstaticâ part isnât as important as the custom prefix anyway)
configure flags: --disable-dependency-tracking --prefix=/global/home/users/dbarter/nix/store/z899n47cy9m793wr4832h1kc85zwjz73-gnum4-1.4.18 --with-syscmd-shell=/global/home/users/dbarter/nix/store/ahvz416s9795rszckcicvghzpd1mkhrw-bootstrap-tools/bin/bash
/global/home/users/dbarter/nix/store/0c1090hab3yb9xzihna7rf7cxhs1ix5i-bootstrap-stage1-stdenv-linux/setup: ./configure: /bin/sh: bad interpreter: No such file or directory
builder for '/global/home/users/dbarter/nix/store/pg6p7xl4dpl86adn0r3r2q7hf3r24lna-gnum4-1.4.18.drv' failed with exit code 126
Not sure what exactly is going wrong here. Iâll try some more tomorrow.
thanks @knedlsepp: building it now. Seems to be working. Was there some sort of aliasing issue which gets fixed by using nix, nix-intermediate and nix-final?
I think it was mostly using overrideAttrs instead of override which did change more than you actually aimed for.
The build.sh Change was just because nix complained that it wonât override existing âresultâ symlinks.
Yep it works perfectly. Thanks for the help everyone. Here is the repo once again for anyone that is trying to figure this out again in the future: GitHub - danielbarter/hpc-nix
path '/nix/store/k7053q8cbnyplysc319n1n1fgbkxidzd-flake-registry.json' is not in the Nix store
Not quite sure the right way to add that in manually. Attemped store add-file and store add-path, but the paths end up wrong. Also tried manually placing it where it needs to be, but the db.sqlite probably needs the entry before that will work.
In case it helps, I just released nix-portable, which is static and uses bubblewrap/proot under the hood. It should work on all kinds of restricted systems and you can still use the cache.
I tried this exact command line and it didnât work:
[0/78 built, 0/43 copied] error: error: executing '/nix/store/2bw77pxjzs7yj09vlchnw96lk0dkxwx6-nix-2.4pre20210319_3e0e443-x86_64-unknown-linux-musl/libexec/nix/build-remote': No such file or directory