Building a statically linked nix for hpc environments

Development
11

@hmenke: Yah, i added the patches.

12

I added a PR that I think should fix your problem: Make use of existing nix parameters by knedlsepp · Pull Request #1 · danielbarter/hpc-nix · GitHub

13

thanks @knedlsepp: building it now. Seems to be working. Was there some sort of aliasing issue which gets fixed by using nix, nix-intermediate and nix-final?

14

I think it was mostly using overrideAttrs instead of override which did change more than you actually aimed for.
The build.sh Change was just because nix complained that it won’t override existing “result” symlinks.

15

Yep it works perfectly. Thanks for the help everyone. Here is the repo once again for anyone that is trying to figure this out again in the future: GitHub - danielbarter/hpc-nix

16

Note that there is a single-file statically linked Nix distribution now. You can get it as follows:

# curl -L https://hydra.nixos.org/job/nix/master/buildStatic.x86_64-linux/latest/download-by-type/file/binary-dist > nix
# chmod +x ./nix

and use it with a non-standard Nix store as follows:

# NIX_REMOTE='local?store=/home/eelco/nix/store&state=/home/eelco/nix/var/nix&log=/home/eelco/nix/var/log/nix' /path/to/nix \
  --experimental-features 'nix-command flakes' \
  build nixpkgs#hello
13 Likes
17 
path '/nix/store/k7053q8cbnyplysc319n1n1fgbkxidzd-flake-registry.json' is not in the Nix store

Not quite sure the right way to add that in manually. Attemped store add-file and store add-path, but the paths end up wrong. Also tried manually placing it where it needs to be, but the db.sqlite probably needs the entry before that will work.

Or I’m missing something simple.

18

Try rm -rf ~/.cache/nix. It probably has files like ~/.cache/nix/flake-registry.json that point to the wrong store.

2 Likes
19

In case it helps, I just released nix-portable, which is static and uses bubblewrap/proot under the hood. It should work on all kinds of restricted systems and you can still use the cache.

1 Like
20

I tried this exact command line and it didn’t work:

[0/78 built, 0/43 copied] error: error: executing '/nix/store/2bw77pxjzs7yj09vlchnw96lk0dkxwx6-nix-2.4pre20210319_3e0e443-x86_64-unknown-linux-musl/libexec/nix/build-remote': No such file or directory
21

You probably should swap the /home/eelco for $HOME

22

Okay, you got me. I didn’t try the exact command line. I adapted $HOME. :wink:

23

Hi there,

I am a Nix newbie and I recently joined the forum. I am having the same issues as @pimiddy trying to run Nix as an unprivileged user on a HPC environment without user namespaces. Is there any update on this?

I managed to get the static nix binary and run it as @edolstra write. Though I am getting several messages like the following when I try to build a derivation (I guess one message for each build dependency of the derivation) :

asked 'https://cache.nixos.org' for '/home/emilio/.nix/store/14xlcm1m7sg1l4di7g9p2kk8pf2l6m4f-hello-2.10.tar.gz' but got '/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz'

and then:

[0/96 built, 0/1 copied] Real-time signal 0

I am on a HPC cluster with an NFS mounted filesystem. Any help would be really appreciated.

Thanks a lot.

Best,
Emilio

24

Have you had a look at nix-portable?

1 Like
25

This error is caused by musl’s nonstandard use of signals. For a quick fix, you may remove all calls to setrlimit from nix.
Ref: Investigate why setrlimit setting in PAL on Alpine in breaks GDB · Issue #6767 · dotnet/runtime · GitHub

26

Thanks @j-k! I have just tried nix-portable but I get this error when I try to run anything, e.g.:

$ NP_DEBUG=2 ./nix-portable nix-shell -p hello
...
proot error: execve("/nix/store/p8d4qqiqcmx935m2b5a1gsmr6sp1ihsn-nix-2.4pre20201201_5a6ddb3/bin/nix-build"): No such file or directory
proot info: possible causes:
  * the program is a script but its interpreter (eg. /bin/sh) was not found;
  * the program is an ELF but its interpreter (eg. ld-linux.so) was not found;
  * the program is a foreign binary but qemu was not specified;
  * qemu does not work correctly (if specified);
  * the loader was not found or doesn't work.
fatal error: see `proot --help`.
proot error: trying to remove a directory outside of '/tmp', please report this error.

proot error: can't chmod '/tmp/proot-13030-ueN2q1': No such file or directory
+ echo 'Fatal error: nix is unable to build packages'
Fatal error: nix is unable to build packages
+ exit 1
27

Hi @NickCao, thanks for your reply. I understand this would mean recompiling nix from source. Though, I am not sure I would really like to do so…

28

If you could open an issue on Issues · DavHau/nix-portable · GitHub with a full log that’d be great. I wouldn’t want to swamp this thread with debugging :sweat_smile:

29

I am in the same situation - I don’t have usernamespaces available on our HPC and it is running on NFS. The process of compiling it on a few computers seems a bit daunting and I would like to avoid it if possible. I also thought maybe I could compile it to a static binary (with the store in my home-folder baked in at compile time) but I couldn’t find how to compile it into a static version.

Did you solve the issue? What I thought would be helpful to have and I am happy to try this, is to have a docker container with all dependencies included so that it is easy to compile a custom binary.

Did you ever solve the issue you described in this thread and are there any instructions?

Thanks

1 Like
30

I think you want Building a statically linked nix for hpc environments - #16 by edolstra

1 Like