I think it was mostly using overrideAttrs instead of override which did change more than you actually aimed for.
The build.sh Change was just because nix complained that it won’t override existing “result” symlinks.
Yep it works perfectly. Thanks for the help everyone. Here is the repo once again for anyone that is trying to figure this out again in the future: GitHub - danielbarter/hpc-nix
Note that there is a single-file statically linked Nix distribution now. You can get it as follows:
# curl -L https://hydra.nixos.org/job/nix/master/buildStatic.x86_64-linux/latest/download-by-type/file/binary-dist > nix
# chmod +x ./nix
and use it with a non-standard Nix store as follows:
# NIX_REMOTE='local?store=/home/eelco/nix/store&state=/home/eelco/nix/var/nix&log=/home/eelco/nix/var/log/nix' /path/to/nix \
--experimental-features 'nix-command flakes' \
build nixpkgs#hello
13 Likes
path '/nix/store/k7053q8cbnyplysc319n1n1fgbkxidzd-flake-registry.json' is not in the Nix store
Not quite sure the right way to add that in manually. Attemped store add-file and store add-path, but the paths end up wrong. Also tried manually placing it where it needs to be, but the db.sqlite probably needs the entry before that will work.
Or I’m missing something simple.
Try rm -rf ~/.cache/nix. It probably has files like ~/.cache/nix/flake-registry.json that point to the wrong store.
2 Likes
In case it helps, I just released nix-portable, which is static and uses bubblewrap/proot under the hood. It should work on all kinds of restricted systems and you can still use the cache.
1 Like
I tried this exact command line and it didn’t work:
[0/78 built, 0/43 copied] error: error: executing '/nix/store/2bw77pxjzs7yj09vlchnw96lk0dkxwx6-nix-2.4pre20210319_3e0e443-x86_64-unknown-linux-musl/libexec/nix/build-remote': No such file or directory
You probably should swap the /home/eelco for $HOME
Okay, you got me. I didn’t try the exact command line. I adapted $HOME. 
Hi there,
I am a Nix newbie and I recently joined the forum. I am having the same issues as @pimiddy trying to run Nix as an unprivileged user on a HPC environment without user namespaces. Is there any update on this?
I managed to get the static nix binary and run it as @edolstra write. Though I am getting several messages like the following when I try to build a derivation (I guess one message for each build dependency of the derivation) :
asked 'https://cache.nixos.org' for '/home/emilio/.nix/store/14xlcm1m7sg1l4di7g9p2kk8pf2l6m4f-hello-2.10.tar.gz' but got '/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz'
and then:
[0/96 built, 0/1 copied] Real-time signal 0
I am on a HPC cluster with an NFS mounted filesystem. Any help would be really appreciated.
Thanks a lot.
Best,
Emilio
Have you had a look at nix-portable?
1 Like
This error is caused by musl’s nonstandard use of signals. For a quick fix, you may remove all calls to setrlimit from nix.
Ref: Investigate why setrlimit setting in PAL on Alpine in breaks GDB · Issue #6767 · dotnet/runtime · GitHub
Thanks @j-k! I have just tried nix-portable but I get this error when I try to run anything, e.g.:
$ NP_DEBUG=2 ./nix-portable nix-shell -p hello
...
proot error: execve("/nix/store/p8d4qqiqcmx935m2b5a1gsmr6sp1ihsn-nix-2.4pre20201201_5a6ddb3/bin/nix-build"): No such file or directory
proot info: possible causes:
* the program is a script but its interpreter (eg. /bin/sh) was not found;
* the program is an ELF but its interpreter (eg. ld-linux.so) was not found;
* the program is a foreign binary but qemu was not specified;
* qemu does not work correctly (if specified);
* the loader was not found or doesn't work.
fatal error: see `proot --help`.
proot error: trying to remove a directory outside of '/tmp', please report this error.
proot error: can't chmod '/tmp/proot-13030-ueN2q1': No such file or directory
+ echo 'Fatal error: nix is unable to build packages'
Fatal error: nix is unable to build packages
+ exit 1
Hi @NickCao, thanks for your reply. I understand this would mean recompiling nix from source. Though, I am not sure I would really like to do so…
If you could open an issue on Issues · DavHau/nix-portable · GitHub with a full log that’d be great. I wouldn’t want to swamp this thread with debugging 
I am in the same situation - I don’t have usernamespaces available on our HPC and it is running on NFS. The process of compiling it on a few computers seems a bit daunting and I would like to avoid it if possible. I also thought maybe I could compile it to a static binary (with the store in my home-folder baked in at compile time) but I couldn’t find how to compile it into a static version.
Did you solve the issue? What I thought would be helpful to have and I am happy to try this, is to have a docker container with all dependencies included so that it is easy to compile a custom binary.
Did you ever solve the issue you described in this thread and are there any instructions?
Thanks
1 Like
@knedlsepp Thanks, this looks promising and I tried it, but just get an error:
warning: error: unable to download 'https://github.com/NixOS/flake-registry/raw/master/flake-registry.json': SSL peer certificate or SSH remote key was not OK (60); retrying in 310 ms
@hhoeflin: I think you need to set NIX_SSL_CERT_FILE. That said, I haven’t actually tried @edolstra way of doing it, which obviously is preferable. GitHub - danielbarter/hpc-nix worked on every cluster i tried it on, but the compile times are pretty painful (everything is rebuilt from scratch) and having the whole nix store in your home dir is inconvenient because of disk quotas.
I ran in to two more issues which aren’t really avoidable:
- If you are doing large multinode computations, you can run into problems where lots of workers are attempting to load shared libraries from your home directory. This is problematic because on clusters, the home dirs are usually NFS and I was seeing mmaps timing out
- The compilers from nixpkgs aren’t as good as the cluster specific ones on the cluster.
In the end, i just use the cluster environment as is, which isn’t ideal. I wonder if there are any HPC admins out there who like nix? It would be so amazing to have it baked into the system