I’m trying to start using a nix+home-manager on my centos 7 workstation to be able to use modern versions of tools and libraries without fiddling with containers and other workarounds. The issue I’m facing with is that the workstation is behind corporate firewall and I have to use custom cabundle to be able to connect to the Internet securely.
I followed the instructions on how to point nix to custom cabundle, but I still get an error when I try to perform an initial set up of home manager with command nix-shell '<home-manager>' -A install:
Creating initial Home Manager generation...
warning: error: unable to download 'https://git.sr.ht/~rycee/nmd/archive/abb15317ebd17e5a0a7dd105e2ce52f2700185a8.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 312 ms
warning: error: unable to download 'https://git.sr.ht/~rycee/nmd/archive/abb15317ebd17e5a0a7dd105e2ce52f2700185a8.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 506 ms
warning: error: unable to download 'https://git.sr.ht/~rycee/nmd/archive/abb15317ebd17e5a0a7dd105e2ce52f2700185a8.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 1064 ms
warning: error: unable to download 'https://git.sr.ht/~rycee/nmd/archive/abb15317ebd17e5a0a7dd105e2ce52f2700185a8.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 2328 ms
error:
… while calling the 'derivationStrict' builtin
at /builtin/derivation.nix:9:12: (source not available)
… while evaluating derivation 'home-manager-generation'
whose name attribute is located at /nix/store/q300rswsxpr2kkng9azzsbfi9m8fdg50-nixpkgs/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:303:7
… while evaluating attribute 'buildCommand' of derivation 'home-manager-generation'
at /nix/store/q300rswsxpr2kkng9azzsbfi9m8fdg50-nixpkgs/nixpkgs/pkgs/build-support/trivial-builders/default.nix:87:14:
86| enableParallelBuilding = true;
87| inherit buildCommand name;
| ^
88| passAsFile = [ "buildCommand" ]
(stack trace truncated; use '--show-trace' to show the full trace)
error: unable to download 'https://git.sr.ht/~rycee/nmd/archive/abb15317ebd17e5a0a7dd105e2ce52f2700185a8.tar.gz': SSL peer certificate or SSH remote key was not OK (60)
Uh oh, the installation failed! Please create an issue at
https://github.com/nix-community/home-manager/issues
Things I’ve done:
Set env vars: export NIX_SSL_CERT_FILE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem same for SSL_CERT_FILE
Pointed nix to the custom openssl.cnf file to enable unsafe legacy negotiation (I’ve got error 35 without it)
Verified that nix is using specified cabundle: nix --extra-experimental-features nix-command show-config | grep ssl returns ssl-cert-file = /etc/pki/tls/cert.pem which is a symlink to tls-ca-bundle.pem
I’ve also put ca-bundle file into nix-store and tried to point nix to it with env vars as well as by specifying it in home.nix file. Nothing changed.
Also tried to set export NIX_CURL_FLAGS="--insecure as well as export CURL_CA_BUNDLE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
I’ve tried to run nix-shell with --impure flag as well, without any effect. I’m able to download this file with curl on the host shell without any issues, also I can launch a nix-shell -p neovim without problems. Nix is installed in single-user mode. Host OS is CentOS 7.
I can conclude that binary packages from cache.nixos.org are downloading without issues, it’s just an issue with fetching files from git repos. Although, I’m failing to find the exact place and reason why.
I would greatly appreciate any advise on how to troubleshoot or fix this.
Doing MITM with TLS is not very secure and the MITM proxy is a really high risk target and when it gets owned, everything is owner.
You also need to add them to the allowed paths in the nix sandbox.
I think that has the sandbox per default disabled which would help your initial bootstrap but it generally unsupported.
I think the error you get is related to home-manager using fetchTarball here https://github.com/nix-community/home-manager/blob/b9046172a580a648045e7c2f7077cc143936ad8f/docs/default.nix#L10
which is bad for evaluation time and forces a regular re-fetch. To fix that I think you need to convince nix or the nix-daemon to load your certificate bundle. I am not fully sure how to do that. The best option would be if home-manager wouldn’t be using fetchTarball.
Well, that’s InfoSec requirement, not mine. All traffic is actively monitored and there is no way to circumvent that.
You also need to add them to the allowed paths in the nix sandbox.
Not in single-user mode, right?
To fix that I think you need to convince nix or the nix-daemon to load your certificate bundle.
nix seems to be convinced as nix --extra-experimental-features nix-command show-config shows correct cabundle file path. nix-daemon is not used in single-user mode.
I think the error you get is related to home-manager using fetchTarball
I guess it should be possible to prefetch required tarballs and put them in cache, so when I build home-manager it will use cached tar instead.
I did another session of troubleshooting this. Given hint from @Sandro about fetchTarball led me to nix cache it’s checking before attempting to download the file. So, I tried to prefetch it. Unsurpisingly, the nix store prefetch-file command failed with the same issue. I went ahead and strace’d it:
strace nix --extra-experimental-features nix-command store prefetch-file https://git.sr.ht/\~rycee/nmd/archive/abb15317ebd17e5a0a7dd105e2ce52f2700185a8.tar.gz
Despite the NIX_SSL_CERT_FILE being set to correct path, nix still tried to get ca certs from the default paths.
newfstatat(AT_FDCWD, "/etc/ssl/certs/ca-certificates.crt", 0x7ffcb60a4020, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt", 0x7ffcb60a4020, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
I copied my ca bundle to the default path, but nothing changed – the error is still there.
but, running on host shell curl -vvvv https://git.sr.ht/\~rycee/nmd/archive/abb15317ebd17e5a0a7dd105e2ce52f2700185a8.tar.gz works just fine and reports certificate expiring on Jul 26th
I don’t get how nix is supposed to work with ca bundles. According to strace it reads the very same file as the host os’ curl. The only difference here is probably libcurl itself. If nix somehow uses other ca-certificates.crt without custom CA it would throw * NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER), not cert expired error.
It turned out to be indeed an expired CA certificate in ca-bundle. Somehow, curl on the host was fine with that making me following wrong path. Now home-manager is installing fine.
I would highly recommend that you write down all the time you waste with this. Maybe you can motivate some colleagues, too. Then collection present the number of wasted hours and maybe something starts moving.
I know a story where people did that and even wrote a little website where people can track it and eventually management saw how much time was wasted on this and things for migrated to only package inspection without weakening TLS.