I’m trying to use Nix for work. I’d like to fetchurls that have credentials in them, but it’s unclear how to approach this without making the username/password part of the derivation.
I ran into this in the past and was able to use the netrc-file option, but that option only worked for pkgs.fetchurlBoot and not pkgs.fetchurl. pkgs.fetchurlBoot has since been removed, so I’m kind of lost how to approach this.
Any ideas? How do you solve fetching private resources for your company?
I’m not sure how to do it with fetchurl but you could just write your own fixed output derivation. That way you can write a bash script build-phase and use whatever command you want. And it will have network access. You just have to get the final hash.
I’m in the same situation, this is what I have now:
fetchurlWithAuth = { url, sha256, ... }@attrs:
pkgs.fetchurl (attrs // {
inherit url sha256;
netrcPhase = ''
cookie_file=/etc/nix/oauth2_cookie
if ! test -r "$cookie_file"; then
echo "error: $cookie_file does not exist or is not readable by the build user ($(id))" >&2
echo "error: Please ensure that the cookie file path is added to extra-sandbox-paths in nix.conf, and restart nix-daemon after" >&2
exit 1
fi
curlOpts="$curlOpts -H Cookie:_oauth2_proxy=$(cat "$cookie_file")"
'';
});
I’m missing a way to do the same with builtins.fetchTarball. AFAICT, netrc cannot be used to inject cookies/headers.
And for private git repositories I made a copy of fetchGitPrivate from nixpkgs @ 2019 (since it was removed in november 2019). Although builtins.fetchGit is very nice, at $dayjob there are many big repositories (with submodules) and the single-threaded builtins.fetchGit can be prohibitively slow.
I’m interested in this too— it seems that NIX_CURL_FLAGS is now a no-op, and netrc-file in the nix.conf doesn’t do anything either. But attempting to use either of these options gives no indication that they’ve been removed and/or have limited scope.
I’m on point of junking it all and doing my own fetcher from scratch that just calls curl or aria2c directly. But there really needs to a standard, supported, universal solution for this. A lot of Google results end up on pages that recommend no-longer-working solutions, and that’s a bummer.
This is solved using Enterprise - NixOS Wiki.
# /etc/nix/netrc
machine DOMAINNAME
login USERNAME
password SECRET
for something like private github repos, you would do:
machine github.com
password TOKEN
Hmm, this was not my experience with Nix 2.4pre and the builtin fetchurl. Possibly I was using it wrong, but there was unfortunately very little in the way of feedback about what was going on— even just a log message indicating whether or not the rcfile had been found, parsed correctly, etc would have been immensely helpful.
In any case, even if does work, requiring an rcfile in the first place is not a great fit for many CI environments which manage secrets exclusively using envvars. I feel that that page would benefit from clearly calling out the option of creating your own impure fetcher, with links to examples.
I wrote that page 
I dug into this again today. Apparently pkgs.fetchurlBoot was indeed removed, but it is still available as stdenv.fetchurlBoot. Confusing to say the least. So, it now is possible to use netrc again. I updated the wiki page.
Because replacing all calls of fetchurl (and other kinds of fetch* calls, like fetchFromGitHub) is cumbersome and sometimes even impossible, I made the following issue in the past:
I still think this is a blocker for companies to adopt Nix in their infrastructure 
A starting point would be fetchurl itself. If I try to minimize that implementation it will look like:
myfetchurl.nix:
stdenvNoCC.mkDerivation {
url = ...;
outputHashAlgo = ...;
outputHash = ...;
builder = ./builder.sh
nativeBuildInputs = [ curl ];
preferLocalBuild = true;
impureEnvVars = [ "MYSECRET" ];
};
builder.sh:
curl --fail "$url" --output "$out" -u "$MYSECRET"
$ nix-build -f myfetchurl.nix
I’m not entirely sure these impureEnvVars will work for flakes (or nix build), as I’ve heard flakes will remain ‘pure’. I’m not on my Nix laptop currently, so haven’t tried this yet.
That said, I do think having a netrc available by default in Nix will make CI easier. I agree that environment variables are more convenient in most CIs, but they can be used to write a netrc file. Having the nix-daemon separate from the build-client often results in confusion, so that’s why I’m opting for a netrc file instead of envvars.
I definitely spent my first hour or so of playing with this trying to just pass a --header arg with NIX_CURL_FLAGS, and repeatedly examining the implementation in https://github.com/NixOS/nixpkgs/blob/a167cb8124663fbe72bfc4b7233a9c68116f21a9/pkgs/build-support/fetchurl/builder.sh to try to understand why in the heck it wasn’t working.
Then I read about the fetchurl/fetchurlBoot situation, but the takeaway I got was that the new builtin fetchurl supplanted both, and then I realised that NIX_CURL_FLAGS is now a “fake” implementation that really only handles the proxy-related stuff but silently drops any other flags you try to pass through it. Auugh.
Anyway, thank you for the effort you’ve put into trying to make Nix more enterprise friendly. As someone currently attempting to pitch Nix to my enterprise, it sure as heck isn’t easy when you have to spend half a day beating your head against how to make it pass the Private-Token header to GitLab. Nix is definitely a hard-stuff-easy/easy-stuff-hard kind of system.
adding a /etc/nix/netrc entry for github.com and api.github.com allows me to use flakes with private repos as inputs. I’m assuming that the other builtins.fetch* would also work. Note this is on unstable, and stable may not have the same features.
Do you think it is possible to pass access token header for gitlab at nixos-20.03?
I don’t think it is possible to set an arbitrary Authorization header in netrc, so I’m afraid it is not possible to do bearer token authentication.
However, if gitlab works like GitHub, you can use your personal access token as your password for basic authentication, which is what netrc provides.
@AleXoundOS Did you ever figure this out? I’m circling back to it now and finding it just as baffling as before. From some basic testing on the commandline, it really does seem like GitLab only accepts the header, and doesn’t allow passing the token in place of password for basic auth, which makes the netrc scheme useless. These are the options GitLab supports for API authentication:
It really would be a lot easier if there was a way to get arbitrary impureEnvVars into the build in NixOS, or at least an option for passing a bearer token with fetchurl.
It seems like the only real escape hatch here is jumping through hoops to set proxy vars on the nix-daemon, and then having to run an external proxy that handles auth. This is unacceptably complex for something that should be very simple and straightforward.
Generally, I don’t like the idea of using a custom authentication scheme like GitLab API tokens involved in the build. For fetching private repos I used builtins.fetchGit that uses ssh keys (from ~/.ssh) of the user who invoked nix-build/nix-shell.
Though, builtins.fetchGit has a serious drawback – nix is unable to recognize the presence of a fetched git repository workdir in /nix/store without an accompanying record in ~/.cache/nix/fetcher-cache-v1.sqlite. Thus, without it you cannot re-build a nix file/expression even if you have all fetched git repository workdirs in /nix/store. And in case the git server is unavailable and you have workdirs only in binary cache or /nix/store, you cannot build.
Of course, if you don’t need modifications to the nix file/expression, you can use a compiled deriver file, for example for nix-shell /nix/store/<HASH-NAME>.drv. But once you need to evaluate nix, you will need git fetch cache.
Maybe, flakes solve the issue, but I haven’t checked this yet.
Yeah I don’t think the Nix daemon should have to contain anything particular to GitLab, so that definitely rules out the PRIVATE-TOKEN header and the query string parameter option. But the Bearer token is still a somewhat generic OAUTH2 thing, so it would be reasonable for Nix to consider supporting that natively. It’s just frustrating that doing so would mean having to either extend the netrc file or create some parallel implementation for the machine → token mapping. There would really need to be at least one other commonly-used source host with this requirement for it to be justifiable.
In any case, I have requested that GitLab consider supporting tokens in basic auth, in line with what Github and others do: Personal Access Tokens should be usable in the API with basic auth (#341157) · Issues · GitLab.org / GitLab · GitLab
There is now this mentioned in the manual: nix.conf - Nix Reference Manual (cannot link deeper than this, search for “Personal” in the page), an option called access-tokens that contains… well, tokens. It is mentioned that the flake commands will use them but it would make sense that it’s used also in conjunction with the builtin fetchers.
I’ve tried using the netrc file trick to fetch commit patches from a private GitHub repo:
patches = [
(pkgs.fetchpatch {
url = "https://github.com/asymmetric/foo/commit/3ad368d93c05bd2744eec6ecaa5bf1fee127c263.patch";
hash = "sha256-MG71IEwRAllacAkRoB7Tn45+FbY7LAqTDkVJkoWuQUU=";
})
];
But I got a 404.
I’ve also added the PAT to access-tokens in both /etc/nix/nix.conf and ~/.config/nix/nix.conf. But I still got a 404.
I’m also not sure how to go the fetchurlBoot route, since it’s not fetchurl directly that I’m calling.
This is in a flake, FWIW.
This is starting to feel like a deep rabbit hole…
Just as a heads up to save you time : the netrc-file setting in nix.conf does not not effect built-in fetchers. The built-ins read /etc/nix/netrc and ~/.config/nix/netrc regardless of the netrc-file setting.
I was told this setting was only intended for supporting Nixpkgs but this isn’t documented.
Hopefully this helps some of you.
Has anyone able to find a solution to this? I attempted some of the things I saw in this thread and Enterprise - NixOS Wiki but to no avail 
I got this to work with private gitlab repositories by including an extra env file in the nix-daemon systemd service, the env file then sets NIX_CURL_FLAGS to add the required header to the curl calls. It’s not the most composable solution, as you can only set the env var in one location, but it does handle the authentication.
The env file looks something like this:
NIX_CURL_FLAGS=--header PRIVATE-TOKEN:glpat-xxxxxxxxxxxxxxxx
The most tricky part was to figure out how to avoid issues with the handling of quotes and spaces in the different layers between where we set the env var, and where it actually gets passed to curl, such that the value of the env var gets passed correctly as two separate arguments to the curl command.