I have a normal shell.nix/default.nix setup where pkgs is a pinned version of nixpkgs, i.e. pkgs ? import (builtins.fetchTarball "https://github.com/NixOS/nixpkgs/archive/<sha256>.tar.gz") {}.
I also have keep-outputs = true and keep-derivations = true in nix.conf.
Now the interesting thing is that all other build inputs in that environment indeed stay put and do not get garbage-collected – just the pinned nixpkgs itself does. And this is a bother for two reasons:
(obvious) if I’m not connected to the internets, I’m screwed
(less obvious) fetching archives from github has gotten extremely slow and unreliable lately
Googling around the issue netted a lot of contradictory info, much of which is also probably obsolete, so it only confused me further.
This is because builtins.fetchTarball has a caching timeout that defaults to 1 hour.
The fetched tarball is cached for a certain amount of time (1 hour by default) in ~/.cache/nix/tarballs/ . You can change the cache timeout either on the command line with --option tarball-ttl number of seconds or in the Nix configuration file with this option: tarball-ttlnumber of seconds to cache .
You can set this with tarball-ttl. When the TTL expires, the ETag header is used to decide over freshness:
The number of seconds a downloaded tarball is considered fresh. If the cached tarball is stale, Nix will check whether it is still up to date using the ETag header. Nix will download a new version if the ETag header is unsupported, or the cached ETag doesn’t match.
Maybe GitHub changes ETag even if the tarball stays the same?
There exists a --no-net flag since nix 2.3 that essentially disables TTL (and all downloading, actually):
Add --no-net convenience flag. This flag disables substituters; sets the tarball-ttl setting to infinity (ensuring that any previously downloaded files are considered current); and disables retrying downloads and sets the connection timeout to the minimum. This flag is enabled automatically if there are no configured non-loopback network interfaces.
Actually in retrospect my question was fairly stupid: my usage of fetchTarball does not create a fixed-output derivation, and Nix has no way to know that the source hasn’t changed (there’s nothing magical about Github URLs, even when they contain commit hashes), so it has to re-download to make sure.
So if I change my usage to fixed-output I should be fine!
This a learning process, with many an insight on the way!
So, first of all: not keeping Nix source bundles (such as a specific Nixpkgs revision obtained by fetchTarball, etc.) is not a bug. Nix source is just not considered a reference for whatever results from evaluating it – otherwise you’d hold on to every channel state on your system.
Secondly, now that we know what the actual problem is (till next bout of enlightenment), the solution is along those lines:
{ nixpkgs ? (builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/<revision>.tar.gz";
sha256 = "<whatever that is>";
}),
... }:
let
pkgs = import nixpkgs {};
...
in stdenv.mkDerivation {
NIXPKGS_SRC = "${nixpkgs}";
# rest of the owl
...
}
The idea being: give a name to the fixed-output derivation of the source bundle itself (not just the result of importing it, as is the usual pattern), and store its path in a derivation artifact (like an environment variable, as in this example). This way the bundle is recorded as a reference of the derivation, and will stay alive as long as the derivation itself is.
I tried to add something like NIX_SOFTWARE_CENTER_KEEP = "${nix-software-center }";, but I get
error: The option `NIX_SOFTWARE_CENTER_KEEP' does not exist. Definition values:
- In `/etc/nixos/configuration.nix': "/nix/store/brynkmlmj9fa5ri201p3npy38n3lzh10-nix-software-center-0.1.2"
Is there any way to prevent the source derivation of a package from being garbage collected?
Your problem statement does not make much sense to me, I’m afraid.
Even if the source of nix-software-center gets re-fetched post-GC for some reason, there is no cause for the derivation to get rebuilt that I can see, because its (fixed-output) dependency does not in fact change – unless some other dependency does, that is.
Also you are trying to work around this by adding an attribute to a configuration module. Configuration modules are not derivations, and can only contain attributes that set declared options.
(Yeah, Nix is confusing an there is a steep learning curve. Sorry!)
The problem here is that your fixed-output-derivation isn’t actually an input to anything. When you do import (fetchFromGitHub {...}) {}, you’re immediately importing it and not making it an input to anything. A source tree of nix expressions is not necessarily an input to the derivations those expressions provide. So it’s just getting added to the store, evaluated by the nix evaluator, and then nothing happens to refer to it anymore.
Actually, I think the proper solution to my issue is to add these packages to nixpkgs, but I will wait for the dev to be ready to release them. In that scenario, the packages would be declared in the configuration and updated when a new version comes in nixpkgs.
In the meantime, I think it makes more sense to install these packages with nix-env or nix profile. That way, I don’t have to redownload the source or rebuild after each GC. I would anyway have to update manually the version in the configuration file, so I better just install the package manually after each release.
@cmm, it is easy to reproduce the package being rebuilt after each GC with something like sudo nix-collect-garbage -d && sudo nixos-rebuild switch. I can repeat this any number of time and each time the package gets rebuilt (unless I have the cachix substituter in my config, in which case it will only download the source again).
Thank you for the clarification about the difference between configuration module and derivation.
@ElvishJerricco is there a way to have the fixed-output-derivation be an input to something that will keep referring to it?