What should I expect to find in `cache.nixos.org`?

Help
1

TLDR

How is https://cache.nixos.org populated?

Details

While experimenting with different approaches to the cross-cutting concerns of nixpkgs pinning and multiple Python versions, I appear to have launched a complete compilation of a large set of Python packages (scipy, jupyter, etc.)

The scheme I’m trying is based around something like this:

{ py-version ? "37" }:

let
  commit-id = "f1a79c86358c5464c64b4fad00fca07a10e62a74";
  nixpkgs-url = "https://github.com/nixos/nixpkgs/archive/${commit-id}.tar.gz";
  pkgs = import (builtins.fetchTarball { url = nixpkgs-url; }) {};
  python = builtins.getAttr ("python" +  py-version ) pkgs;
  pypkgs = python.pkgs;
in

pkgs.mkShell {
  buildInputs = [
    pypkgs.scipy
    etc etc
 ]
}

The commit id corresponds to the nixos-20.03 found on https://status.nixos.org/ a few hours ago (which seems to have been updated about 18 hours before I triggered the compilation described below).

Activating this environment with any of

  • direnv
  • nix-shell shell.nix
  • nix-shell shell.nix --argstr py-version 37
  • nix-shell shell.nix --argstr py-version 38

activates the environment without serious delay, with first attempts on any specific python version clearly downloading lots from the cache, as witnessed by multiple lines of the form

copying path '/nix/store/vrxkfqqkvl9g0q386zc4nlhd7mx1mh6w-python3.8-pyOpenSSL-19.1.0' from 'https://cache.nixos.org'...

However, in the case of Python 3.6

  • nix-shell shell.nix --argstr py-version 36

launched a bunch of compilations which have been going for over an hour and seem in no danger of finishing any time soon.

So I would like to understand more about how https://cache.nixos.org is populated and what I can reasonably expect to be available in the cache.

2

See How long until an output path "disappears" from nixpkgs cache?.

3

Sorry if I’m being thick, but I don’t see how this leads me to understand how things get into the cache in the first place. It explains why things might be missing, but not why anything should ever be there at all.

What is the process that populates the cache?

(Are build jobs launched automatically under some circumstances? Is it users’ builds which feed build results into the cache? …)

4

That is described e.g. at https://howoldis.herokuapp.com/

5

specifically for python3.6, this package set hasn’t been built by hydra for a while, and that’s why you’re not getting any cache hits.

I think there’s ~3000 python packages per package interpreter, so we only keep around the two latest stable package sets

python2Packages is probably overdue for not building the entire package set…