I’ve been observing the following strange behaviors.
- Nix rebuilding a package with a built
outPathin the Nix store. - Rebuilding when building from
nix replwith:b, but not withnix build. - Rebuilding when building the derivation, but not with
python.withPackages.
To illustrate what I mean, consider the following “minimal” working example based on the flake maipkgs. The exact structure of the flake isn’t terribly important; the relevant parts are that it exposes the Python package jax-triton through packages.${system}.jax-triton as well as through legacyPackages.${system}.python.withPackages thanks to a custom python3Packages (the logic is here and the python3Packages.overrideScope is here). The choice of jax-triton is arbitrary, it’s just an easy Python package to build from maipkgs.
Now, attempting to build with
nix build github:stephen-huan/maipkgs#jax-triton --dry-run
will give the following list, as it hasn’t been built yet.
these 4 derivations will be built:
/nix/store/jjr5j53aa6sravk1l32qx9jsdf2rgkxm-triton_call-annotation.patch.drv
/nix/store/vs9qb8bslcrk5crcgi17v09n2rfvwabp-cpu-backend.patch.drv
/nix/store/zp7z9mhribrbnc2g6snag7kpbcz1i2n6-jax_triton-0.2.0.tar.gz.drv
/nix/store/3zwncibwsls5diamvn4kf9f0nykn8zsa-python3.12-jax-triton-0.2.0.drv
these 20 paths will be fetched (32.77 MiB download, 149.11 MiB unpacked):
/nix/store/m451blikv3a8qkgdvycpaf8n1zsw57pc-curl-8.11.0
/nix/store/1jvwjgdcql9sf3dzlrwmawk8h0jwrvv3-curl-8.11.0-bin
/nix/store/nxlbfypkj6yasm2ncbnpk111rznv1yar-curl-8.11.0-dev
/nix/store/zxk392rmg10nmmakhb2h1rmdlsbws4sz-curl-8.11.0-man
/nix/store/b7a4blipfgnyisg9zbyqy03w9bgc4v13-krb5-1.21.3
/nix/store/2d9mb8lwkkwhyg20s0bimcn1h4n02sh0-krb5-1.21.3-dev
/nix/store/wlnnqz3kvrxc5c61agpqnbkwl9ldhs7d-krb5-1.21.3-lib
/nix/store/5c2gpskzha2ajsajsna02ab41n05ifqg-libssh2-1.11.1
/nix/store/ykrrg1dlmd1awjpxbazxfpf6rsjd59vr-libssh2-1.11.1-dev
/nix/store/fvd90pv9l7bzgszciv0adhivysb95jnh-mirrors-list
/nix/store/n1ym5671hgzb5av57snjpjlj8lp190pn-nghttp2-1.64.0
/nix/store/c66hvpbf3n6fpf80v32m0algn26zgph3-nghttp2-1.64.0-dev
/nix/store/3ksmqmxiqdz5yxglkgrnanbgca8kgq0a-nghttp2-1.64.0-lib
/nix/store/nx4gsm1d7n3xnn2ab5vqq7hvdp4264bb-openssl-3.3.2
/nix/store/zfvb25n6y60k6nkfhqj2bl89c9zy551y-openssl-3.3.2-bin
/nix/store/hybgiy7s6m987dw6fxjrs6h0pz85k6i6-openssl-3.3.2-dev
/nix/store/8a8bp36da0kqj3lhb5lp462ygjy6s5a7-patchutils-0.4.2
/nix/store/lflc8h97ld3p0ximi4qhrdlrr2317pgf-python3.12-setuptools-scm-8.1.0
/nix/store/bkg5gkkwb8pvhyjknsisdn8pzbizdlb1-python3.12-triton-3.1.0
/nix/store/spb2bpcnw0gbbr4x94cq8xs9n72hipwj-stdenv-linux
Actually building with
nix build github:stephen-huan/maipkgs#jax-triton --no-link --print-out-paths
gives the path
/nix/store/qiljpd83hlx0vv7cli75ng4nx9dn9h1b-python3.12-jax-triton-0.2.0
Now, pin a python.withPackages environment. For concreteness, I used the below flake
{
description = "mwe";
inputs = {
nixpkgs.follows = "maipkgs/nixpkgs";
maipkgs.url = "github:stephen-huan/maipkgs";
};
outputs = { self, nixpkgs, maipkgs }:
let
inherit (nixpkgs) lib;
systems = lib.systems.flakeExposed;
eachDefaultSystem = f: builtins.foldl' lib.attrsets.recursiveUpdate { }
(map f systems);
in
eachDefaultSystem (system:
let
pkgs = nixpkgs.legacyPackages.${system};
inherit (maipkgs.legacyPackages.${system}) python;
python' = python.withPackages (ps: with ps; [ jax-triton ]);
in
{
devShells.${system}.default = pkgs.mkShell { packages = [ python' ]; };
}
);
}
and nix-direnv to add it as a gc root, but you can do this however you like. Note that it should not rebuild when you make this developer environment, as it was already built.
If you didn’t build with --no-link originally, make sure to remove the result symlink which will keep the package as a gc root. Then collect garbage with
nix-collect-garbage
sudo nix-collect-garbage
This will remove some things but not the python.withPackages environment (which, again, should be registered as a gc root). I have keep-outputs enabled if that matters.
Running
nix build github:stephen-huan/maipkgs#jax-triton --dry-run
prints nothing as expected, so the package is still cached, right?
Enter the REPL with
nix repl github:stephen-huan/maipkgs
and note that the derivation and outPath are the same as above (as expected).
nix-repl> packages.x86_64-linux.jax-triton
«derivation /nix/store/3zwncibwsls5diamvn4kf9f0nykn8zsa-python3.12-jax-triton-0.2.0.drv»
nix-repl> packages.x86_64-linux.jax-triton.outPath
"/nix/store/qiljpd83hlx0vv7cli75ng4nx9dn9h1b-python3.12-jax-triton-0.2.0"
But when building in the REPL, this triggers a rebuild, even though (1) the outPath exists in /nix/store and is nonempty and (2) nix build doesn’t rebuild!
nix-repl> :b packages.x86_64-linux.jax-triton
error: interrupted by the user
[0/9 built, 7/1/19 copied (0.3/18.3 MiB), 0.1/4.8 MiB DL] fetching openssl-3.3.2 from https://cache.nixos.org
However, building a Python environment with python.withPackages is cached.
nix-repl> :b legacyPackages.x86_64-linux.python.withPackages (ps: [ ps.jax-triton ])
This derivation produced the following outputs:
out -> /nix/store/fffw0rmmy10j5dsyk99j74f88gi72031-python3-3.12.7-env
What’s going on here? I thought if outPath exists, Nix should cache it. I suspect why python.withPackages works is you can convert a derivation to an output path, i.e.
nix-repl> "${packages.x86_64-linux.jax-triton}"
"/nix/store/qiljpd83hlx0vv7cli75ng4nx9dn9h1b-python3.12-jax-triton-0.2.0"
without actually building the derivation, and the way python.withPackages works is by just symlinking paths, so it only needs the paths (and not the derivations). But when a path is used in a derivation, Nix has to materialize the output paths, and this doesn’t explain why nix build works and not the REPL (trying :b packages.x86_64-linux.jax-triton.out and :b packages.x86_64-linux.jax-triton.dist still causes a rebuild).
The actual issue I’m trying to address is a flake like
{
description = "mwe";
inputs = {
nixpkgs.follows = "maipkgs/nixpkgs";
maipkgs.url = "github:stephen-huan/maipkgs";
};
outputs = { self, nixpkgs, maipkgs }:
let
inherit (nixpkgs) lib;
systems = lib.systems.flakeExposed;
eachDefaultSystem = f: builtins.foldl' lib.attrsets.recursiveUpdate { }
(map f systems);
in
eachDefaultSystem (system:
let
pkgs = nixpkgs.legacyPackages.${system};
inherit (maipkgs.legacyPackages.${system}) python;
# TODO: replace with pkgs.emptyDirectory
no-op = derivation {
inherit system;
name = "no-op";
builder = "/bin/sh";
args = [ "-c" "echo > $out" ];
};
python' = python.withPackages (ps: with ps; [
(kernels.override {
triton = triton-cpu;
torch = torch.override { triton = no-op; };
})
(torch.override { triton = no-op; })
]);
in
{
devShells.${system}.default = pkgs.mkShell { packages = [ python' ]; };
}
);
}
Here torch.override { triton = no-op; } is cached from a previous python.withPackages (but not the derivation), so when I use it in the context of kernels.override it triggers a rebuild (which seems to violate referential transparency).
What am I doing wrong? Ran nix store verify --all --repair, deleted my eval cache at ~/.cache/nix/eval-cache-v5 (which has been the source of weird flake behaviors in the past), but still running into these rebuilds.