Why am I missing substitutes for some of my packages?

ianthehenry · March 10, 2021, 2:46am

After a nix-channel --update I no longer have substitutes for various packages, some of which I can’t build from source, so my nix-env -u is failing. This is my first time upgrading packages after installing Nix a few days ago, so I don’t really know how to diagnose or fix the issue myself.

I’m on nixpkgs-unstable, just using the default Nix install:

$ cat ~/.nix-channels
https://nixos.org/channels/nixpkgs-unstable nixpkgs

$ nix show-config | grep subst
builders-use-substitutes = false
extra-substituters =
substitute = true
substituters = https://cache.nixos.org/
trusted-substituters =

Here’s an example of a package for which no substitute is available and that I cannot build locally:

$ nix-env -qas python3
---  python3-3.10.0a5
--S  python3-3.6.13
--S  python3-3.6.13
--S  python3-3.7.10
--S  python3-3.7.10
-PS  python3-3.8.8
--S  python3-3.8.8
--S  python3-3.8.8
-PS  python3-3.8.8
--S  python3-3.9.2
--S  python3-3.9.2

$ nix-env -u python3
upgrading 'python3-3.8.8' to 'python3-3.10.0a5'
these derivations will be built:
  /nix/store/3xx5xjrlmyia3kji1lrj08s98awk55c0-python3-3.10.0a5.drv
building '/nix/store/3xx5xjrlmyia3kji1lrj08s98awk55c0-python3-3.10.0a5.drv'...
unpacking sources
unpacking source archive /nix/store/6qjykfvkri219wk15gjijkfjs9ci0588-Python-3.10.0a5.tar.xz
source root is Python-3.10.0a5
setting SOURCE_DATE_EPOCH to timestamp 1612298741 of file Python-3.10.0a5/Misc/NEWS
patching sources
applying patch /nix/store/345r2zz7pgiyk91j89qlf7mhs95jrv6f-no-ldconfig.patch
patching file Lib/ctypes/util.py
applying patch /nix/store/9kwzs3pplms8sijf55sdryypzvic4x1s-python-3.x-distutils-C++.patch
patching file Lib/_osx_support.py
patching file Lib/distutils/cygwinccompiler.py
Hunk #1 succeeded at 123 (offset -2 lines).
Hunk #2 succeeded at 140 (offset -2 lines).
Hunk #3 succeeded at 170 (offset -2 lines).
Hunk #4 succeeded at 310 (offset -2 lines).
patching file Lib/distutils/sysconfig.py
Hunk #1 FAILED at 170.
Hunk #2 FAILED at 187.
Hunk #3 succeeded at 231 (offset 23 lines).
2 out of 3 hunks FAILED -- saving rejects to file Lib/distutils/sysconfig.py.rej
patching file Lib/distutils/unixccompiler.py
Hunk #3 FAILED at 183.
1 out of 3 hunks FAILED -- saving rejects to file Lib/distutils/unixccompiler.py.rej
patching file Makefile.pre.in
Hunk #1 succeeded at 619 (offset 35 lines).
builder for '/nix/store/3xx5xjrlmyia3kji1lrj08s98awk55c0-python3-3.10.0a5.drv' failed with exit code 1
error: build of '/nix/store/3xx5xjrlmyia3kji1lrj08s98awk55c0-python3-3.10.0a5.drv' failed

I have one other package that needs upgrading (according to nix-env -u --dry-run) and also has no substitute:

$ nix-env -qas cabal-install-3.4.0.0
---  cabal-install-3.4.0.0

I don’t know if its build works, because I believe it would take several hours to complete.

I don’t really know what to do here: I thought that the channel lagged the nixpkgs repo by exactly long enough to populate the binary cache, but I have no idea “which” versions of which packages are put in the binary cache.

I tried to check the Hydra status:

$ cat ~/.nix-defexpr/channels/manifest.nix
[ { meta = { }; name = "nixpkgs-21.05pre274251.f5f6dc053b1"; out = { outPath = "/nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1"; }; outPath = "/nix/store/mi0xpwzl81c7dgpr09qd67knbc24xab5-nixpkgs-21.05pre274251.f5f6dc053b1"; outputs = [ "out" ]; system = "x86_64-darwin"; type = "derivation"; } ]%

And I did manage to find it, but it says that it succeeded:

https://hydra.nixos.org/build/138335539

And now I really have no idea what else to check to understand what’s wrong.

It occurs to me in writing this question that nix-env -u is trying to upgrade my python3 to something I probably don’t want. nixpkgs.python3 does not point to this alpha version, and I do still have a substitute for the version at nixpkgs.python3. But that’s not the version to which nix-env -u wants me to upgrade. This is making me think I should always just reinstall nixpkgs.whatever instead of upgrading. Should I just not ever use nix-env -u? What can I do to improve my experience here?

Note however that the newer version of cabal-install is the version at nixpkgs.cabal-install, so that one is a top-level package that just doesn’t have a substitute for some reason. I am confused.

NobbZ · March 10, 2021, 6:09am

nix-envs update behaviour is usually not what you expect…

Even if you installed your packages using -iA, nix-env doesn’t store the attribute used anywhere. When you ask it to update, it will just look at name attribute of a package, and then check the version. Among all found packages which share the same name, it will then install that one that has the highest version. And I am not even sure which version schema is used to determine and if it would aware of pre-releases.

It is easier to generally not use nix-env, but instead manage the installed software declaratively in the systems configuration (“/etc/nixos/configuration.nix”) or via home-manager.

edit

I’d also go as far as not managing languages compilers or runtimes through system or user configuration, but project specific shell.nix instead. That way you can easily switch the version of your python and cabal per project.

ianthehenry · March 10, 2021, 4:42pm

Thank you! That’s good advice as a new Nix user. But using nix-env -u is sort of tangential to this question – I’m curious about the expectations I should have around substitute availability and what my reaction should be in the case that something is missing. The question applies equally whether I’m using nix-env or a shell.nix (I am not using NixOS and am hesitant to try using home-manager until I understand a little more about vanilla Nix). I happened to encounter this in the course of a nix-env -u, but the same problem would apply if I just run nix-env -iA nixpkgs.cabal-install or referenced the derivation from my shell file or however else I tried to install it.

The Nix manual says:

By default, Nix will use the binary cache https://cache.nixos.org ; it contains binaries for most packages in Nixpkgs.

(Emphasis mine). Later in that paragraph it says:

The Nixpkgs channel is only updated after all binaries have been uploaded to the cache, so if you stick to the Nixpkgs channel (rather than using a Git checkout of the Nixpkgs tree), you will get binaries for most packages.

When I had first installed cabal-install, a substitute was available, and everything worked fine. Now there is a new cabal-install without a substitute, which leads me to believe that either:

cabal-install was removed from the list of blessed packages for which binary caches would be distributed.
Something is wrong, and a channel was unintentinally updated before all binary caches were built.
cache.nixos.org tried to build cabal-install, but the package is currently broken so it was unable to.

If (1), how can I find that out for myself? If this is how it works, is there some way to see which packages are “supposed to” be cached by the binary cache at https://cache.nixos.org?

If (2) or (3), who should I notify / how can I diagnose that this is the case?

ianthehenry · March 10, 2021, 5:13pm

I should probably add for context: a substitute is available for x86_64-linux, which makes me think that something is wrong with macOS builds (?) right now (?) or somehow packages are not supported equally (?) across different architectures. I still would like to know how I can learn more about the official caching infrastructure / policies.

$ nix-env -qas cabal-install --argstr system x86_64-linux --file https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz
--S  cabal-install-3.4.0.0

$ nix-env -qas cabal-install --argstr system x86_64-darwin --file https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz
---  cabal-install-3.4.0.0

austin · April 18, 2021, 8:12pm

Did you ever figure this out? Right now my laptop is trying to build Libreoffice from source and I’m guessing it will either fail or take long enough that the binary will finally be in the cache by then.

ianthehenry · April 18, 2021, 11:31pm

I didn’t, no. In fact this particular version of this particular package was never cached for my platform – although it was cached for other platforms, and a different one (different hash, same version number) did eventually show up in the cache.

This makes me think that waiting for something to appear in the cache isn’t sufficient, and something was broken with the package definition that forced a new one. Unfortunately I don’t really know where to look to try to blame this package, since it’s not a normal one but is rather assembled using all of the Nix Haskell machinery that I don’t really understand.

So you might need to update your channel in order to pick up the new Libreoffice? I still have no idea how the actual cache mechanism works.

I happen to notice a similar thing is happening here…

# latest nixpkgs
$ nix-env -qasA libreoffice --argstr system 'x86_64-linux' --file https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz
---  libreoffice-7.0.4.2

# not the latest nixpkgs, but the same "version" of the package
$ nix-env -qasA libreoffice --argstr system 'x86_64-linux' --file https://api.github.com/repos/NixOS/nixpkgs/tarball/f5f6dc053b1
unpacking 'https://api.github.com/repos/NixOS/nixpkgs/tarball/f5f6dc053b1'...
--S  libreoffice-7.0.4.2

I don’t know why though. It’s a workaround, but not a very satisfying one.