I suspect some of my user overlays are not applied in nix-env --upgrade

Help
1

I’m using overlays both in /etc/nixos/configuration.nix and both in my ~/.config/nixpkgs/overlays. I have the suspicion that the user defined overlays are not applied when I run nix-channel --update && nix-env --upgrade`. Here is why:

When I run nixos-rebuild --upgrade, from time to time I can see that a certain derivation I patch in a system overlay is being rebuilt - I think it’s because one of it’s dependency got updated.

For reference, here it is. 
      # Patches that fix a bug with multiple simultaneout connections generating
      # duplicate entries in resolv.conf
      # https://lists.01.org/pipermail/connman/2019-April/023424.html
      (
        self: super: {
          connman = super.connman.overrideAttrs (oldAttrs: rec {
            patches = [
              ./connman-resolv.conf-duplicates.1.patch
              ./connman-resolv.conf-duplicates.2.patch
            ];
          });
        }
      )

In comparison, I have many more overlays for my user environment, and some of them include patches and I never got to see them rebuilt - ever since I got wrote them and tested them with nix-env -iA and nix-build -A.

Here's an example overlay. 
self: super:

{
  octave = super.octave.overrideAttrs (oldAttrs: rec {
    buildInputs = with super; [ gfortran readline ncurses perl flex texinfo qhull
      graphicsmagick pcre pkgconfig fltk zlib curl openblas libsndfile fftw
      fftwSinglePrec portaudio qrupdate arpack libwebp
      # ----------
      # my removal
      # qt
      # qscintilla
      # ----------
      #
      # This is commented along with --enable-jit because of:
      # - https://savannah.gnu.org/bugs/?func=detailitem&item_id=55469
      # - https://savannah.gnu.org/bugs/?func=detailitem&item_id=55492
      # - https://savannah.gnu.org/bugs/?func=detailitem&item_id=46192
      # llvm_8
      ghostscript
      hdf5
      glpk
      suitesparse
      openjdk11
      gnuplot
      python37
      libGLU
      xorg.libX11
      # ----------
      # My Addition
      sundials
      # ----------
      ];
    configureFlags =
      [ "--enable-readline"
        "--enable-dl"
        "--with-blas=openblas"
        "--with-lapack=openblas"
        "--enable-64"
        # "--enable-jit"
      ];
  });
}

I have installed it via nix-env -iA octave and the build took a few hours but it worked eventually. What makes me suspicious is the fact I can’t recall any other time it got the build during a nix-env --upgrade, and I wrote this overlay back in August. Is this normal? I can hardly believe it maybe got rebuild and I didn’t noticed it - it takes hours for the build to complete…

Moreover, what’s suspicious to me is the fact that if I run nix-env -iA nixpkgs.octave now, I can see it start building.

2

My observation is that nix-env -u only applies if the version of a derivation has changed. I have some package sets defined with mkEnv where this problems occurs. An update only happens if the name attribute is changed from myenv-1 to myenv-2 for example. Otherwise I need to remove and re-install it. Could this apply to your situation too?

3

That’s what I was thinking. I’m not sure how to test this since for example my octave overlay needs the same version attribute in order to not try to fetch sources which don’t exist. Isn’t it intriguing that nixos-rebuild knows when to rebuild a derivation, just like hydra but nix-env doesn’t?

4

nixos-rebuild rebuilds everything, it just reuses existing build results if they exist. It ignores versions and only cares about .drv differences.

nix-env -u on the other hand doesn’t even know what the source derivation was for any given installed package, it only knows the name/version, so it looks for packages with the same name and a newer version (and has heuristics for dealing with multiple packages that share a name). It then installs just the packages it finds, and leaves everything else untouched. Any package with the same name/version combo is also ignored because it only picks up on newer versions.

Ultimately this comes down to nix-env being imperative instead of declarative. If you want a declarative user package setup, you can use home-manager or you can hack it together with your own scripts. Personally I use this script which is based on one by @LnL7. With that script I just run nix-rebuild and it updates my environment to match a declarative set of packages configured in an overlay.

4 Likes
5

I see. Thanks for the suggestions. I was considering moving over to such a system as home-manager but I’m surprised to learn that nix-env doesn’t know anything about attribute names - I expected it to have a local database of some sort that will link current installed derivations to the original attributes requested via nix-env -iA. Maybe I’ll talk about it with the developers at GitHub - NixOS/nix: Nix, the purely functional package manager …