Why does upgrading a fully up-to-date system create garbage?

Help
1

(Full disclosure: I’m a nixos noobie. I’ve read the nix pills but am still trying to get a handle on how things work.)

I installed nix package manager on my Devuan system and am trying to understand why upgrading always creates garbage, even when the upgrade is performed on a fully up-to-date system.

See below (things to note: I start at generation 2 and a garbage collection finds no garbage. Then I perform an upgrade operation that does nothing because the system is already fully up-to-date. After the upgrade I’m still at generation 2):

$ nix-env --list-generations
   2   2020-08-06 14:42:32   (current)

$ nix-collect-garbage -d
removing old generations of profile /nix/var/nix/profiles/per-user/bruno/channels
removing old generations of profile /nix/var/nix/profiles/per-user/bruno/profile
finding garbage collector roots...
deleting garbage...
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves -0.00 MiB
0 store paths deleted, 0.00 MiB freed

$ nix-channel --update; nix-env --upgrade
unpacking channels...

$ nix-env --list-generations
   2   2020-08-06 14:42:32   (current)

But after the above, nix-collect-garbage -d finds a ton of garbage:

$ nix-collect-garbage -d
removing old generations of profile /nix/var/nix/profiles/per-user/bruno/channels
removing old generations of profile /nix/var/nix/profiles/per-user/bruno/profile
finding garbage collector roots...
deleting garbage...
deleting '/nix/store/1yic9l55n84mklzlf3pqrg9jqzxkz7y6-openmpi-4.0.4.drv'
deleting '/nix/store/8xb9a32cgl1xiq77ns0vl14jzdzmcnsc-ucx-1.8.1.drv'
deleting '/nix/store/062pyx3824q3bkmi15001121mn4cprvj-rdma-core-30.0.drv'
deleting '/nix/store/1n77zjzs790lw1fxwzl82lmjb472mjl8-pandoc-2.9.2.1.drv'
deleting '/nix/store/qda5wfi6hxb615bzsqm995xbdk8f73b7-texmath-0.12.0.2.drv'
deleting '/nix/store/6p1ylm7pjndvva2nsxg1lz8hhz6bzw9s-pandoc-types-1.20.drv'
deleting '/nix/store/00jqykjj439g8sixpq1c6q4shli9i7h8-string-qq-0.0.4.drv'
deleting '/nix/store/v19drkx1p50gvsspyd34npbmppj6nkpm-http-client-tls-0.3.5.3.drv'
deleting '/nix/store/08lpjf8a1n5a7652d2fz9hqhzll5f2y0-connection-0.3.1.drv'
deleting '/nix/store/6xhbifrlzr7gba4khrq1pav3382z2zx0-x509-system-1.6.6.drv'
---snip---
deleting '/nix/store/p46prhgmv7ibjh9igfkbc6zfxbbi6sk5-dont-hardcode-cc.patch'
deleting '/nix/store/gm1vihrf3d8hks2fgjfgfyn5wm2rs49a-locales-builder.sh'
deleting '/nix/store/shpjchz7kli6y3xbdbab34q009v5hl4h-Use-stdlib-uint-instead-of-u_int.patch'
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves -0.00 MiB
2304 store paths deleted, 7.90 MiB freed

Can someone please help me understand how an upgrade operation on a fully up-to-date system creates garbage?

2

I did a little more investigating. The nix-env --upgrade brings in several derivations that don’t seem to be needed by any of my manually-installed derivations (right now the only manually-installed derivations I have are nix-2.3.7 and hello-2.10).

openmpi and ucx, for example, are not present in /nix/store at first, but nix-env --upgrade drags them in.

So my more specific question is: Why does nix-env --upgrade bring in derivations that are not needed?

3

It seems this is a known bug: nix-env -u pulling in many unexpected files · Issue #1991 · NixOS/nix · GitHub

4

This will upgrade existing packages installed with nix-env. Nix derivations know what their entire dependency tree is, and can “move” their dependency tree and gcroots over. But this probably frees some previously used derivations from the previous version of the packages. nix-collect-garbage will then notice that the gcroots pointing to the old derivation has been removed, and is free to clean up those derivations. The other thing which may be going on is that nix-env is also deleting some stale gcroots

I’m not sure this is a “bug”, but rather hard to determine behavior.

You can look at /nix/var/nix/gcroots/ to see what nix intending to keep onto.

You can also remove stale links by doing sudo find /nix/var/nix/gcroots/ -xtype l -delete and then clean up the garbage with nix-collect-garbage. (This will likely cleanup old profiles as well, so only use it when you intend to do a nix-collect-garbage -d).

5

if you can find the store path, you can do nix-store --query --referrers <nix-path> to find all other nix derivations referring to it

This should work if the derivations were all built with the same nixpkgs channel:

nix-store --query --referrers $(nix eval --raw nixpkgs.openmpi.outPath)
1 Like
6

Hi, Jon. Thanks for your input. I did an upgrade an it brought in all the “garbage” derivations as usual. Let’s pick on openmpi:

$ nix-store --query --referrers /nix/store/1yic9l55n84mklzlf3pqrg9jqzxkz7y6-openmpi-4.0.4.drv
$
$ nix-channel --list
nixpkgs https://nixos.org/channels/nixpkgs-unstable # I only have this one channel
$ nix-store --query --referrers $(nix eval --raw nixpkgs.openmpi.outPath)
$

I think this confirms that the derivation has no reverse dependencies (right?). It remains a mystery why openmpi (among 2000+ others) is being installed.

I’d like to pursue this further if you can help me.

7

That didn’t help:

$ sudo find /nix/var/nix/gcroots/ -xtype l -delete
$ nix-collect-garbage -d
$ nix-channel --update; nix-env --upgrade
$ nix-collect-garbage -d
---same output as usual here, with 2000+ derivations being deleted---
8

nix commands also have the behavior of talking to the nix-daemon, so you still need to see what ‘root’ user has as a nix-channel, and it may be different

$ sudo nix-channel --list
9

root user does not seem to have a nix channel:

$ . $HOME/.nix-profile/etc/profile.d/nix.sh
$ sudo nix-channel --list
$

P.S. I don’t think there is a nix daemon running on my system.

10

oh sorry, you mentioned you’re not using NixOS. You’re correct, your system wouldn’t have a nix-daemon.

I’m not sure why there’s 2000+ derivations that are spaming your system. :(. But I also don’t really use nix-env.

Someone else may know better.

11

Hard to say what’s going on without seeing your configuration, but something similar is happening to me as well, albeit not at this scale. When I collect garbage and then run nixos-rebuild switch --upgrade, Nix downloads about 10 or so derivations. This is due to the fact that in my /etc/nixos/configuration.nix there is a

{
  imports = [ (builtins.fetchTarball { ... }) ];
}

and to evaluate the import, Nix first has to fetch and unpack the tarball, for which it needs some stuff like curl, etc.

Nevertheless, pulling in 2000+ derivations when updating means that Nix evaluates some huge subtree. Maybe check whether there are any potential candidates in your overlays.

12

Hi, hmenke. I’ll be happy to share any of my nix configuration that would be helpful (just let me know what you’d like to see), but honestly it’s just the vanilla configuration that was created when I installed the nix package manager with this command:

$ curl -L https://nixos.org/nix/install | sh

I did not change any of the default configuration. These are the nix-related directories I’ve found on my system: /nix, ~/.nix-defexpr, and ~/.nix-profile. I don’t have a /etc/nixos directory (presumably because I have only the package manager and not the whole OS).

13

Maybe you could post the output of

nix-env --query

and the content of ~/.config/nixpkgs/config.nix and any overlays you might have in this directory.

As an aside, you have to hit the “reply” button on my post, otherwise I don’t get a notification.

14

Hi, hmenke. Sorry for the oversight. I think I responded properly this time.

Here you go:

$ nix-env --query
hello-2.10
nix-2.3.7

I don’t have ~/.config/nixpkgs directory. See my prior post for the nix-related directories that I do have.

15

I can confirm the issue after setting up a similar Nix installation in a systemd-nspawn container based on the Devuan Docker image. However, note that all of those files that are created are only .drv files, so nothing is actually being built.

The same actually happens on my mutli-user Nix installation on top of Arch Linux

$ sudo nix-collect-garbage -d
$ sudo nix-channel --update
unpacking channels...
created 1 symlinks in user environment
$ ls -1q /nix/store | wc -l
9990
$ nix-env --upgrade
$ ls -1q /nix/store | wc -l
11316

and it even happens on a NixOS machine (although normally I never use nix-env on NixOS in the first place)

$ sudo nix-collect-garbage -d
$ sudo nix-channel --update
unpacking channels...
created 1 symlinks in user environment
$ ls -1q /nix/store | wc -l
2707
$ nix-env --upgrade
building '/nix/store/xsk9wjy0dpp7w7d3r3bkzxzaklqdzq6q-user-environment.drv'...
created 0 symlinks in user environment
$ ls -1q /nix/store | wc -l
4140
16

Wow, hmenke, thanks for taking the time to reproduce the issue. Good to know the fault is not my system or the way I’m using the nix package manager.

Why does the upgrade operation download over two thousand files (even if they are only .drv files) that are not actually needed? Maybe the install script is not creating a lean enough environment?

Is there a way to perform a cleaner upgrade–either by changing something in the default configuration or by adding some options to the upgrade command?

17

Oh no, nothing is actually being downloaded. All those .drv files are just results of instantiated derivations. Let’s take a simple Nix expression for an example that I saved as test.nix:

with import <nixpkgs> {};

mkShell {
  buildInputs = [ hello ];
}

This will create a shell environment where the hello package is available. When we instantiate the derivation we will get a derivation output (the .drv file).

$ nix-instantiate test.nix
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/2g2i5d6zqa8vpqdr1yajyfqxg0qnqvn0-nix-shell.drv

We could take a look at the derivation by either using cat on this path, or we can convert it to JSON using nix show-derivation and pipe the result to jq for pretty-printing:

$ nix show-derivation /nix/store/2g2i5d6zqa8vpqdr1yajyfqxg0qnqvn0-nix-shell.drv | jq

This .drv file lists all the inputs and outputs of the Nix expression in test.nix (including some special defaults that are pulled in by mkShell). Only once the derivation is actually realised, Nix will start downloading stuff:

$ nix-store --realise /nix/store/2g2i5d6zqa8vpqdr1yajyfqxg0qnqvn0-nix-shell.drv
these derivations will be built:
  /nix/store/2g2i5d6zqa8vpqdr1yajyfqxg0qnqvn0-nix-shell.drv
these paths will be fetched (0.04 MiB download, 0.20 MiB unpacked):
  /nix/store/9pqfirjppd91mzhkgh8xnn66iwh53zk2-hello-2.10
copying path '/nix/store/9pqfirjppd91mzhkgh8xnn66iwh53zk2-hello-2.10' from 'https://cache.nixos.org'...
building '/nix/store/2g2i5d6zqa8vpqdr1yajyfqxg0qnqvn0-nix-shell.drv'...
nobuildPhase

This derivation is not meant to be built, aborting

builder for '/nix/store/2g2i5d6zqa8vpqdr1yajyfqxg0qnqvn0-nix-shell.drv' failed with exit code 1
error: build of '/nix/store/2g2i5d6zqa8vpqdr1yajyfqxg0qnqvn0-nix-shell.drv' failed

In this case it fails because mkShell does not have build outputs, since it’s a shell environment, but you can see that it started downloading only once I specifically asked for realisation.

I don’t think it is possible when using nix-env. You could maybe think about switching to a declarative package management style, like home-manager, which might not have this problem. At least running nixos-rebuild on NixOS on an up-to-date system does not have this problem:

$ sudo nix-collect-garbage -d
[...]
$ ls -1q /nix/store/ | wc -l
2739
$ sudo nixos-rebuild switch --upgrade
unpacking channels...
building Nix...
building the system configuration...
updating GRUB 2 menu...
activating the configuration...
setting up /etc...
reloading user units for user...
setting up tmpfiles
$ ls -1q /nix/store/ | wc -l
2740

One derivation is created which is for the new generation that was added by the upgrade but that’s it.

But then again, these .drv files are actually pretty harmless. The don’t take considerable disk space and nothing has to be downloaded.

1 Like
18

hmenke, thank you so much for your knowledgeable input. So it seems the issue is harmless, is not using up bandwidth, and is apparently related to using nix-env rather than full-blown NixOS. Fair enough. I can live with that :slight_smile:

Eventually I would still like to understand why this happens and why these particular .drv files (and not others) are the ones that show up unsolicited. I’ll explore these questions myself when I have a deeper understanding of how nix-env works, will post here if I have any revelations.

Thanks again for all your help. Happy hacking!

19

Well, if you manage to understand why that happens that would be the fix for the issue you linked previously. I suspect that this is highly non-trivial, though.

https://github.com/NixOS/nix/issues/1991