Redundancy when pinning

Development
1

If I am prudent then I should precisely pin the version of nixpkgs that each of my projects uses so that if I return to it in a few months then it still builds correctly.

This works well for the first few times, it works well for the first few months but if over the course of a year or ten years I amass 100 or 1000 projects each with their own individual pin then the approach starts to break down.

In each folder I have a pin to a specific version of nixpkgs so in my store I will have potentially 1000 different copies of GHC for each one of my projects. Some of these might be significantly different versions of the compiler but mostly they are “duplicates” where an unimportant dependency has been changed which caused a rebuild. Each of these is a GB so I would require 1TB drive just to store all the versions of GHC for all my projects.

Over the course of 10 years there will only be 20 different versions of GHC so instead I might try to abstract away the pin and maintain a separate pin overlay which contains information about the pins I might want to use for the projects. These attributes are immutable but new ones are added over time. This somewhat solves the problem but now if someone else wants to build the project then I also have to distribute the pin layer.

pins = {
 ghc-8.2.2 = fetchFromGitHub { ... };
 ghc-8.4.3 = fetchFromGitHub { ... };
}

Possible solutions to this problem:

  1. Add a compilation step which inlines the pin definition into the default.nix file before distributing the package.
  2. Create a set of blessed pins which are distributed with nixpkgs itself. These are immutable and a new one is added once a month.

An advantage to (2) is that if many different people are sharing pins then the amount of rebuilding necessary for pinning projects will be reduced. At the moment if you build at a very old pin you have to rebuild the whole world usually.

Has anyone else considered this problem?

2

You have to find a balance between reproducibility and store path reuse. I think for most people there is no point in pinning an exact Nixpkgs version with fetchTarball, however, they may want to “pin” it to a certain channel. While it won’t give the exact same builds, one typically does get the same desired tools at a certain major/minor version.

1 Like
3

It seems that you are suggesting a different approach than the FAQ/Pinning Nixpkgs page. It would be good to update this page with your suggestion as it is the first result on google when searching for “pinning nixpkgs”.

4

I don’t see how it is different. If you aim for reproducibility, you need to pin it with fetchTarball/fetchGit and include a hash, just as how it is described on the page you link.

Your question was about how to reduce the amount of store paths, while still aiming for reproducibility. In that case I suggest to “pin” approximately i.e., fetch from a stable channel and don’t use a hash. This is something different, but yes, maybe it could be included there as well.

5

This works well for the first few times, it works well for the first few months but if over the course of a year or ten years I amass 100 or 1000 projects each with their own individual pin then the approach starts to break down.

This may be slightly off-topic, but you can use [1] (packaged as
nix-du) to get an overview of where you’re spending pins. And you can
then upgrade the lagging pins so that you minimize your overall
dependencies.

[1] GitHub - symphorien/nix-du: Visualise which gc-roots to delete to free some space in your nix store

1 Like