Coupling between transitive dependencies

Help
1

Hey there - first off, massive thanks for all the work you folks have been doing!

I’ve been using nix for a short while and I’m now moving to nixOS which, for the most part, has been a great experience! A key issue I’ve had while using nix however has been package versioning.

I’m not sure how to best describe this so I’ll just provide a comparison to how package versions are managed in nodejs.

Node package managment (npm)

A list of dependencies are declared as follows (psudeosyntax)

{
  "package1": "latest",
  "package2": "latest",
  "package3": "latest"
}

In the above example, because the dependency version is not pinned, the latest version from the registry will be used. This resolved “latest” version is then pinned to a lockfile and can only change once the lockfile is rebuilt.

If a user wants to explicitly use a particular version of a package, they can specify the version of the package they want - this will have no impact on other packages.

{
  "package1": "latest",
  "package2": "latest",
  "package3": "1.0.0"
}

Here’s the important bit - no matter how many new releases of a package are pushed to the registry, the previous releases are still available.

All packages have access to all versions of current and previous packages. If we consider transitive dependencies, updates of packages happen in isolation and do not introduce restrictions on their peers. Packages are not coupled to their peers - only their own version and dependencies

{
 "package1": "latest"
}

// package1 trans deps
{
 "package3": "1.0.0" 
}

// package1 trans deps
{
 "package3": "2.0.0" 
}

In this sense, packages are very much contained.

Note: Sometimes some magic takes place to deduplicate transitive dependencies for code-reduction reasons but that’s outside the scope of this discussion.

Nix package management

A list of dependencies are declared as follows (psudeosyntax)

pkgs.package1
pkgs.package2

In this case, versioning is pinned to the version of the registry, not the version of the package. This is where the problems start.

Updating package1 requires an update of the registry (i.e. nixpkgs) which will in turn update package2 and package3. This means, while nix packages are designed to be used in complete isolation, they are inherently coupled to a monolithic registry.

Overlays can solve this problem for the most part (e.g. pulling different releases of registries and overlaying the top-level package) but in the case where different versions of transitive dependencies are required - you’re kind of stuck.

I came across this issue when trying to use a previous version of Gnome3 packages, while also using a recent registry release.

error: builder for '/nix/store/yf1j7wq8jq6kdfw32x9gq5whxg1mgqpv-yelp-tools-40.0.drv' failed with exit code 1;
       last 10 log lines:
       > Program python3 found: YES (/nix/store/66fbv9mmx1j4hrn9y06kcp73c3yb196r-python3-3.8.9/bin/python3)
       > Found pkg-config: /nix/store/b8mzg1b1s3lh2zvmz1yichqprzhi0f2d-pkg-config-wrapper-0.29.2/bin/pkg-config (0.29.2)
       > Dependency yelp-xsl found: NO found 3.36.0 but need: '>= 3.38.0'
       > Did not find CMake 'cmake'
       > Found CMake: NO
       > Run-time dependency yelp-xsl found: NO
       >
       > meson.build:30:0: ERROR: Invalid version of dependency, need 'yelp-xsl' ['>= 3.38.0'] found '3.36.0'.
       >
       > A full log can be found at /build/yelp-tools-40.0/build/meson-logs/meson-log.txt
       For full logs, run 'nix log /nix/store/yf1j7wq8jq6kdfw32x9gq5whxg1mgqpv-yelp-tools-40.0.drv'.
error: 1 dependencies of derivation '/nix/store/qc5wv1fkkmyja9cvlj3axy5khwp74crm-gnome-user-docs-40.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/8cfi7c29p519rc9ygpwgf7imkgfzv7sz-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/z146mqk8rxqvl4qpb4zyp7v317r4vgs7-nixos-system-nixos-21.11.20210603.95222ea.drv' failed to build

I was now stuck with the options of:

  • Rolling back the whole registry (and all system package versions)
  • Overlaying the gnome packages, and every single one of it’s transitive dependencies (which would impact all packages)

Solutions

There’s a good chance I’m may be missing something but as far as I can tell, there isn’t really a solution for version management currently.

While the NPM approach isn’t perfect, I think there are some key takeaways that nix could massively benefit from:

  • New releases to the registry (or of the registry in nix’s case) don’t need to break previous releases
  • Packages can be updated independently (not a monolithic system)
  • Redundancy when working with transitive dependencies is a good thing

The most primitive way I can see this working in nix would be by keeping release versions of packages. In the case where an unpinned version is used, an update to the registry would then cause the unpinned version to be updated.

{
  mypackage = mypackage@1; # manually updated on major releases
  
  mypackage@1 = mypackage@1.3; # manually updated on minor releases

  mypackage@1.3 = mypackage@1.3.0; # manually updated on patch releases
  mypackage@1.3.0 = pkgs.stdenv.stdenv.mkDerivation # ...

  mypackage@1.2 = mypackage@1.2.1; # manually updated on patch releases
  mypackage@1.2.1 = pkgs.stdenv.stdenv.mkDerivation # ...
  mypackage@1.2.0 = pkgs.stdenv.stdenv.mkDerivation # ...
}

In this example, the following dependency would work indefinitely, regardless of how many future releases of it’s dependent packages (or the registry) are published:

{
  somepackage = somepackage@1;
  somepackage@1 = somepackage@1.1;
  somepackage@1.1 = somepackage@1.1.0;
  somepackage@1.1.0 = (pkgs.stdenv.mkDerivation {
    buildInputs = [mypackage@1];
  });
}

It would also automatically have transitive dependencies updated as new registry versions are pulled.

I’ve spent way too long on this so i’m going to stop rambling but it would be great to get some thoughts on this!

2

In general, the problem you’re describing is solved by “pinning nixpkgs”. This is a good article on the topic: Towards reproducibility: pinning Nixpkgs — nix.dev documentation but there are many out there.

I myself make use of niv (which uses the same declaration-and-lockfile pattern as npm) within my home manager setup in order to prevent the mass upgrade of all my packages when the channels update.

Hope this helps!

roni

3

Hey Roni thanks for the response.

Funnily enough, I’m using flakes so the version of nixpkgs I’m using is always pinned. I agree that pinning can work around the mutable nature of packages in the registry for root dependencies, but that approach still doesn’t solve the issue of transitive dependencies.

As far as I can tell, any dependencies required by a overriden package will still be from the “root” nixpkgs instance (i.e. nixpkgs.lib.nixosSystem). Hence we end up with that lockout condition where two unrelated packages that depend on different versions of the same package cannot both coexist.

// package1 transitive deps
{
 "package3": "1.0.0" 
}

// package2 transitive deps
{
 "package3": "2.0.0"  
}

In the above example, where nixpkgs doesn’t version “package3” - users have to decide between having package1 or package2 in their system. Both work in isolation but the transitive dependency binds them to a registry release.

So for the most part, mixing and matching registries will likely result in breakages if derivative packages have undergone major version releases. And there doesn’t seem to be an (easy) way around this.

This wouldn’t be the case if versioning was tied to packages (as traditionally is the case) rather than registry versions.

4

I hope someone else will chime in with better technical details, because my own understanding of this remains quite abstract.

But, I know that what you describe is supposed to be possible, at least in some cases: in particular, if different versions of package3 are needed by different packages (even if one of those packages is simply a dependency of the second package), then the magic of static linking and rpaths will allow for both package versions to coexist without crashing into each other.

The situation is different if package3 provides executables that go into your user profile. But there I suspect you’d have package builds behaving badly if they depend upon stuff being placed in the user profile as a side effect of their own build or runtime requirements. And in any case, that situation is unsupported by package managers in general (without some sort of on-the-fly profile switching or something, but I’m not aware of a feature like that in any package manager).

Sorry–I don’t think any of this is very helpful. As I mentioned above, I hope someone much more knowledgeable shows up to provide better insight.

5

I think you’re being slightly imprecise with language here (nixpkgs itself is never really updated so I’m not sure what you mean by that), but changing a transitive dependency will always require that derivation is completely rebuilt, and there’s not really a way around this.

6

Hi Andy,

Here’s an example that might help. Suppose I’d like to install a recent version of vim with python support, but for some reason I also need an old version of Python outside of vim. The following flake will install vim from the lastest version of nixpkgs, but install Python 3.5 from 2016 release of nixpkgs. Crucially, vim’s python dependency will be satisfied by the version of Python in the recent nixpkgs, and not Python 3.5.

{

  inputs.nixpkgs.url = github:nixos/nixpkgs;

  inputs.nixpkgs-old = {
    url = github:nixos/nixpkgs/release-16.09;
    flake = false;
  };

  outputs = { self, nixpkgs, nixpkgs-old }: 
  let
    pkgs = import nixpkgs {
      system = "x86_64-linux";
    };

    oldpkgs = import nixpkgs-old {
      system = "x86_64-linux";
    };

  in
  {
    defaultPackage.x86_64-linux = pkgs.buildEnv {
      name = "test";
      paths = [
        # install vim from 2021
        pkgs.vim_configurable

        # but python from 2016
        oldpkgs.python3
      ];
    };
  };

}

Running python3 import sys; print(sys.version) inside of the built vim, I see that it has Python 3.8.9. But if I run the built python from 2016, it is version 3.5.2. Vim’s python version is the one in the recent version of nixpkgs, not the python version in the 2016 release.

When you use an overlay of nixpkgs to change a package’s version, all of its dependencies will be injected from that version of nixpkgs. This may be OK for small version changes, but if you need a drastically older version it makes sense to use an entirely different version of nixpkgs.

7

I hear you - no matter how hard I try to put this together, it’s a hard one to communicate!

I’ve put together a repo with some examples of dependency management and cases where (IMHO) it falls apart - GitHub - andyrichardson/nix-dep-managment: Some examples of nix dep management

8

So you’re saying when a package is overlayed, all transitive dependencies will be sourced from the same nixpkgs release as the package (i.e. not the one declared in the root of the flake)?

I wasn’t seeing this when doing this myself but there’s a good chance I was doing something wrong in that case.

I’m still not sure this is the best way to go about things - imho having to thing about both package and registry versioning adds a lot of unnecessary complexity.

9

The thing to realize is that nixpkgs is not a package registry but a repository. The difference is that repositories are usually manually curated and geared towards a certain purpose (e.g. set of software packages that are reasonably up to date, secure and compatible together). Registries, on the other hand, allow people to publish packages and do not care about compatibility.

Package managers using registries can ensure some degree of compatibility using dependency constraints but that brings its own problems:

  • proliferation of different versions of a single package in the closure (when dependency bounds are non-overlapping)
  • not all environments support multiple versions of a single dependency (e.g. shared libraries tend to get symbol clashes)
  • you get combinatorial explosion of package closures so caching build products from compiled languages is no longer feasible
  • older versions often contain accrue security vulnerabilities so they either need to be marked as such or manually updated, adding maintenance costs

So really, this is just trading coverage for manageability and maintainability.

2 Likes
10

Thanks for the explanation - this makes sense!

I guess the main reason for my confusion is because the “compatible together” side of things is the bit that I felt was unnecessary for NixOS (with some low-level exceptions).

Striving for system-wide compatibility on a system that is designed to avoid system-wide installation of packages feels like a contradiction.

In terms of maintenance, I would have guessed that removing the “system wide compatibility” aspect of things would actually remove the maintenance burden.

Untitled
Untitled1292×478 11.4 KB

11

Depends on the overlay.

Overlays are intended to modify Nixpkgs at run time in such a way that when your overlay replaces an attribute, all packages that depend on the attribute in the overlaid package set will see the attribute from the overlay instead the one from the overlaid package (actually, from the composition of overlays when there is a stack of them). Basically, it is hooking into a fix point.

For this to work, overlays need to be pure functions (do not depend on anything else than its final and prev arguments). And since such overlays can only get packages from the previous layer or from the final package set, they will use dependencies from there as well.

But since Nix is a full-blown programming language, the purity is not enforced and you can have impure values in overlays. Those impure values do not have to depend on final or prev arguments at all and so they can retain their own closure.

1 Like
12

Yes, global horizontal compatibility (across dependency trees of different applications) is not an issue for us since Nix store can contain multiple mutually incompatible versions of the same software. But the compatibility may be still an issue for linking shared libraries (symbol clashes) within a single dependency tree.

And even if we fixed that by patching the linker to do some kind of symbol namespacing, there is still the issue of vertical compatibility (which version of a dependency should a package choose). And I would say that is the harder problem – see the problems listed above.

13

So you’re saying when a package is overlayed, all transitive dependencies will be sourced from the same nixpkgs release as the package (i.e. not the one declared in the root of the flake)?

Just to make sure we’re avoiding a potential point of confusion: my code example isn’t using overlays. Had I overlaid Python from 2016 onto the more recent nixpkgs, then vim would be built with the old python version.