Nix flake to aggregate and concurrently update some dependencies

Help
1

Hi everyone!

I’m using flakes for building various software packages authored by different people in various languages in CI. My flake.lock files are checked in and my build’s are reproducible. I’m really happy with what I have so far.

One of the remaining pain points is that a lot of projects depend on the same outside inputs (like nixpkgs), but if those inputs are independently updated using

nix flake update --recreate-lock-file

they end up depending on different versions of the same thing which is inefficient in various ways*.

My idea to solve this would be creating an aggregate flake which does nothing but aggregate some external dependencies centrally (mostly nixpkgs, but potentially also some others) and re-export them in a specific version determined by its own lock file.

Instead of depending on these external dependencies directly my other projects could then depend on the versions provided by the aggregate flake, which gives my more control over the update cycle. I could first update the aggregate and then my other projects individually, to get the same version everywhere.

I wanted to try this but I’m having a hard time figuring out what the aggregator flake would have to look like.

Can anyone give me some input on how that aggregate flake idea could be implemented in a flake?

(* My issue is not only with downloading and storing slightly different versions of the same software. I also wrote a bot that automatically creates/updates pull requests for outdated flake inputs. Right now changes to nixpkgs lead to a lot of noise in those pull requests, which I would like to concentrate in the aggregate flake’s repo by updating it at a lower frequency, like weekly or every two weeks so that in downstream projects those updates only show up with that frequency.)

2

you can use --override-input nixpkgs XXXX to set your version of the input, you can also rewrite the dependencies via

inputs = {
    unstable.url = "github:nixos/nixpkgs/nixos-unstable";

hm = {
      url = "github:nix-community/home-manager";
      inputs.nixpkgs.follows = "unstable";
    };
    };
1 Like
3

Thanks for the suggestions @teto .

I think both of those solutions could solve some of my issues, but they are not expressive enough to give me what I actually want in terms of workflow and expressiveness directly inside the flake file.

Instead of

inputs = {
    unstable.url = "github:nixos/nixpkgs/nixos-unstable";

    hm = {
        url = "github:nix-community/home-manager";
        inputs.nixpkgs.follows = "unstable";
    };
};

I would want to be able to write something like

inputs = {
    unstable = hm.nixpkgs;
    hm.url = "github:nix-community/home-manager";
};

or in verbal terms for this specific example

I want hm to pick my specific version of nixpkgs, and when I update my dependencies I want the latest one that hm picked in my lock file.

This is more powerful because it works for direct dependencies, so it does not involve re-targeting some pre-existing dependency later on.

hm does not make sense as an example for why I would want this, for that I would replace hm with the aggregate I mentioned in my original post. It still shows nicely what I think is not possible with those solutions.

4

It looks like what I’m asking can be done by omitting nixpkgs from the input like this

inputs = {
    hm.url = "github:nix-community/home-manager";
};

and then refering to it inside the outputs block as hm.inputs.nixpkgs. :man_facepalming:

The downside of that being that it turns into an transitive dependency even though I would prefer to still have it listed as a direct dependency in the inputs. I just wanted nix to look at another dependency for resolving which specific version to include in the lockfile and have the flake file be as explicit as possible about what I actually depend on and in which version.

5

It seems like you want a parent flake to use/inherit an input as defined by a child flake? Can you have the child flake expose the input as an output, and then just explicitly reference that output? Something like this: (untested, just an idea)

github/cole/aggregator/flake.nix:

inputs = {
  nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  hm.url = "github:nix-community/home-manager/master";
  hm.nixpkgs.follows = "nixpkgs";
  n-w = "github:colemickens/nixpkgs-wayland/master";
  n-w.nixpkgs.follows = "nixpkgs";
}
outputs = inputs: 
  {
    aggregatedInputs = inputs;
  }

and then a consumer that would use the aggregated inputs:
github/cole/consumer/flake.nix:

inputs = {
  aggregator.url = "github:cole/aggregator/main";
}
outputs = inputs_: 
  let
    inputs = inputs_.aggregatedInputs;
  in {
    # do something with the aggregated inputs:
    #mysystem = inputs.nixos.mkSystem {...}
  }
6

I do something like that (closed repository, sorry). I have a syntaxdot flake which has a nixpkgs and utils inputs. Then I use these inputs in another flake syntaxdot-models (the rec might be superfluous?):

{
  description = "SyntaxDot models";

  inputs = rec {
    nixpkgs.follows = "syntaxdot/nixpkgs";
    syntaxdot.url = "github:tensordot/syntaxdot";
    utils.follows = "syntaxdot/utils";
  };

  outputs = { self, nixpkgs, syntaxdot, utils }: {
    # ...
  };
}

(I don’t think this was documented anywhere, I just got here by trial and error, reading the error messages, and reading Nix source code. These stringly-typed inputs are not really helpful and quite ugly.)

11 Likes
7

There needs to be a :exploding_head: post-react button in Discourse!

1 Like
8

Thank’s everyone. :exploding_head: from me as well. With @danieldk’s suggestion I can express exactly what I wanted in the downstream project (The rec in his reply is indeed superfluous.):

  inputs = { 
    flock-of-flakes.url = "/home/mschwaig/flakes/flock-of-flakes";
    naersk = {
      url = "github:nmattia/naersk/master";
      inputs.nixpkgs.follows = "nixpkgs";
      inputs.utils.follows = "utils";
    };
    nixpkgs.follows = "flock-of-flakes/nixpkgs";
    utils.follows = "flock-of-flakes/utils";
  };

And the upstream aggregate (named flock-of-flakes and only a local path for now) looks like this:

{
  description = "A collection of flakes we depend on and want to use the same version of";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = inputs : {};
}

At least thats the best way that I could figure out to make a collection of flakes. Note that this flake itself can’t be built because it lacks a defaultPackage, but it doesn’t have to be a buildable thing for this use case. You can run nix flake check and nix flake update on it and it produces a working a flake.lock file to check in.

If we look at the output of the nix flake list-inputs command we can see that the .follows are working as expected.

mschwaig@hydralisk ~/f/proj_c (main)> nix flake list-inputs
warning: Git tree '/home/mschwaig/flakes/proj_c' is dirty
git+file:///home/mschwaig/flakes/proj_c
├───flock-of-flakes: path:/home/mschwaig/flakes/flock-of-flakes?narHash=sha256-Ky2wGWhdRKIDDwIOQXGD6UFSKThK6IsmNFR5HLEq8y4=
│   ├───nixpkgs: github:nixos/nixpkgs/2080afd039999a58d60596d04cefb32ef5fcc2a2
│   └───utils: github:numtide/flake-utils/08c7ad4a0844adc4a7f9f5bb3beae482e789afa4
├───naersk: github:nmattia/naersk/a76924cbbb17c387e5ae4998a4721d88a3ac95c0
│   └───nixpkgs follows input 'nixpkgs'
├───nixpkgs follows input 'flock-of-flakes/nixpkgs'
└───utils follows input 'flock-of-flakes/utils'
1 Like
9

I guess validated: it is. Thanks!