How to structure own `pkgs` and `lib` in own monorepo

[gabyx]

I would have liked to share some setup we do in our monorepo custodian.
Since we are new to Nix and the ways people do theses things with Nix, I would have been interested in knowing, if the following is actually senseful and what can be improved.

Design Choices:

We wanted to contain everything of our monorepop in an attrset custodian = { pkgs = ...; , lib = ...; } where lib contains common functions we need in the different places where we build things and pkgs containts custodian related configured packages/tools.

A build function default.nix for a component looks like:

{ lib, custodian }:
let
  cnPkgs = custodian.pkgs;
  cnLib = custodian.lib;
  compName = "policy-decision-point";
in

cnPkgs.buildGoModule {
  pname = compName;
  version = cnLib.components.readVersion compName;

  src = cnLib.filesets.toSource [ 
    compName
    "lib-common"
  ];
  ...
}

• Also it is nice to use pkgs.buildGoPackage from nixpkgs, we use our own modified version of it
  to not be dependend of nixpkgs. Also we didnt like the full blown bash scripts in buildGoPackage and we use anyway Go scripts to tool everything we directly use our tool to build in custodian.pkgs.buildGoModule which depends on a pinned version of pkgs.go and our ci-tool (Go scripting tool).

• We use cnLib.filesets.toSource which lets use fuse certain filesets of different components (parts of our monorepo) together. So the above uses its own fileset and also the one from lib-common. The filesets are configured globally in custodian.lib.filesets which reads over the monorepo and searches for .component.yml files etc.

Questions

• Should the custodian attrset be overlayed into nixpkgs.pkgs to have pkgs.custodian eventhough it is itself not a derivation perse (argument for that is: pkgs.lib is also not a derivation?
  Currently we need to call the package above like pdp = pkgs.callPackage <path>/tools/nix/package { inherit custodian; };
  which could be simplified to pkgs.callPackage ... {}.

• Is the own custodian.pkgs.buildGoPackage a good idea?

• Are the arguments to default.nix above problematic? (You dont see any go deppendency since its hidden away in cnPkgs.buildGoPackage etc…)

• Currently we use overlays only to modify or add the pinned package set in nixpkgs.pkgs and do not add our custodian attrset. Should we think about injecting all our packages into nixpkgs.pkgs to make other components depend on them in their default.nix function and use nixpkgs.pkgs.callPackage or write our own modified version for custodian.pkgs.callPackages which would first fill all default arguments from custodian.pkgs and then use also nixpkgs.pkgs as a second step.

• How does nixpkgs deal with non-derivations in nixpkgs.pkgs ? How can we build recursively values from an attrset which are derivations and disgard other stuff?

Thanks for your comments/inputs/critics.

[jakehamilton]

Hey there! The issues with organization are pretty common as a Nix project grows. Things tend to grow in a pretty similar way between projects so patterns can be extracted and reused. I did this to my own projects and built out a library which wires everything together by enforcing a project structure for your Nix files: GitHub - snowfallorg/lib: Unified configuration for systems, packages, modules, shells, templates, and more with Nix Flakes.

With that in mind I’ll answer these questions as best I can through the lens of using Snowfall Lib.

• Should the custodian attrset be overlayed into nixpkgs.pkgs to have pkgs.custodianeventhough it is itself not a derivation perse(argument for that is: pkgs.lib is also not a derivation?
  Currently we need to call the package above like pdp = pkgs.callPackage <path>/tools/nix/package { inherit custodian; };
  which could be simplified to pkgs.callPackage ... {}.

Sure! Preferably builders and derivations are the only things on pkgs. pkgs.lib is the outlier here. If you use Snowfall Lib then lib and pkgs will automatically be extended with your own namespace for helpers and packages. For your buildGoModule replacement, you should probably use an overlay to manually add the builder to the package set.

• Is the own custodian.pkgs.buildGoPackage a good idea?

If you mean having your own builder instead of the Nixpkgs one then I think it makes sense so long as it solves problems you have! If it does things you need then go for it.

• Are the arguments to default.nix above problematic? (You dont see any go deppendency since its hidden away in cnPkgs.buildGoPackage etc…)

This is a common issue with how Nixpkgs builders compose. It may be a good idea to support passing a golang package through to use or, alternatively providing version-specific builders.

• Currently we use overlays only to modify or add the pinned package set in nixpkgs.pkgs and do not add our custodian attrset. Should we think about injecting all our packages into nixpkgs.pkgs to make other components depend on them in their default.nix function and use nixpkgs.pkgs.callPackage or write our own modified version for custodian.pkgs.callPackages which would first fill all default arguments from custodian.pkgs and then use also nixpkgs.pkgs as a second step.

I think this would be reasonable if done under a namespace. That way things are configurable while not conflicting with existing packages.

• How does nixpkgs deal with non-derivations in nixpkgs.pkgs ? How can we build recursively values from an attrset which are derivations and disgard other stuff?

Nixpkgs doesn’t really care what you put on pkgs itself. What makes you want to supply packages in a recursive structure?

[gabyx]

Thanks for the snowfall lib thing and the nice input!. That seems very useful. Although I will probably not directly now use it, since I do not want to add another entry level difficulty to understand things for my co-workers, I would go right away with this lib/layout I guess.

I refactored with your input into having pkgs.custodian and lib.custodian basically (where pkgs is the nixpkgs instance) and I loaded these variables in an overlay function.

What I saw in snowfall is that adding a Library has no pkgs as input. In my case lib.custodian.readYAML is setup with some workaround to read a YAML over runCommand with pkgs.remarshal… Technically then it would not be a lib anymore but, I was not sure how to overcome this, it would then live in pkgs.custodian.readYaml consequently maybe?

My final setup with the overlay is here:

fix: niceify the nix tooling (!468) · Merge requests · data-custodian /...

k

The snowfall lib is nice. Do you have some repo examples where this is used?

Thanks a lot!

[jakehamilton]

it would then live in pkgs.custodian.readYaml consequently maybe?

I think that is a reasonable solution. Nixpkgs doesn’t have a great solution here since things such as importJSON live in lib, but are implemented with builtins (not to mention the use of IFD). Since it depends on the package set I think you can either put it on pkgs itself (perhaps the simplest solution) or expose a library function that requires a pkgs instance to be passed in as an argument for each use (which would be more cumbersome).

The snowfall lib is nice. Do you have some repo examples where this is used?

Indeed!

GitHub - jakehamilton/config: One Nix flake to rule them all.

GitHub - jakehamilton/neovim: My customized version of Neovim, built with Nix.

GitHub - snowfallorg/cowsay: More cows, optionally rendered as images!

GitHub - snowfallorg/vhs: Render VHS tapes with Nix.

[jakehamilton]

Also, a heads up, I saw you were adding to lib in your overlay by using // to merge attr sets. This will run into issues for consumers of your project because any call to lib.extend will not include your changes. You should replace lib // myLib with a call to lib.extend which will ensure every consumer downstream has your additions.

[terlar]

Another great asset to organise code is to use flake-parts. With this you could create your own DSL using the module system to fit your use-case. It also makes it easy to propagate arguments to all the ”parts”. Especially if the end product is going to be a flake.

I did work on Nix monorepos before with the approach to add the project namespace to the package set via overlays and in that sense propagating it down to each derivation. That works as well.

[gabyx]

Thanks for mentioning this. I still dont get it. Isnt it that when you would load my flake (which is not the scope for now) → you would get myflake.lib which is correctly build. Or what do you mean with consumer?

so your are saying

lib = pkgsPrev.lib // { custodian = 2 }

in the overlay is not the same as

lib = lib.extend (final: prev: prev // { custodian = 2 }; )

in the overlay?

https://www.reddit.com/r/NixOS/comments/q8ukbn/comment/hgvfxpr/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

[jakehamilton]

Yes, the refactored version is correct. The reason being that any project which adds to lib using extend will only have access to the propagated fixed point. Without using extend your changes would not be included. Projects like home-manager use lib.extend to add their own helpers and you don’t want your packages to suddenly and unexpectedly break because they cannot find library functions.

[gabyx]

Sorry I was a bit sloppy:

The refactored version is:

inputs:
pkgsFinal: pkgsPrev:
let
  cn = (import ./custodian { inherit inputs; }) {
    rootPath = ../..;
    pkgs = pkgsFinal;
  };
in
{
  # We inject all our custodian packages in namespace `custodian`.
  custodian = cn.pkgs;

  # Add out custodian lib to the
  # existing lib under the namespace `custodian`.
  lib = pkgsPrev.lib.extend (
    final: prev:
    prev
    // {
      custodian = cn.lib;
    }
  );
}

Thats what you meant right?

[jakehamilton]

That should do it! One of those quirks with how Nixpkgs’ lib does composition.

[terlar]

You don’t need to and probably shouldn’t merge prev with the overlay value. So change to this:

  lib = pkgsPrev.lib.extend (
    final: prev: {
      custodian = cn.lib;
    }
  );