Umport: Automatic Import of Modules

Automatic import of all necessary modules with support for recursion and copying to /nix/store (optional), instead of default.nix with imports in each folder

ylib.umport {
  path ? null, # path (required parameter if paths is [])
  paths ? [], # multiple paths
  include ? [], # file paths that will be included anyway
  exclude ? [], # file paths that will be excluded anyway
  copyToStore ? false, # copy found files to /nix/store before importing them
  recursive ? false, # recursively import a path or paths
};

Usage

flake.nix

inputs.nypkgs.url = "github:yunfachi/nypkgs";
nixosConfigurations.<configuration> = lib.nixosSystem {
  # just a use case
  modules = 
    modules =
      [
        ./configuration.nix
      ]
      ++ ylib.umport {
        paths = [../programs ../services];
        recursive = true;
      };

  specialArgs = {
    ylib = nypkgs.lib.<system>;
    /* if you want to use ypkgs
    ypkgs = nypkgs.legacyPackages.<system>;
    ylib = ypkgs.lib; */
  };
}

default.nix

# just a use case
{ylib, ...}: {
  imports = ylib.umport {
    paths = [../programs ../services];
    recursive = true;
  };
}

Examples

Examples with execution result and tree
ylib.umport {
  paths = [./programs ./services];
  recursive = true;
};
#=> [ "/path/programs/cli/more/end.nix" "/path/programs/cli/default.nix" "/path/programs/cli/raboof.nix" "/path/programs/fuga.nix" "/path/programs/hoge.nix" "/path/services/bar.nix" "/path/services/eric.nix" "/path/services/foo.nix" ]
/path
β”œβ”€β”€ programs
β”‚   β”œβ”€β”€ cli
β”‚   β”‚   β”œβ”€β”€ default.nix
β”‚   β”‚   β”œβ”€β”€ more
β”‚   β”‚   β”‚   └── end.nix
β”‚   β”‚   └── raboof.nix
β”‚   β”œβ”€β”€ fuga.nix
β”‚   └── hoge.nix
└── services
    β”œβ”€β”€ bar.nix
    β”œβ”€β”€ eric.nix
    └── foo.nix



ylib.umport {
  path = ./programs;
  include = [./services/foo.nix ./services/bar.nix];
  exclude = [./programs/hoge.nix];
};
#=> [ "/path/programs/cli" "/path/programs/fuga.nix" "/path/services/foo.nix" "/path/services/bar.nix" ]
/path
β”œβ”€β”€ programs
β”‚   β”œβ”€β”€ cli
β”‚   β”‚   └── default.nix (when importing the directory, default.nix will be imported)
β”‚   └── fuga.nix
└── services
    β”œβ”€β”€ bar.nix
    └── foo.nix



ylib.umport {
  paths = [./programs ./services];
  include = [./default.nix];
  copyToStore = true;
  recursive = true;
};
#=> [ "/nix/store/kzx79d3j4hjcqk2yxvl153xmqh5qlvw4-more/end.nix" "/nix/store/pa64cc7vv8m1zvf6w4s9hvh4yz3hcx1f-default.nix" "/nix/store/py7029p6vcarj19mn8a8mbcqxjz4a318-cli/default.nix" "/nix/store/py7029p6vcarj19mn8a8mbcqxjz4a318-cli/raboof.nix" "/nix/store/y6ljmqzqrqka3y9mqkb0sp2r6d80dznv-programs/fuga.nix" "/nix/store/y6ljmqzqrqka3y9mqkb0sp2r6d80dznv-programs/hoge.nix" "/nix/store/l3sn7p8mlb95zdlv0mwnqdiyy6k1adhb-services/bar.nix" "/nix/store/l3sn7p8mlb95zdlv0mwnqdiyy6k1adhb-services/eric.nix" "/nix/store/l3sn7p8mlb95zdlv0mwnqdiyy6k1adhb-services/foo.nix" "/nix/store/pa64cc7vv8m1zvf6w4s9hvh4yz3hcx1f-default.nix" ]
/
└── nix
    └── store
        β”œβ”€β”€ kzx79d3j4hjcqk2yxvl153xmqh5qlvw4-more
        β”‚   └── end.nix
        β”œβ”€β”€ l3sn7p8mlb95zdlv0mwnqdiyy6k1adhb-services
        β”‚   β”œβ”€β”€ bar.nix
        β”‚   └── foo.nix
        β”œβ”€β”€ pa64cc7vv8m1zvf6w4s9hvh4yz3hcx1f-default.nix
        β”œβ”€β”€ py7029p6vcarj19mn8a8mbcqxjz4a318-cli
        β”‚   β”œβ”€β”€ default.nix
        β”‚   └── raboof.nix
        └── y6ljmqzqrqka3y9mqkb0sp2r6d80dznv-programs
            β”œβ”€β”€ fuga.nix
            └── hoge.nix

Why is this needed?

If you have a lot of modules or packages, then you can automatically call or import them using this, but only if you find it more convenient than regular imports

GitHub: GitHub - yunfachi/nypkgs: yunfachi's Nix Packages collection

9 Likes

This could be a great application of the file set library, which could potentially reduce this tool to ~10% of its size. Right now the file set library only provides toSource as a usable representation (which seems to correspond to the copyToStore behavior), but one could fairly simply add a transformation to a list of paths.

7 Likes

How easy would it be to create the transformation to a list of paths without accessing the library’s internal functions?

Maybe it should be included in the fileset library itself?

One of the most common problems I come across in nixos configuration repos is the burden of importing nix files recursively.
digga’s rakeLeaves and haumea are often used to solve this problem but now that lib.filesets exists they seem like (reinventing the wheel)/overkill.

3 Likes

Indeed there currently is no way to turn a fileset into a list of paths, but that’s only because I wasn’t necessary until now. I think it would be very useful to be able to do that now that I’m seeing this post :smiley:

Could you open a new issue in Nixpkgs to request this and ping me? You can also link it to File set library tracking issue and feature requests Β· Issue #266356 Β· NixOS/nixpkgs Β· GitHub

8 Likes

Hey @yunfachi, this looks great!

Can you please help me use umport?

I’m currently trying to use it in a flake.nix (nix-darwin) > home.nix (home manager) > modules context and really strugging to get past issues ( similar to that I’ve encountered since migrating from just a home manager flake.nix > modules setup) because I no longer have a modules block as per your examples in my home manager config (home.nix, no longer flake.nix) other than in nix-darwin’s flake.nix, and I want my home manager modules loaded via home.nix). I have tried in the home manager module in nix-darwin’s flake but failed.

My goal is to later use umport too in the nix-darwin flake for any modules eventually installed via it system-wide, so if that is something you can advise on too while helping me understand how to use umport in home manager, that would be much appreciated!

PS Thank you for such a great tool and one that I’m surprised didn’t exist until you built it and it appears for upgrading it to use lib.fileset (another great addition to nix I’m grateful for as I will hopefully learn properly after resolving this issue).

Abbreviated flake.nix (nix-darwin):

{
  description = "xxx's nix-darwin flake";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-23.11-darwin";
    nixpkgs-unstable.url = "github:NixOs/nixpkgs/nixpkgs-unstable";
    home-manager.url = "github:nix-community/home-manager/release-23.11";
    home-manager.inputs.nixpkgs.follows = "nixpkgs";
    nix-darwin.url = "github:LnL7/nix-darwin";
    nix-darwin.inputs.nixpkgs.follows = "nixpkgs";
    nur.url = "github:nix-community/NUR";
    nypkgs.url = "github:yunfachi/nypkgs";
    nypkgs.inputs.nixpkgs.follows = "nixpkgs";

  };

  outputs = inputs@{ self, nixpkgs, nixpkgs-unstable, nix-darwin, nur, home-manager, nypkgs,  ... }:

  let

    unstableOverlay = final: prev: {
      unstable = import nixpkgs-unstable {
       system = "aarch64-darwin";
      };
    };

    ylib = nypkgs.lib.aarch64-darwin; <---##### Added because otherwise ylib was unknown

    configuration = { pkgs, ... }: {
      environment.systemPackages = [ ];
      services.nix-daemon.enable = true;
      nix = {
        extraOptions = ''
          extra-platforms = x86_64-darwin aarch64-darwin
        '';
        settings = {
          experimental-features = "nix-command flakes";
        };
      };
      programs.zsh.enable = true;  # default shell on catalina
      system = {
        configurationRevision = self.rev or self.dirtyRev or null;
        stateVersion = 4;
      };
      nixpkgs = {
        hostPlatform = "aarch64-darwin";
        config = { allowUnfree = true; };
        overlays = [
          nur.overlay
          unstableOverlay
        ];
      };
      imports = [ ./options-nix-darwin/options-nix-darwin.nix ];
        # eg settings to change dock location etc.
    };
  in
    {

    darwinConfigurations."myhostname" = nix-darwin.lib.darwinSystem {
      specialArgs = { inherit inputs; inherit ylib; };
      modules = [
        configuration
        home-manager.darwinModules.home-manager {
          home-manager = {
            useGlobalPkgs = true;
            useUserPackages = true;
            users.xxx = import ./nixpkgs/home.nix;
            extraSpecialArgs = { inherit inputs; };
          };
          users.users.xxx.home = "/Users/xxx";
        }
      ];
    };
    darwinPackages = self.darwinConfigurations."myhostname".pkgs;
      # Expose the package set, including overlays, for convenience.
  };
}

Abbreviated home.nix

{ pkgs, inputs, ... }: 

let

  unstable = import inputs.nixpkgs-unstable {
    system = pkgs.system;
  };

  ylib = import inputs.nypkgs.lib.aarch64-darwin;

in

  {
    home = {
      enableNixpkgsReleaseCheck = true;
      stateVersion = "23.11";
      packages = with pkgs; [
        pkgs.unstable.mise
        lunarvim
      ];
    };

    };

    fonts.fontconfig.enable = true;

    programs.home-manager.enable = true;
    programs.home-manager.path = "$HOME/.config/nix-darwin/nixpkgs/";

### How I am trying and knowingly failing to use umport:

imports = [
          ylib.umport {
          paths = [ ./pkgs-home-manager/modules-macos-active ./options-home-manager];
          recursive = true;
          }
         ];


   ### How I currently am importing home manager modules/programs into home.nix, which works but is very annoying at the actual list of imports is much longer with different paths:

    #imports = [
    #  ./home_manager/modules/cli/git.nix
    #  ## Home Manager Options
    #  ./home-manager-options.nix
    #];

}

defaults.nix

{ylib, ...}: {
  imports = ylib.umport {
    paths = [./modules-macos-active];
    recursive = true;
  };
}

All files above:

~/.config/nix-darwin/flake.nix
~/.config/nix-darwin/nixpkgs/home.nix
~/.config/nix-darwin/nixpkgs/pkgs-home-manager/default.nix
~/.config/nix-darwin/nixpkgs/pkgs-home-manager/modules-macos-active/cli, gui, ... each containing home manager modules (programs).

I have tried the way you present in the example, but for example ++ ylib.umport { ... gives a concat error.

As you’ll see in my home.nix, I previously imported every module (program) as path/module.nix, and now am wanting to use the method you describe for a restructred fileset (more organised directories), which allows automatic importing once a default.nix file in the root of the fileset. Mine is similar to yours, but all files are under . ./pkgs-home-manager/modules-macos-active and subdirectores eg /cli, /gui…

As predicted I get error: cannot coerce a set to a string, because I’m using imports = [ … that gets a fileset (or just because I’m using {} ), and I don’t know how to remove/replace the import to bring the fileset’s modules (programes eg path/git.nix) into home.nix.

Thanks

you can use inherit ylib; in extraSpecialArgs in the home-manager module in flake.nix.

in home.nix you should remove the square brackets and leave only ylib.umport:

imports = ylib.umport {
  paths = [ ./pkgs-home-manager/modules-macos-active ./options-home-manager];
  recursive = true;
};

ylib.umport returns a list. in my example I used ++ to combine the two lists, which can be useful in certain situations

1 Like

Thanks @yunfachi, that works!

(Although, I had to add ylib as an attribue at the top of home.nix too. Without it, extraSpecialArgs in nix-darwin’s home manager module doesn’t work for me, or perhaps it doesn’t work at all - I saw an issue re this while originally trying to get this working, but wasn’t sure if it related to my situation.)

I’m now struggling to get exclude to work when using recursive = true;. You don’t have an example for this, so is it yet possible? Or perhaps my setup is causing a problem, as I import each of the files I’m trying to exclude in files that umport imports as a list. I will show below:

.config/nix-darwin/nixpkgs/home.nix

imports = ylib.umport {
    path = ./pkgs-home-manager/modules-macos-active;
    exclude = [./cli/git/gitignore-global.nix ./gui/firefox/firefox-bookmarks.nix];
    recursive = true;
  };

nixpkgs/pkgs-home-manager/defaults.nix

{ylib, ...}: {
  imports = ylib.umport {
    path = ./modules-macos-active;
    exclude = [./cli/git/gitignore-global.nix ./gui/firefox/firefox-bookmarks.nix];
    recursive = true;
  };
}

The excluded files are imported as follows -

./cli/git/git.nix imports ./gitignore-global.nix via:

programs.git.ignores = import ./gitignore-global.nix;

and,

./gui/firefox/firefox.nix imports ./firefox-bookmarks.nix via:

ο»Ώbookmarks = import ./firefox-bookmarks.nix;

An error looks like this and stops at the first failed import:

error: module /nix/store/53pb0zvx3zrdl4hly5cp9rqxswzwaksq-source/nixpkgs/pkgs-home-manager/modules-macos-active/cli/git/gitignore-global.nix (/nix/store/53pb0zvx3zrdl4hly5cp9rqxswzwaksq-source/nixpkgs/pkgs-home-manager/modules-macos-active/cli/git/gitignore
-global.nix) does not look like a module.

I was getting this error when not excluding the above file (and same error type re firefox-bookmarks.nix), hence why I want to exclude them via umport but still import them as I was doing before using umport (as I am above).

I’m not sure why the erorr shows twice. Because the file is in home.nix umport and defaults.nix?

For exclude you need to specify the path relative to the file it is in, not path

exclude = [
  ./pkgs-home-manager/modules-macos-active/cli/git/gitignore-global.nix
  ./pkgs-home-manager/modules-macos-active/gui/firefox/firefox-bookmarks.nix
];

You can also specify programs.git.ignores in the gitignore-global.nix file (by making it a module), which will be easier when using umport.

1 Like

Thanks again @yunfachi !

Excludes now works. :slight_smile:

That’s a great suggestion re programs.git.ignores. I previously tried it, but used the way commented out below, because when I try import ./gitignore-global.nix; in git.nix with ο»Ώο»Ώprograms.git.ignores = [ ... ]; in gitignore-global.nix, I get this:

error: syntax error, unexpected PATH, expecting β€˜.’ or β€˜=’

32|
33|     import ./gitignore-global.nix;
  |            ^
34|

I can use imports by itself in .nix files, but it appears I haven’t yet learned how to use import without assigning it.

How do you do that? (Thanks!)

git.nix

{ config, pkgs, libs, ... }:
{

  programs.git = {
    enable = true;
    userName = "xxxxx xxxx";
    userEmail = "xxxxxxxxxxxxx";

    extraConfig = {
      init.defaultBranch = "main";        
      core.editor = "lvim";
      core.fileMode = false;
      core.ignorecase= true;
    };
  };

  import ./gitignore-global.nix;

  #programs.git.ignores = import ./gitignore-global.nix; <-- Before umport
}

gitignore-global.nix

ο»Ώ programs.git.ignores = [ 
   ...
 ];

I wrote something similar:

{ lib, ... }:

rec {
  flattenTree =
    tree:
    let
      op =
        sum: path: val:
        let
          pathStr = builtins.concatStringsSep "." path; # dot-based reverse DNS notation
        in
        if builtins.isPath val then
          # builtins.trace "${toString val} is a path"
          (sum // { "${pathStr}" = val; })
        else if builtins.isAttrs val then
          # builtins.trace "${builtins.toJSON val} is an attrset"
          # recurse into that attribute set
          (recurse sum path val)
        else
          # ignore that value
          # builtins.trace "${toString path} is something else"
          sum;

      recurse =
        sum: path: val:
        builtins.foldl' (sum: key: op sum (path ++ [ key ]) val.${key}) sum (builtins.attrNames val);
    in
    recurse { } [ ] tree;

  rakeLeaves =
    dirPath:
    let
      seive =
        file: type:
        # Only rake `.nix` files or directories
        (type == "regular" && lib.hasSuffix ".nix" file) || (type == "directory");

      collect = file: type: {
        name = lib.removeSuffix ".nix" file;
        value =
          let
            path = dirPath + "/${file}";
          in
          if (type == "regular") || (type == "directory" && builtins.pathExists (path + "/default.nix")) then
            path
          # recurse on directories that don't contain a `default.nix`
          else
            rakeLeaves path;
      };

      files = lib.filterAttrs seive (builtins.readDir dirPath);
    in
    lib.filterAttrs (n: v: v != { }) (lib.mapAttrs' collect files);

    filterPort =
    {
      include ? [ ],
      exclude ? [ ],
    }:
    let
      toList =
        list:
        lib.lists.unique (
          lib.concatLists (
            map (
              x:
              if builtins.isAttrs x then
                lib.attrValues (flattenTree x)
              else if builtins.isPath x then
                [ x ]
              else
                throw "Unexpected include or exclude list"
            ) list
          )
        );
    in
    lib.subtractLists (toList exclude) (toList include);
}

The flattenTree and rakeLeaves were stealed from digga/src/importers.nix at 0595ae70cdb5ccf1ab031199fe98551c4b378bd9 Β· divnix/digga Β· GitHub.

For example, you have a top-level directory to manage all modules, say modules, you should add mods = rakeLeaves ./modules to specialArgs of all module systems. So you can write

{
  imports = filterPort {
    include = with mods; [
      foo
      bar.baz
    ];
  };
}
1 Like