Pre-RFC: Modularized flakes

rehno · April 28, 2022, 11:00pm

I haven’t been too involved in nix development but I couldn’t help but get pulled into some thoughts about flake schema in this thread: Custom flake outputs for checks - #20 by rehno

If you modify the flake schema with top-level attributes that look like this:

{
  inputs = {
    nixos.url = "github:nixos/nixpkgs";
    home-manager.url = "github:nix-community/home-manager";
    nixops.url = "github:nixos/nixops";
    templates.url = "github:nixos/templates";
  };
  outputs = { self, ... }: {
    nixos = {
      modules = ...;
      configurations = ...;
      packages = ...;
      overlays = ...;
    };
    homeManager = {
      modules = ...;
      configurations = ...;
      templates = ...;
      overlays = ...;
    };
    nixops = {
      modules = ...;
      resources = ...;
      deployments = ...;
    };
    templates = {
      nixos-example = ...;
      nixops-example = ...;
      home-manager-example = ...;
    };
  };
}

Then you could imagine a world where those input flakes look like this:

nixpkgs/flake.nix:

{
  inputs.lib.url = "github:nixos/lib";
  options = { lib }: {
    nixos = {
      modules = lib.mkOption { ... };
      configurations = lib.mkOption { ... };
      templates = lib.mkOption { ... };
      overlays = lib.mkOption { ... };
      packages = lib.mkOption { ... };
    };
  };
  outputs = { ... };
}

home-manager/flake.nix:

{
  inputs.lib.url = "github:nixos/lib";
  options = { lib }: {
    homeManager = {
      modules = lib.mkOption { ... };
      configurations = lib.mkOption { ... };
      templates = lib.mkOption { ... };
      overlays = lib.mkOption { ... };
    };
  };
  outputs = { ... };
}

nixops/flake.nix:

{
  inputs.lib.url = "github:nixos/lib";
  options = { lib }: {
    nixops = {
      modules = lib.mkOption { ... };
      resources = lib.mkOption { ... };
      deployments = lib.mkOption { ... }; # deployment targets (aka "machines")
    };
  };
  outputs = { ... };
}

templates/flake.nix:

{
  inputs.lib.url = "github:nixos/lib";
  options = { lib }: {
    templates = lib.mkOption { ... };
  };
  outputs = { ... };
}

Then this starts to resemble the existing module system a great deal.

inputs = { ... }; resembles imports = [ ... ];
outputs = { ... }; resemble config = [ ... ];
options = { ... }; obviously resembles the attribute with the same name

Is this a new observation, or am I just re-treading a well-trodden design landscape? I know flakes have been discussed to death and I could imagine that there’s been many other people drawing comparison to the module system. (Apologies if that’s the case!)

I’m curious if it breaks any desirable properties of flakes. Thanks

rehno · April 28, 2022, 11:38pm

I’m curious if it breaks any desirable properties of flakes.

I suppose the obvious “broken” property is that the nix cli as it currently stands “depends” on the nixos options, making the nixos input mandatory or implicit.

In a similar way home-manager as a standalone tool would “depend” on the home-manager input being present, nixops would “depend” on the nixops input being present. Etc.

rehno · April 29, 2022, 12:39am

I’m curious if it breaks any desirable properties of flakes.

Namespace collisions

Another argument against would be collisions in the schema. E.g. who gets to “own” the top-level templates attribute in the outputs schema?

A partial mitigation for this is that tools for things like search.nixos.org could simply look for github:nixos/templates in the lock file to know that templates match the expected schema.

(It reminds me of xmlns for xml namespacing - e.g. <svg xmlns="http://www.w3.org/2000/svg">)

Any ability to transform / override options provided by another flake would be problematic though, so collisions still seem like a bit of an issue.

An alternative resolution would be that the output namespace is given by the inputs, though I’m not too sure how I feel about it.

{
  inputs.nixos.url = "github:nixos/nixpkgs/nixos-21.11";
  inputs.nixos-unstable.url = "github:nixos/nixpkgs/nixos-unstable";

  outputs = { self, ... }: {
    # Type checks against the options provided in the nixos input
    nixos.modules = { ... };
    nixos.configurations = { ... };

    # Type checks against the options provided in the nixos-unstable input
    nixos-unstable.modules = { ... };
    nixos-unstable.configurations = { ... };
  };
}

search.nixos.org would need to index both nixos.* in and nixos-unstable.* in the example above, recognizing that both originate from github:nixos/nixpkgs.

Yet another resolution would be overriding namespace in the inputs

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-21.11";
  inputs.nixpkgs.namespace = "nixos";

  inputs.nixpkgs-unstable.url = "github:nixos/nixpkgs/nixos-unstable";
  inputs.nixpkgs-unstable.namespace = null; # unused

  outputs = { self, ... }: {
    # Type checks against the options provided in the nixpkgs input
    nixos.modules = { ... };
    nixos.configurations = { ... };
  };
}

NobbZ · April 29, 2022, 4:50am

Can you elaborate on this? There is no nixos input in any of my flakes, and it still works.

In general, I am a bit on the fence about this proposal, it increases the already annoying nesting by yet another layer… nixpkgs.nixos.packages.x86_64-linux.hello… Also, it will probably not be clear to non-nixos users to use nixos. While it would be weird to have nixpkgs.nixpkgs.modules (because of the double nixpkgs and the modules from nixpkgs).

Also I do not think, we should change the semantics of modules, or how they are written, just to have a nicer flake schema.

Pacman99 · April 29, 2022, 5:52am

I think you practically described almost our exact api in divnix/digga.

There’s also hercules-ci/flake-modules-core that has similar idea.

Both projects use the module system to describe flake configurations. In digga the API is separated by configuration systems(home, nixos, darwin). Inside you can add shared modules for all hosts. There’s also ways to export certain things.

You find the digga API docs here: digga/api-reference.md at 0ffa2dff5ede12a03ca83fbc514972f94cf18f42 · divnix/digga · GitHub

regnat · April 29, 2022, 9:34am

That goes actually deeper than that, because afaik having a module system was part of the initial end-goal of flakes (and is still on the radar somewhere). The big limitation is that this shouldn’t rely on the NixOS module system (for a lot of reasons, the most obvious being layering constraints and performances) but rather be something build-in to Nix, and that would require a lot of design to get right.

rehno · April 29, 2022, 12:31pm

This approach would be renaming things such that

nixosConfigurations → nixos.conifigurations
nixosModules → nixos.modules
packages → nixos.packages

So the nix build command is very closely tied to at least the packages input/output. Without the packages input there’s no type definition to check the packages output against. So to be consistent nix would need to report that "nixos.packages" is an unknown output if the nixos input is not listed. For the record I think this is a problem with my proposal.

I also think that packages probably wants to live in the root namespace similar to my templates example earlier. Though on the other hand, I could imagine package sets as “namespaces” being kind of ergonomic: nixos.packages, python.packages, vim.packages, rust.packages etc.

Some possibilities are

packages.url = "github:nixos/packages
packages.url = "github:nixos/nixpkgs?dir=schema/packages
packages is provided by a special implicit input (E.g. "github:nixos/base" or whatnot) that merges into the root namespace.

nixpkgs gets to use the root namespace somehow. Something like:

{
  inputs.nixos.url = "github:nixos/nixpkgs";
  inputs.nixos.namespace = "."; # root namespace 

  # Alternative idea:
  # inputs.nixos.options.namespace = "."; # root namespace

  inputs.homeManager.url = "github:nix-community/home-manager";

  outputs = {
    # These are type checked against options defined in nixos (github:nixos/nixpkgs)
    nixosConfigurations = { ... };
    nixosModules = { ... };
    packages = { ... };

    # These are are type checked against options in homeManager (github:nix-community/home-manager)
    homeManager = {
      modules = { ... };
      configurations = { ... };
    };
  }
}

There’s some facility to import / collapse the namespaces (sort of like with / inherit, but not as a keyword I would think. This seems like it would make things complicated).

The challenge is to support various nix ecosystem tools nixops / home-manager / deploy-rs / etc in a uniform way in flakes. I think this is clearly not the only solution - I’m just not super clear on what the other solutions would look like. However, I’m pragmatic about it.

(Some of the github issues from the linked thread are symptoms of the underlying problem I think.)

Gotcha, thank you. I hope that my little exploration here contributed something to the design space at least but it’s good to hear that it’s on people’s radar.