Custom flake outputs for checks

Help
2

To be clear, do you mean a check output type that is not a derivation?

{
  # Executed by `nix flake check`
  checks."<system>"."<name>" = not-a-derivation;
}

(As far as I know I think you can put arbitrary derivations under checks."<system>"."<name>")

3

I am trying to get rid of these warnings:

warning: unknown flake output 'installMedia'
warning: unknown flake output 'homeManagerConfigurations'
warning: unknown flake output 'deploy'

That might be what I am trying to go for but not sure how to use it, checked the nix manual but it is not documented much.

4

Right, I’ve run into this myself. In my case it was

warning: unknown flake output 'lib'
warning: unknown flake output 'nixopsModules'
warning: unknown flake output 'nixopsConfigurations'

I think the idea with the flake schema is to have a “canonical” schema that everyone agrees on and can write tooling against, so I think the warning may be fair.

For myself I just decided to live with the warning, but personally I think it’s worth at least formalizing lib as an official output / escape hatch.

Note that even nixpkgs itself has non-standard outputs in its flake:

5

Related:
https://github.com/NixOS/nix/issues/4744
https://github.com/NixOS/nix/issues/6381
https://github.com/NixOS/nix/issues/5716

6

And also just recently:

https://github.com/NixOS/nix/issues/6453

(This post remembered me that I wanted to post that issue already about a month ago)

1 Like
7

Thanks for the issue links, so it seems its not a solved problem. I really like your solution NobbZ and was what I was hoping already existed. The proposal is what I hoped nix flake check could do.

Regardless if the solution is hiding the warnings or a new output, It definitely should have some sort of solution as giving a warning with no way of fixing doesn’t make much sense. The use of non-standard outputs is so prevalent throughout the flake ecosystem that even if a “standardized schema” was the original intention, the actual use cases have expanded.

8

Although I do think there’s definite value in an escape hatch, I don’t entirely agree with this. I think that:

  1. If flakes aren’t a standardized format, then what are they? Why didn’t we just stick with the status quo and put outputs in e.g. release.nix etc?

  2. Many outputs from nix ecosystem tools could potentially be standardized without a huge amount of effort IMHO.

    Maybe even a pattern like ____Configuration.
    E.g. deployRsConfiguration, nixopsConfiguration, homeManagerConfiguration, etc.

    Alternatively, I think ____Modules could also make a lot of sense.
    E.g. deployRsModules, nixopsModules, homeManagerModules

  3. For other unusual output I think it’s worth considering if they they really belong in top-level attributes. E.g.

  • htmlDocs in nixpkgs: Possibly these are just packages?

  • installMedia: Maybe this is a nixosConfiguration ? I have an installer in my nixos configurations which I build with nix build .#nixosConfigurations.installer.config.system.build.isoImage.

    Thinking about it another possibility is:

    { self, ... }:
{
  nixosModules.installer = 
    import ./nixos-modules/profiles/installer;

  nixosConfigurations.installer = 
    lib.nixosSystem {
      system = "x86_64-linux";
      modules = [ self.nixosModules.installer ];
    };

  packages.installerIso = 
    self.nixosConfigurations.installer.config.system.build.isoImage;
}

    Now I can build .#installerIso…
    …or I can deploy .#installer to a VM for testing
    …or I can consume this flake in another flake and build on top of nixosModules.installer to make my own custom installer

    So the flake schema helped me get this “right” from the start.

So in my opinion the warnings are actually quite helpful as a nudge in the right direction.

1 Like
9

P.S. I don’t have the bandwidth to babysit an RFC, but I think it would be :heart: if someone found the time to standardize output schema patterns a bit more.

I think it’s nice that there’s a bunch of non-standard flakes out there now that can be used to inform such an RFC.

10

I think there is a place for community extensions even in a standard. home-manager, assuming it never becomes official, is the perfect example - we don’t want to acknowledge it as an official part of the standard and have support for it in nix itself, but it is still widely used in the community.

It’d be awesome if nix flake check could at least resolve my home configuration and tell me if it’s horribly broken before I push it, but that won’t happen as long as there is no support for extensions.

Trying to map all possible use cases would simply take too much effort to sensibly support upstream. You’ve already come up with more than flakes support today, and even what exists today seems to take almost too much work to make anything near ergonomic enough for a general audience.

Things like lib absolutely should become part of the default output set, though.

11

Trying to map all possible use cases would simply take too much effort to sensibly support upstream.

Sure, like I said, I think some kind of escape hatch is definitely helpful (though I’d also say the impact of the warning is pretty low for now - mostly just an irritation)

It’d be awesome if nix flake check could at least resolve my home configuration and tell me if it’s horribly broken before I push it, but that won’t happen as long as there is no support for extensions.

Isn’t it already possible to do this via something like say

let mkCheck = ???
in
{ checks."<system>".home = mkCheck self.homeManagerConfigurations.myHome; }

?

home-manager , assuming it never becomes official, is the perfect example - we don’t want to acknowledge it as an official part of the standard

I think that once we have an RFC with a pattern for these things that we like, something like homeManagerConfigurations or homeManagerModules is hopefully just a reasonably small uncontroversial PR away…

I’d certainly like to have a standard way of listing home-manager modules so that when I trawl through other people’s dotfiles it’s quick to find what I’m looking for.

12

Yep, but that takes extra effort and isn’t very ergonomic. In theory we don’t need checks either, you could just put your checks in packages.

13

Yep, but that takes extra effort and isn’t very ergonomic.

I think with a lib output it could be?

{ 
  inputs.home-manager.url = "github:....";

  outputs = { self, home-manager }: {
    homeManagerConfigurations = {
      home = 
        home-manager.lib.mkConfiguration ./home-manager-configurations/home;
    };

    checks = 
      home-manager.lib.mkChecks self.homeManagerConfigurations;
  };
}

In theory we don’t need checks either, you could just put your checks in packages.

Yeah I mean, I guess the things I can think of for checks are:

  • Build derivations - most things can be expressed as a derivation
  • Check that flake output schema is correct
  • Type check nix modules (not implemented for nixosModules right now - not sure if implementable)
  • Checking that flake inputs resolve (although building derivations lead to this anyway)
14

Is it possible for this thread to get moved to development as it seems the topic has shifted. I am a fan of extending nix flake check command functionality. I think it should stick to being provided by the user as it takes advantage of the decentralization of nix flakes. Each flake provider can choose how its output works, and update it easily.

One thought is to make an outputChecks output. It is a new output used to verify custom flake outputs.

extraOutputs = {
   "homeManagerConfigurations" = {
         check = input: checkHome;
        metadata = ...
   };
   "deploy" = {
       check = input: checkDeploy;
   };
}

The check is of type Any -> bool. If an output exists and it doesn’t correspond to an attribute in outputCheck then get an unknown output warning. If the function returns false fail the nix flake check. This would also allow extending something like metadata (out of scope of this discussion) as it would just extend the extraOutputs schema.

15

Is it possible for this thread to get moved to development as it seems the topic has shifted.

Maybe @moderators can help?

One thought is to make an outputChecks output. It is a new output used to verify custom flake outputs.

I’m still a little confused about the difference to the existing checks output to be honest, aside from having them nested under outputs?

(EDIT: I did not read properly!)

The check is of type Any -> bool .

So I think I would say the existing checks have a drv type. Which seems ok to me because:

  • You can build them from any arbitrary arguments (even supplying multiple outputs to build a single drv)
  • bool as an output type doesn’t really give very much information on what is wrong. Instead I think I can hopefully just do something akin to throw "this exact thing is wrong" in my check
  • Derivations let you run arbitrary shell scripts so you can use external tooling as part of your checks.
  • For things like metadata we probably want type checks (maybe via the module system)
16

Oh sorry, just to say I missed your point about extraOutputs. I didn’t read carefully enough!

(Not sure this is the exact form it (an escape hatch) should take and I do think checks can continue as a standalone - but your suggestion is worth consideration)

1 Like
17

The derivation part makes sense, it just has to be built for the check to pass. The issue with the existing checks option is how would one give context to what a known output is. It just is a derivation that needs to be built. We need to tell nix that we want to add a custom output, and check if the custom output is valid.

2 Likes
18

Yep, the ergonomics were just a quick thought I had. Can be changed. I see the checks as arbitrary checks that can be applied. While something along the line of outputChecks as the formal way to expand the nix schema.

1 Like
19

Sorry I’m super chatty today for some reason and probably ought to get back to work but I will say:

The reason for schema is for tooling like this

I think the way to think about the flake schema is: how do you achieve the same kind of registry/discoverability/tooling across the ecosystem of nix-powered tools.

  • search.deploy-rs.org?query=aws
  • search.nixops.org?query=aws
  • search.home-manager.org?query=vim
  • etc

and then work backward.

Basically, to emphasize, it doesn’t matter if the output schema lives in nix, or if the output schema lives in home-manager, as long as there is one central place where the schema is defined.

So if there is some form of extraOutputs (how about schemas or outputSchemas to make it clear that this defines a schema, not a buildable output), then I suppose that you’d want it to be supplied by the github:nix-community/home-manager input flake.

In other words your own flake that concretely has a homeManagerConfiguration output wouldn’t define this schema, but it would come from an input.

I think I’ve come around to this… It might be what others were already imagining, and I just caught on - not sure!

1 Like
20

Flip-flopping again: I think the test of this would be if output schemas is actually required to handle any real world use cases - or if a whitelist of possible outputs is sufficient.

My worry is that if you’re starting to define schemas in flakes then you’re asking for some form of extensible module system, which we already have.

I guess in an alternate world flakes would look like:

{
inputs = {
  nixos.url = "github:nixos/nixpkgs";
  home-manager.url = "github:nix-community/home-manager";
  nixops.url = "github:nixos/nixops";
};
outputs = {
  nixos = { 
    modules = ...;
    configurations = ...;
    templates = ...;
    overlays = ...;
  };
  home-manager = {
    configurations = ...;
    templates = ...;
    modules = ...;
    overlays = ...;
  };
  nixops = {
    resources = ...;
    deployments = ...;
    modules = ...;
  };
};
}

Then each of these top-level attributes would be defined as modules.

  • inputs = { ... }; starts to look a lot like imports = [ ... ];
  • outputs = { ... }; starts to look a lot like config = { ... };
  • adding something like schemas = { ... }; looks a lot like options = { ... };

:man_shrugging:

1 Like
21

:slight_smile: I like your thread in development. But to point it out, I think that is indeed pretty close to what is described @NobbZ in Extend `nix flake check` to support custom attributes · Issue #6453 · NixOS/nix · GitHub. I think we were all focused specifically on checks though, and not quite far-sighted enough to consider uprooting the entire flake syntax just to get rid of our warnings :wink:

1 Like