Flakes/Nix3.0 issues

gytis-ivaskevicius · October 22, 2021, 10:10am

Hi all, this thread is inspired by Improving flakes - #11 by nrdxp, lets try to list design issues or changes that you guys would like to see with flakes/Nix3.0

Here is my list, most of it is just improvements according to my fetishes

General ergonomics and UX improvements

Partially merge flake-utils:

eachSystem
Expose tier1/tier2/tierX systems support as arrays to be used as such nixpkgs.lib.eachSystem nixpkgs.lib.systems.tier1 or something similar
check-utils - Maybe not a ‘must’ but I quite like it
filterPackages and flattenTree would be nice.

Partial devShell implementation
flake.nix output changes:

lib.<name> output. Where name evaluates to a function or attrset which looks something like this:

lib.add = { 
    function = a: b: a + b;
    description = "Adds arguments"; 
    docs = ''
        The addition of two whole numbers results in the total amount or sum of those values combined. The example in the adjacent image shows a combination of three apples and two apples, making a total of five apples. This observation is equivalent to the mathematical expression "3 + 2 = 5" (that is, "3 plus 2 is equal to 5").
    '';
};

nixosModules and similar should be namespaced to something like modules.<name>. This would allow to have something like: modules.nixos, modules.darwin , modules.home , modules.customOptions , modules.suites
Would be nice to see NixOS modules system to get a description argument (This way we could present it in nix flake show)
nixosConfigurations.<hostname> evaluates to system architecture specific module. I think it would make more sense to change its definition to nixosConfigurations.<system>.<hostname>
If there are disagreements about point #6 - maybe nixosConfigurations should be moved under modules.nixos namespace mentioned in #4?
defaultPackage and similar attributes should be moved to packages.<system>.default
legacyPackages should be replaced with packages and something similar to flake-utils.lib.flattenTree. Tho, this might have a large impact on evaluation performance.
inputs.<name>.sourceInfo should include more information like repository name and owner
Extra CLI commands would be nice like nix fmt, nix info nixpkgs#bash, nix tree -> https://github.com/utdemir/nix-tree
Support flake references to patches · Issue #3920 · NixOS/nix · GitHub

Brainstorming

Location of nix registry and system configuration should be moved under /nix folder since these days /etc basically is ‘immutable global system configurations’
Registry sources (git repos) should be symlinked to /nix/registry/sources
apps vs packages - the difference between these two is that apps can not be built or installed as easily and are used only by nix run. (nix run abc#xyz currently checks apps.xyz → packages.xyz → legacyPackages.xyz)
I am thinking that maybe apps should be removed or maybe represent devshell (like nix shell -c xyz) since packages can be executed anyways. The end result should be something like "If explicitly defined runnable exists in devshell - use it, otherwise default to packages.xyz.
TL;DR: I am not a fan of apps and packages separation but I am not sure whats the solution is here nor how it’s supposed to work.

gytis-ivaskevicius · October 22, 2021, 11:12am

Brainstorming alert:
So I was thinking about apps thing, since I’m not a fan that it basically represents packages in a different namespace.
I feel like its a super odd API but something like this would be interesting:

outputs = { nixpkgs }:{
  devshell.x86_64-linux = nixpkgs.legacyPackages.x86_64-linux.mkShell { 
    runnables = {
      xyz = nixpkgs.legacyPackages.x86_64-linux.xyz;
    };
 }) ; 
}

this way xyz would be part of devshell and nix run .#xyz could evaluate outputs.devshell.x86_64-linux.runnables.xyz.
If nix run does not find such attribute in devshell - evaluate outputs.packages.x86_64-linux.xyz

TL;DR: This way nix run functions the same way as we are used to + you can create a custom builder so you could add defined apps as part of devshell.

Downsides:

Weird API
One may think that apps are started from devshell when they are not (or maybe they should be started from devshell?)

One may argue that what I’m looking for is namespaced package sets (like packages.<system>.<pkgsetName>.<name>) but at that point, we are reimplementing legacyPackages for better or for worse

Definitely not saying that any of this is a way to go, just a couple of thoughts.

NobbZ · October 22, 2021, 12:03pm

Personally I agree with some points here:

Lets add lib and make us able to give descriptions and docs to a function, perhaps we can re-use the __functor-idiom?
Lets add a way to add short descriptions to modules, configurations and apps or any other output that currently gets just a constant description on nix flake show (modules could get meta.description or meta.title, apps could get a description or title, etc)

I do not agree though with the following:

Rename *Modules to modules.*, instead provide a way to keep them flat as nixosModules, darwinModules, homeModules or whatever else might follow, by giving an interface that allows us to declare custom attributes with types.
Renaming of default* to *.$system.default, instead I’d prefer using the default prefix more consistently: defaultNixosModules, devaultDevShell, etc.
Removal/flattening of legacyPackages. Please instead give a way to properly nest packagesets in packages, but keep legacyPackages as legacyPackages in nixpkgs up until the point when really there is not a single function, not even builders and fetchers left in it, but really only nested package sets.
Also I strictly disagree with removal of apps. They are fine as they are, leaving them in doesn’t hurt anyone who doesn’t use them, but removing them hurts those who do.

gytis-ivaskevicius · October 22, 2021, 12:22pm

Rename *Modules to modules.* , instead provide a way to keep them flat as nixosModules , darwinModules , homeModules or whatever else might follow, by giving an interface that allows us to declare custom attributes with types.

Yeah, that would be an ideal option, unfortunately, I don’t see how does one implement it in a sane usable manner. It is very possible that I’m just not smart enough for that

Renaming of default* to *.$system.default , instead I’d prefer using the default prefix more consistently: defaultNixosModules , devaultDevShell , etc.

Consistency is what I’m looking for, I’m fine with both options it’s just that I’d prefer my option since all same type (‘type’ in terms of ‘package’, ‘app’, ‘overlay’, ‘devshell’) of attributes are under the same namespace.

Removal/flattening of legacyPackages . Please instead give a way to properly nest packagesets in packages , but keep legacyPackages as legacyPackages in nixpkgs up until the point when really there is not a single function, not even builders and fetchers left in it, but really only nested package sets.

Yeah, I believe that flattening is required due to performance reasons and the amount of packages there are. Please correct me if I’m wrong.

Also I strictly disagree with removal of apps. They are fine as they are, leaving them in doesn’t hurt anyone who doesn’t use them, but removing them hurts those who do.

From my point of view apps aren’t doing much since packages and devshell exist. Since I have not thought of a solution that I’d be happy with - I am NOT recommending the removal of apps, just asking to brainstorm to see if we can do better

NobbZ · October 22, 2021, 12:30pm

The problem I see with your way, what happens when someone actually wants to package something under the attribute name default?

Having self for the current flake, while there is also a pkgs.self trips me up regularly when I use callPackage and forget to pass in self explicitely, similar could happen with your proposal.

gytis-ivaskevicius · October 22, 2021, 12:54pm

Yeah, this makes default a reserved word. I just see it as a cleaner option compared to alternatives.
Of course, word itself does not matter, one could call it _default or something but that’s just pushing the edge case further to the edges.

As noted in the first message - these are just my thoughts/ideas based on my fetishes, I am just throwing this all out here to see if I find other like-minded individuals or maybe some other better ideas will come out of this (Tho, I expected to see more people posting what they themselves think that can be improved I guess we will see how its going to go, its been just few hours)

nrdxp · October 22, 2021, 8:40pm

I know @blaggacao has been recently pushing for some kind of json schema validation for Nix. Something like a builtins.validateJSONSchema. If something like this does catch on, we could use a json schema to define what the flake can output. As a bonus, since we now control the schema ourselves, we can make Nix error instead of just warn if the schema is broken.

blaggacao · October 22, 2021, 8:44pm

github.com/NixOS/nix

builtins.validateAsJSON

NixOS:master ← kamadorueda:builtins-validate-json-schema

opened 10:29AM - 18 Oct 21 UTC

kamadorueda

+2761 -0

- Add https://github.com/pboettch/json-schema-validator as a dependency …`json-schema-validator` is a library that uses `nlohmann` internally, which is being used in other components of Nix already `json-schema-validator` is MIT licensed, complies with the latest draft of JSON schema and is the second one with most stars on GitHub - Add a new builtin for validating data with a JSON schema. This builtin receives two arguments: `schema` and `data`, both in Nix format - Demo: ```nix let schema = { title = "A person"; properties = { age = { description = "Age of the person"; type = "number"; minimum = 1; maximum = 200; }; name = { description = "Complete Name for the person"; first.type = "string"; last.type = "string"; required = [ "first" "last" ]; type = "object"; }; }; required = [ "name" "age" ]; type = "object"; }; in map (validateAsJSON schema) [ { } { age = 24; name.first = "Jane"; } { age = 24; name.first = "Jane"; name.last = "Doe"; } ] ``` ```bash $ nix-instantiate --eval --strict test.nix [ { success = false; value = "At '/', required property 'name' not found in object"; } { success = false; value = "At '/name', required property 'last' not found in object"; } { success = true; value = { age = 24; name = { first = "Jane"; last = "Doe"; }; }; } ] ``` - This demo was also added to the documentation in a simplified form: renderized: ![image](https://user-images.githubusercontent.com/47480384/140006637-93e9e0e0-866a-4a80-8b03-2af0e06702fd.png) - Added language tests Mentions #5392

kamadorueda · October 22, 2021, 8:47pm

( Any feedback on that PR is very appreciated )

cdepillabout · October 23, 2021, 1:00am

After using flakes for a month or two, here are my annoyances as an end user:

While there is nice support with flakes for building on different systems (x86_64-linux, aarch64-linux, etc), there is no built-in support for cross-compiling, or easily specifying the system you want to cross-compile to.

[flakes] Where do cross compiled packages go? · Issue #5157 · NixOS/nix · GitHub

At work we worked around this by specifying both native packages and cross compiled packages as different packages exposed by our flake, but it feels somewhat hacky.
There is no good way to pass options to flakes: Passing options to flakes

This seems like a really big blow to usability. Judging from that thread, it seems like passing options to nix-build is a relatively common thing to do.
The nix registry stuff seems like it is nice, but by default it doesn’t pin nixpkgs to a specific commit (or even a Hydra channel). This means if you do something like nix shell nixpkgs#foo, it is possible you have to build from source, and you may just have to download the latest Nixpkgs master every time.

I created a thread explaining how to pin to a commit on one of the channels, but I feel like something like this should be done by default: How to pin nix registry nixpkgs to release channel
nix flake show doesn’t handle IFD in a graceful way: Don't allow IFD in flakes by default by edolstra · Pull Request #5253 · NixOS/nix · GitHub

Instead of immediately failing out when reaching IFD, it would be great if nix flake show instead displayed an error message only for those attributes that require IFD, but continued on showing the rest of the attributes defined in the flake.

NobbZ · October 23, 2021, 7:15am

nix flake show handles any error it encounters this way. It just fails. So you are totally right, instead of printing errors about unexpected things it should warn properly.

Instead of “expected derivation” (without even telling what attribute it is currently evaluating) it could display the attribute in red and instead of name/description it could write out the warning/error it encountered…

NobbZ · October 23, 2021, 9:25am

Just stumbled over another one…

nix edit should really be renamed. It does not allow editing at all! cat, less or maybe page would be so much better options for that subcommands name…

If nix edit is to be kept, it should really make sure that what ever is opened is not in the store! Even if I do nix edit .#foo it will copy . to the store first and then open my editor on the file from the store…

edit

Opened an issue for this: flakes: `nix edit` does not actually allow to edit the opened file · Issue #5417 · NixOS/nix · GitHub

cdepillabout · November 29, 2021, 5:49am

Here’s another annoyance I forgot about originally:

The nix shell command doesn’t set the environment variable IN_NIX_SHELL: Set `IN_NIX_SHELL` or similar for `nix shell` · Issue #3862 · NixOS/nix · GitHub

As an end-user, it is somewhat unfortunate that this issue was closed unilaterally, despite there appearing to be a lot of support for it being fixed.

abathur · November 29, 2021, 6:18am

I agree with this general perception, but I think this is an issue of people talking past each other? I suspect there needs to be some other mechanism than IN_NIX_SHELL that can more granularly reflect the conditions? I’ve wondered if maybe just NIX_COMMAND=<invocation> would suffice, but I haven’t tried to survey how people are using IN_NIX_SHELL to understand it.

fgaz · November 30, 2021, 10:29am

I think most people are using $IN_NIX_SHELL to customize their shell prompt

abathur · November 30, 2021, 3:11pm

This is what I assume, since it is how I use it. I get the impression that the objections are about avoiding overloading the semantics of the env and stuff like Set `IN_NIX_SHELL` or similar for `nix shell` · Issue #3862 · NixOS/nix · GitHub.

Assuming the latter is solved by clearing any such envs from the build environment, it seems like an env reflecting the invocation in all of the ~subshell-creating-commands would defuse the semantics concern, and avoid needing a new env for each distinct semantic. (with the downside of not being able to preserve an old env for backwards compact if a command is renamed?)