My painpoints with flakes

1

I wanted to play with flakes, and therefore kumped into them, roughly after summer holiday.

So I’m using flakes since ~3 months now.

I really like the evaluation cache, and its well defined interface about inputs and outputs. Though there are also some things, that are troubling me and I want to talk about them in a broader context than the chat, perhaps I’m still misunderstanding things about it, perhaps my view makes people think differently and causes a change? Or perhaps its just that someone finds my thoughts and considerations a nice read.

Despite my thoughts, I will continue to use flakes for my personal stuff, but in their current form probably not for work (see below)

The global flake registry

It is there, and I can not do anything against it.
flake:nixpkgs points at github:nixos/nixpkgs (master branch), while flake:nixpkgs/release-19.09 points to a last years fork of github:nixos/nixpkgs/release-19.09 by edolstra.

Also there are other entries in the global registry, that seem to be random personal projects of edolstra (which is fine, though why are those in, but not eg. github:numtide/flake-utils).

Also flake:nixpkgs pointing at master makes it barely usable and running the same nix shell nixpkgs#hello -c hello results in redownloading and re-evaluation of the attributes quite often, as the evaluation cache can’t be re-used with the new commit. I think individual branches wouldn’t advance that fast and therefore be able to re-use evaluation cache more often.

Overwriting flake:nixpkgs with a local registry entry (eg. github:nixos/nixpkgs/nixos-20.09) seems to cause much less downloads and re-evaluations. And after using nix flake pin there haven’t been any downloads anymore, though then the local registry entry points to github:nixos/nixpkgs/$sha1, the information to which branch it pointed is lost. To “advance” the registry entry I have to actually know where it pointed to before and overwrite, and then pin again.

Also once I have a local entry and create a flake it will create a lock file according to my local entry, and when contributors with a different local entry (or no local entry at all) would update or recreate the lockfile, all of a sudden the lockfile would point not only to a different commit, but a different branch then before!

Forced VCS

I love git, I like to use flakes, especially as they make pinning easy.

I’d like to use flakes for defining my dev-env at work, though I can not commit any nix file to the repo, I have to sideload them.

So I do not have any choice than to not use flakes, but instead manually fiddle around with rev-shas in builtins.fetchGit and builtins.fetchTarball as well as the slowness of IFDs (which luckily aren’t that bad when it is just a nixpkgs commit within a shell.nix).

I’m missing some “ad-hoc” mode, that enables me to use version controlled flakes at the root, of course it is impossible to use those as inputs in any other kind of flake (even other ad-hoc ones).

Manipulation of registry

For old NIX_PATH/channel style nix files, I was able to pin channels prior to the nixos-rebuild and home-manager through nix-shell magic in the NIX_PATH and running the mentioned commands within the pinned environment. At the same time I used either model to set the NIX_ENV environment variable for the system/user to roughly match the inputs by doing environment = { NIX_PATH = "nixpkgs=${<nixpkgs>}"; } (pseudo code), I’m missing something equivalent for flakes to be able to alter system/user wide pins through my configuration.

Also I am missing a way to delete global entries, as I do not consider them important for me.

Local registry

I have not yet found any hint about whether “local” entries are per user or systemwide.

16 Likes
2

A doubt I have is if I use flakes to manage my system and use nixos-unstable as input, when I tell flakes to update it will update nixos-unstable to the master of the github repo or it will update to the last version that succeeded to build correctly in Hydra?

3

As I understand the process, the branches represent what you would get from the channel URLs. As the branch will only be updated after it’s corresponding “pre” branch has been successfully evaluated and tested.

For the unstable channels that’s staging and staging-next, AFAIK, for the numbered stable those are prefixed with “release” rather than nixos/nixpkgs.

But again, this is only from my understanding.

4

I saw that nixos-unstable branch head is in the same commit of nixos-unstable channel.

5

Today I realised another thing:

nix flake check does not accept an attribute

If one has more than a single check in a flake, it seems not to be possible to run it selectively.

In a playground project of mine I have basically 3 checks (pseudoecode and simplified):

checks = {
  nix-format = runTest "nixpkgs-fmt --check {,**/}*.nix";
  rust-format = runTest "cargo fmt -- --check";
  build = self.packages.${system}.theProject;
};

Now I can run them using nix flake check, which will do all the checks. Though the build check is quite costly, therefore I’d prefer to only run it every once in a while, while the other 2 could be part of my commit hooks, though:

nix flake check .#nix-format
error: --- Error ----------------------------------------------------------------------------------------------------------------- nix
unexpected fragment 'nix-format' in flake reference '.#nix-format'
1 Like
6

You can nix build a check

nix build .#checks.x86_64-linux.nix-format

but it would be more convenient, if like nix build with packages, it could handle the arch for you.

1 Like
7

Thanks, that helps indeed, though it would be nice to have it in the interface of nix flake check.

2 Likes
8

YIL (the Y stands for “yesterday”), that one can indeed pin entries in the registry declaratively!

  1. I had to make sure that the module system will see the inputs (not sure if this way was idiomatic though, but it works for now: provide inputs to the modules · NobbZ/nixos-config@7e4d374 · GitHub
  2. set nix.registry.<name>.flake to whatever one wants to pin for that registry entry: pin nixpkgs in flake registry · NobbZ/nixos-config@bda3cef · GitHub
1 Like
9

I use flakes to build my system, home-manager and now some dev-env.

I agree with all the comments on global registry:

  • strange content (projects and branches)
  • no way to remove unused stuff

Indeed I use the nix.registry.<name>.flake option to pin nixpkgs so that all references for my system and users is using the same version.

Basically my organization is:

  • One configuration flake containing:
    • All inputs to upstream branches (nixpkgs, home-manager, …) used to pin everything with the flake.lock
    • A custom package set for local stuff
    • An overlay to help combining all the packages
    • Some nixosModules.* for my custom services and to generalize my configs
    • Some nixosConfigurations.* to handle my system configurations
    • Some homeModules.* to generalize my home configs (following the nixos* pattern)
    • Some homeConfigurations.* to handle home-manager configs

Part of my packages a flake-mgr script to:

  • Update the lock file of all flake inputs
  • Build/Switch homeConfigurations
  • Build/Test/Boot/Switch nixosConfigurations

This handles my system and user persistent configurations.

For my dev-env, I write another flake depending on the first to see my overlay and pin all upstream flakes. I simply defines a devShell.

BTW: My flakes are currently not in a git repository, and it still works.

Pro Tip: when defining a dev-env, dont put your flake.nix file at the root … otherwise all the folder and subfolders will be pushed to the nix-store and caching will be very bad. I use:

  • Main Project
    • .envrc Direnv config using ./nix/flake.nix as a source
    • nix
      • git
      • flake.nix Used to pin deps, and provide CI context
      • flake.lock
    • app1 Git clone of a backend app
      • .git
      • nix Specific env for this app
        • default.nix
      • Cargo.toml
      • src
    • app2 Git clone of a frontend app
      • .git
      • nix Specific env for this app
        • default.nix
      • src
      • package.json
      • package.lock
2 Likes
10

I can’t reproduce…

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs/unstable";

  outputs = { self, nixpkgs }:
    let
      pkgs = nixpkgs.packages.x86_64-linux;
    in
      {
        devShell.x86_64-linux = pkgs.mkShell {
          buildInputs = [ pkgs.go_1_14 pkgs.nixpkgs-fmt pkgs.lefthook pkgs.docker-compose pkgs.entr ];
        };
      };
}

This is my .flake.nix at the company project:

nix develop
warning: unknown setting 'extra-sandbox-paths'
error: --- Error ---------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
source tree referenced by 'git+file:///home/nobbz/path/to/repo?ref=branch-name&rev=0e69a288c41b89dec8566b6d109f6c7a9e3fabaf' does not contain a '/flake.nix' file

If though I git add the flake.nix, it will ask for the flake.lock.

So maybe you can use flakes for things that aren’t in git yet, but I’m working with things that are already in git, and I’d prefer to have the dev shell defined through flakes, to not have to switch between niv and flake or even manually updating SHAs in the fetchGitHub or similar call depending on the flake…

1 Like
11

Ah yes, if it is inside a git repo, it is forcing the non dirty behavior

1 Like
12

Would you mind sharing the implementation of that runTests function? I’m struggling to find documentation for how to perform a check like cargo fmt -- --check or cargo clippy. It seems like the check has to be a derivation but I don’t know how to make a derivation that runs a shell command and then how/if flakes would respect the error code of the command.

1 Like
13

I don’t actually have a runTest function, that was just part of the simplification for the post, but you can take a look at the testing derivations:

1 Like
14

I’d like to address some points regarding the registry from your original post, because I have some workarounds for these problems. First of all I show the configuration for Nix:

# inputs are the flake inputs
{
  nix = {
    package = pkgs.nixFlakes;

    registry = { # (1)
      self.flake = inputs.self; # (2)
      nixpkgs = {
        from = {
          id = "nixpkgs";
          type = "indirect";
        };
        to = {
          owner = "NixOS";
          repo = "nixpkgs";
          rev = inputs.nixpkgs.rev; # (3)
          type = "github";
        };
      };
    };

    # (4)
    extraOptions = ''
      experimental-features = nix-command flakes ca-references
      flake-registry = /etc/nix/registry.json
    '';
  };
}

  1. Using nix.registry you can set entries in the local registry which is stored in /etc/nix/registry.json.

  2. First I register the system configuration itself as a flake in the registry under the name self. This prevents me from accidentally using it as a flake input elsewhere because the name self is reserved. At the same time I can export legacyPackages with all my local overlays from the self flake and get instant Nix shells.

  3. I could do the same for nixpkgs, i.e. nix.registry.nixpkgs.flake = inputs.nixpkgs;, however, that would write the local store path to the registry and when I then nix flake update in one of my other flakes it will write the store path into the lock file and others will not be able to build the flake anymore. Instead I just write the registry entry by hand but pin the rev to the revision of the nixpkgs that the system was built from.

  4. The global flakes registry can be configured in nix.conf and defaults to https://github.com/NixOS/flake-registry/raw/master/flake-registry.json. It accepts a URI or a local path. Unfortuantely, you can’t leave the field empty (flake-registry =) and settings it to /dev/null also doesn’t work because a special JSON format is expected. To this end I just use the local registry located at /etc/nix/registry.json again. Now all entries in my registry are duplicate but at least the global registry is gone.

4 Likes
15

After I originally posted my way to pin, I also read deeper into the documentation of nix.registry.* though I really didn’t understand the from and to attributes. Therefore I left everything as I have it.

Though I can not really understand your #3, as I never run nix flake update on other people projects and in my own projects I never rely on flake:nixpkgs but instead github:nixos/nixpkgs/<branch> to avoid any registry mismatches from the get-go.

I’m pretty sure I’ve said it several times, though missed it in this thread it seems. The registry is a nice thing when used on the terminal for quick accessing some often used repos. but in actual flake.nix I really wish flake:* URLs as well as implicit usage in the arguments to outputs function would be forbidden…

1 Like
16

You can build your flake from a path rather than VCS if you are explicit about it:

nix run path:$PWD
Hello World

# flake isn't in VCS -> fail
nix run git+file:$PWD
error: --- Error ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
source tree referenced by 'git+file:/tmp/test?ref=master&rev=407a304d0991580a37cea8d53adefb65b447387b' does not contain a '/flake.nix' file
1 Like
17

What I wanted to express is that I’d like to have a flake.nix which is not version controlled within a git repository to define its corresponding dev-env. Currently I have to use nix-shell for my work related projects, though I’d prefer to have unified nix devel regardless if its work or private project.

1 Like
18

Agree with NobbZ. nix develop should relax the rules a little. Why should a flake.nix have to be in version control? At the moment I can’t have my flake.nix in .gitignore but must stage and commit it and only then is it happy. This feels like a workflow speedbump to any project that isn’t happy having flake.nix committed into version control… and try to remember not to push the committed flake.nix. Maybe I’m missing something but as a workflow it seems sub-optimal?

3 Likes
19

If I don’t want to add the flake.nix inside my git repo I just call this way:

nix develop path:$PWD

If flake.nix is in other folder I call this way:

nix develop path:/my/flake/folder
6 Likes
20

That way Nix will copy the entire tree into the Nix store: Copy local flakes to the store lazily · Issue #3121 · NixOS/nix · GitHub

I’d suggest to use Git pre-commit and post-commit hooks to unstage and restage the flake.{nix,lock} files.