Flakes: Re-locking necessary at each evaluation when import sub-flake by path

I have a flake dev-env that itself references a sub-flake myNixpkgs from the same local git repo by relative path.
As the sub-flake input is also locked in the main flake.lock file, its path is recorded in there.

Unfortunately I’ve discovered that each evaluation of the flake now changes the flake.lock file again.

$ nix develop
warning: Git tree '[…]' is dirty
warning: updating lock file '[…]/flake.lock':                                                                                                           
• Updated input 'myNixpkgs':                                                                                                                                                                 
    'path:/nix/store/whdbxck4f86n3244v9pqmvl81p1qsg65-source/flakeSupport/myNixpkgs?lastModified=1&narHash=sha256-6+DqW6+6Bf1Ea3t8SaGhB9zfx2fkl3ZtolWM8LC7Oto=' (1970-01-01)
  → 'path:/nix/store/3y3cfmdy6vnv4pgl30r883j37p98nikv-source/flakeSupport/myNixpkgs?lastModified=1&narHash=sha256-6+DqW6+6Bf1Ea3t8SaGhB9zfx2fkl3ZtolWM8LC7Oto=' (1970-01-01)
warning: Git tree '[…]' is dirty

$ nix develop
warning: Git tree '[…]' is dirty
warning: updating lock file '[…]/flake.lock':
• Updated input 'myNixpkgs':
    'path:/nix/store/3y3cfmdy6vnv4pgl30r883j37p98nikv-source/flakeSupport/myNixpkgs?lastModified=1&narHash=sha256-6+DqW6+6Bf1Ea3t8SaGhB9zfx2fkl3ZtolWM8LC7Oto=' (1970-01-01)
  → 'path:/nix/store/69skdc3dhdmf5g0iq7gh6r7pr2jic1r7-source/flakeSupport/myNixpkgs?lastModified=1&narHash=sha256-6+DqW6+6Bf1Ea3t8SaGhB9zfx2fkl3ZtolWM8LC7Oto=' (1970-01-01)
warning: Git tree '[…]' is dirty

Especially with direnv this is a no-go behaviour, as the flake is re-evaluated after each command run in the shell.

The reason for this is quite clear: For pure evaluation, the whole git repo of the flake is copied to the nix store. Its out path there depends on the repo’s content. So the following happens:

  1. nix copies the whole repo to store
  2. nix detects a path mismatch of the relatively referenced flake file
  3. nix updates the path in the lock file
  4. this causes the repo contents to change…
  5. …causing the out path of the repo to change when it is copied to the store at the next evaluation. GOTO 1.

I mainly wonder whether I am holding this wrong and there is a better solution to this? Or is this still one of the cases why flakes are experimental and the issue about relative flake path references is still open? Allow flakes to refer to other flakes by relative path · Issue #3978 · NixOS/nix · GitHub

I’d be so bold to say that my approach is a rather obvious one, so even if there is a better approach, it is easy to do things the wrong way.


the flake.nix at the repo top-level:

{
  inputs = {
    myNixpkgs = {
      url = "./flakeSupport/myNixpkgs";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    nixpkgs = {
      #url = "github:NixOS/nixpkgs/nixos-23.05";
      url = "nixpkgs";
    };
    nixpkgsOld.url = "github:NixOS/nixpkgs/nixos-21.05";
    poetry2nix = {
      url = "github:nix-community/poetry2nix";
      inputs.nixpkgs.follows = "myNixpkgs";
    };
  };

  outputs = { self, myNixpkgs, poetry2nix, ... }:
    let
      inherit (myNixpkgs) lib;
      supportedSystems = [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin" ];
      forAllSystems = lib.genAttrs supportedSystems;
      pkgs = forAllSystems (system: myNixpkgs.legacyPackages.${system});
      p2n = forAllSystems (system: poetry2nix.legacyPackages.${system});
    in
    {
      devShells = forAllSystems (system: {
        default = pkgs.${system}.mkShellNoCC {
          packages = with pkgs.${system}; [
            (p2n.${system}.mkPoetryEnv poetryCommonOpts.${system})
            pkgs.${system}.python38
            poetry
            openssl_1_1
          ];
        };
      });
    };
}

The sub-flake flakeSupport/myNixpkgs/flake.nix:

# vaguely inspired by https://github.com/numtide/nixpkgs-unfree/blob/9545d844027c1b91b14b19d225856efc931b22b2/flake.nix
{
  description = "nixpkgs reexported with some required configuration, e.g. permitted insecure packages";

  outputs = inputs@{self, nixpkgs}:
  let
    inherit (nixpkgs) lib;
    # re-use same supported systems as in the upstream nixpkgs
    systems = lib.systems.flakeExposed;
    forEachSystem = lib.genAttrs systems;

    nixpkgsConfig = {
      permittedInsecurePackages = [
        "openssl-1.1.1w"
      ];
    };
  in
  nixpkgs // { legacyPackages = forEachSystem (system:
    import nixpkgs {
      inherit system;
      config = nixpkgsConfig;
    });};
}

Update: I was able to work around the issue by changing the relative sub-flake reference to git+file:

myNixpkgs = {
      #url = "./flakeSupport/myNixpkgs";
      url = "git+file:./?dir=flakeSupport/myNixpkgs";
      inputs.nixpkgs.follows = "nixpkgs";
    };

I specifically say work around because this does not just look suspiciously quirky, but also requires changes in the sub-flake to be commited before they are picked up when evaluating the top-level flake.
This cannot be deemed an intended workflow and design choice.

You could also try setting the flake type to path:

      url = "path:./flakeSupport/myNixpkgs";

This should work exactly as you expect and won’t require committing anything before the changes are reflected.

Not quite— the relative path is relative to the invocation point, not the flake file. This is almost certainly a design error, but the result right now is that this only works when you’re invoking the flake from within a local checkout of it; it doesn’t work at all if you’re referencing any kind of remote or registry-based flake.

1 Like

Oh wow yes, that’s right. How annoying.

Thanks, for my workflow of a dev shell that actually works well-enough – but in principle is broken indeed as a concept.

I was struggling a bit on what is going on there, but a diff cleared things up:

diff --git a/flake.lock b/flake.lock
index c91b30f..9958335 100644
--- a/flake.lock
+++ b/flake.lock
@@ -27,11 +27,11 @@
       "locked": {
         "lastModified": 1,
         "narHash": "sha256-6+DqW6+6Bf1Ea3t8SaGhB9zfx2fkl3ZtolWM8LC7Oto=",
-        "path": "./flakeSupport/myNixpkgs",
+        "path": "/nix/store/z1ggrmm333xfgblvn4cx8zygjryhkgbv-source/flakeSupport/myNixpkgs",
         "type": "path"
       },
       "original": {
-        "path": "./flakeSupport/myNixpkgs",
+        "path": "/nix/store/z1ggrmm333xfgblvn4cx8zygjryhkgbv-source/flakeSupport/myNixpkgs",
         "type": "path"
       }
     },
diff --git a/flake.nix b/flake.nix
index 7cde2b0..c3f5c1a 100644
--- a/flake.nix
+++ b/flake.nix
@@ -1,7 +1,7 @@
 {
   inputs = {
     myNixpkgs = {
-      url = "path:./flakeSupport/myNixpkgs";
+      url = "./flakeSupport/myNixpkgs";
       #url = "git+file:./?dir=flakeSupport/myNixpkgs";
       inputs.nixpkgs.follows = "nixpkgs";
     };

So url = "path:./a/relative/path" causes this path to still land as relative in the lock file, while url = "./a/relative/path/" causes it to be converted into an absolute path at locking time – which can change and then result in a behaviour as I’ve described.

But how does this fit the nix docs?

Flakes corresponding to a local path can also be referred to by a direct path reference, either /absolute/path/to/the/flake or ./relative/path/to/the/flake (note that the leading ./ is mandatory for relative paths to avoid any ambiguity).

The semantic of such a path is as follows:

If the directory is part of a Git repository, then the input will be treated as a git+file: URL, otherwise it will be treated as a path: url;

My flake lives in a git repo, so the sub-flake does as well – shouldn’t this then be treated as git+file then?
Okay, let’s just assume they only mean top-level references to git repos, then still: it should be treated as a path: URL. And as my specified path is relative, so should the path url one be – unless the docs mention any absolute resolving.

Maybe I should really bring this up as a Nix bug…

Yeah this really should be a bug report. I tried to search for a duplicate on github but couldn’t find one. Maybe this one? Not sure if it’s the exact same issue.

Welp, for whatever reason the relative path solution stopped working. I did not even change anything in the flake.nix:

error: cannot fetch input 'path:./flakeSupport/myNixpkgs?lastModified=1&narHash=sha256-6+DqW6+6Bf1Ea3t8SaGhB9zfx2fkl3ZtolWM8LC7Oto=' because it uses a relative path

The input spec still looks like the following:

    myNixpkgs = {
      url = "path:./flakeSupport/myNixpkgs";
      #url = "git+file:./?dir=flakeSupport/myNixpkgs";
      inputs.nixpkgs.follows = "nixpkgs";
    };

Removing the leading ./ of the path resolves the error:

    myNixpkgs = {
      url = "path:flakeSupport/myNixpkgs";
      #url = "git+file:./?dir=flakeSupport/myNixpkgs";
      inputs.nixpkgs.follows = "nixpkgs";
    };

This is of course still a relative path – which the error claims to be the problematic aspect, but suddenly it does not complain anymore.
I also wonder how the previous path with a ./ ever worked. Because it definitely did work at the start.

This became a Nix issue now.

After a system reboot, the new path input does not work anymore either:

$ nix develop                                                                                                                                                                        130 ↵
error: cannot fetch input 'path:flakeSupport/myNixpkgs?lastModified=1&narHash=sha256-6+DqW6+6Bf1Ea3t8SaGhB9zfx2fkl3ZtolWM8LC7Oto=' because it uses a relative path

I’m fed up and switching back to the git+file approach despite all its inconveniences – at least it works somehow.

url = "git+file:./?dir=flakeSupport/myNixpkgs";