Can I use flakes within a git repo without committing flake.nix?

NobbZ · May 30, 2023, 9:41am

It is not tied to git.

But the “type inference” is “version control first”, which in my opinion is a reasonable approach.

Infinisil · May 30, 2023, 10:51am

Ah I didn’t realize it worked even without git.

However, at least the evaluation cache is tied to git.

NobbZ · May 30, 2023, 11:12am

As far as I remember it is tied to any identifiable snapshot in the underlying flake type. A path can’t be snapshotted, nor can dirty-git commits.

But a “clean” mercurial or whatever should again benefit from the eval cache.

srghma · May 30, 2023, 1:27pm

@NobbZ

“type inference” is “version control first”

what it means? (cannot find, could You teach me, please)

You mean, it is using the .git to cache the "type inference"? to which folder? /home/user/.cache ?

NobbZ · May 30, 2023, 2:24pm

If you do not specify a type when referring to a flake, then nix will try to infer the type, or scheme if you want it like this.

Nix will then check if the given location is under supported source control, and if this is true, this source control will be the infered “type” or “scheme” of the flake.

If there is no such supported VCS is found, path will be used.

If the specified location actually doesn’t exist as a local filesystem object, then flake will be used and that will be looked up in the registry.

You can always specify the schema manually.

srghma · May 30, 2023, 2:52pm

So, the scheme can be either git or path or flake or mercury?

What is a “scheme”?
Could you send the link to parts in source code?

If I search nix flake scheme in Google it tells that scheme is { input, output }

NobbZ · May 30, 2023, 2:55pm

Sorry, no source code, only documentation.

https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake.html#types

srghma · May 30, 2023, 3:20pm

ok, now I know that

nix develop . on a folder with .git is identical to nix develop git+file:. (NOTE: not to nix develop git:. and not to nix develop file:.)
and nix develop . on a folder without .git is identical to nix develop path:.

Example

I have an empty dir with two files (from Super fast nix-shell in Flakes - NixOS Wiki), git add was not yet called

shell.nix

{ pkgs ? import <nixpkgs> { } }:
with pkgs;
mkShell {
  buildInputs = [
    nixpkgs-fmt
  ];

  shellHook = ''
    # ...
  '';
}

flake.nix

{
  description = "my project description";

  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let pkgs = nixpkgs.legacyPackages.${system}; in
        {
          devShells.default = import ./shell.nix { inherit pkgs; };
        }
      );
}

$ git init
Initialized empty Git repository in /home/srghma/projects/hello/.git/

$ git status
On branch main

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        flake.nix
        shell.nix

nothing added to commit but untracked files present (use "git add" to track)

$ nix develop .
warning: Git tree '/home/srghma/projects/hello' is dirty
error: getting status of '/nix/store/0ccnxa25whszw7mgbgyzdm4nqc0zwnm8-source/flake.nix': No such file or directory

$ nix develop git+file:.
warning: Git tree '.' is dirty
error: getting status of '/nix/store/0ccnxa25whszw7mgbgyzdm4nqc0zwnm8-source/flake.nix': No such file or directory

$ nix develop path:.
warning: creating lock file './flake.lock'

[srghma@machine:~/projects/hello]$

$ cat flake.lock
{
  "nodes": {
    "flake-utils": {
      "inputs": {
        "systems": "systems"
      },
      "locked": {
        "lastModified": 1681202837,
        "narHash": "sha256-H+Rh19JDwRtpVPAWp64F+rlEtxUWBAQW28eAi3SRSzg=",
        "owner": "numtide",
        "repo": "flake-utils",
        "rev": "cfacdce06f30d2b68473a46042957675eebb3401",
        "type": "github"
      },
      "original": {
        "owner": "numtide",
        "repo": "flake-utils",
        "type": "github"
      }
    },
    "nixpkgs": {
      "locked": {
        "lastModified": 1685399834,
        "narHash": "sha256-Lt7//5snriXSdJo5hlVcDkpERL1piiih0UXIz1RUcC4=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "58c85835512b0db938600b6fe13cc3e3dc4b364e",
        "type": "github"
      },
      "original": {
        "id": "nixpkgs",
        "type": "indirect"
      }
    },
    "root": {
      "inputs": {
        "flake-utils": "flake-utils",
        "nixpkgs": "nixpkgs"
      }
    },
    "systems": {
      "locked": {
        "lastModified": 1681028828,
        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
        "owner": "nix-systems",
        "repo": "default",
        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
        "type": "github"
      },
      "original": {
        "owner": "nix-systems",
        "repo": "default",
        "type": "github"
      }
    }
  },
  "root": "root",
  "version": 7
}
$ rm -frd flake.lock
$ nix develop path:.
warning: creating lock file './flake.lock'

[srghma@machine:~/projects/hello]$
exit
$ cat flake.lock
{
  "nodes": {
    "flake-utils": {
      "inputs": {
        "systems": "systems"
      },
      "locked": {
        "lastModified": 1681202837,
        "narHash": "sha256-H+Rh19JDwRtpVPAWp64F+rlEtxUWBAQW28eAi3SRSzg=",
        "owner": "numtide",
        "repo": "flake-utils",
        "rev": "cfacdce06f30d2b68473a46042957675eebb3401",
        "type": "github"
      },
      "original": {
        "owner": "numtide",
        "repo": "flake-utils",
        "type": "github"
      }
    },
    "nixpkgs": {
      "locked": {
        "lastModified": 1685399834,
        "narHash": "sha256-Lt7//5snriXSdJo5hlVcDkpERL1piiih0UXIz1RUcC4=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "58c85835512b0db938600b6fe13cc3e3dc4b364e",
        "type": "github"
      },
      "original": {
        "id": "nixpkgs",
        "type": "indirect"
      }
    },
    "root": {
      "inputs": {
        "flake-utils": "flake-utils",
        "nixpkgs": "nixpkgs"
      }
    },
    "systems": {
      "locked": {
        "lastModified": 1681028828,
        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
        "owner": "nix-systems",
        "repo": "default",
        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
        "type": "github"
      },
      "original": {
        "owner": "nix-systems",
        "repo": "default",
        "type": "github"
      }
    }
  },
  "root": "root",
  "version": 7
}

$ rm -frd flake.lock
$ git add --all
$ git status
On branch main

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
        new file:   flake.nix
        new file:   shell.nix

$ nix develop git:.
ssh: Could not resolve hostname git: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
warning: could not read HEAD ref from repo at 'git:.', using 'master'
evaluating derivation 'git:.#devShells.x86_64-linux.default'ssh: Could not resolve hostname git: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
error: program 'git' failed with exit code 128
(use '--show-trace' to show detailed location information)
$ nix develop file:.
error: unable to download 'file:.': URL using bad/illegal format or missing URL (3)
(use '--show-trace' to show detailed location information)

$ nix develop git+file:.
warning: Git tree '.' is dirty
warning: creating lock file './flake.lock'
warning: Git tree '.' is dirty

[srghma@machine:~/projects/hello]$
exit

Example 2 (use `nix develop git+file:.` on a folder without `.git`)

$ rm -frd flake.lock
$ rm -frd .git
$ nix develop .
warning: creating lock file '/home/srghma/projects/hello/flake.lock'

[srghma@machine:~/projects/hello]$
exit
$ ls
flake.lock  flake.nix  shell.nix
$ rm -frd flake.lock
$ nix develop git+file:.
ssh: Could not resolve hostname file: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
warning: could not read HEAD ref from repo at 'file:.', using 'master'
evaluating derivation 'git+file:.#devShells.x86_64-linux.default'ssh: Could not resolve hostname file: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
error: program 'git' failed with exit code 128
(use '--show-trace' to show detailed location information)
$ nix develop git:.
ssh: Could not resolve hostname git: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
warning: could not read HEAD ref from repo at 'git:.', using 'master'
evaluating derivation 'git:.#devShells.x86_64-linux.default'ssh: Could not resolve hostname git: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
error: program 'git' failed with exit code 128
(use '--show-trace' to show detailed location information)

$ nix develop file:.
error: unable to download 'file:.': URL using bad/illegal format or missing URL (3)
(use '--show-trace' to show detailed location information)

Question:

how git is used for “type inference”?
Why not to make the nix develop . on a folder with .git identical to nix develop path:. instead?

I still dont understand what throwing No such file or directory error in

$ nix develop .
warning: Git tree '/home/srghma/projects/hello' is dirty
error: getting status of '/nix/store/0ccnxa25whszw7mgbgyzdm4nqc0zwnm8-source/flake.nix': No such file or directory

gives us

NobbZ · May 30, 2023, 3:33pm

It ensures that your flake is self-contained.

srghma · May 30, 2023, 3:48pm

What means self-contained?

NobbZ · May 30, 2023, 3:53pm

Same as “self-describing” but with “containing things” rather than “describing things”.

misuzu · May 30, 2023, 4:43pm

At least the flakes are tied to the evaluation cache. That should have been two separate features. Flakes could already be stable for several NixOS releases, but instead “the evaluation cache feature” is still not ready and so is flakes.

Infinisil · May 30, 2023, 4:56pm

I agree that it’s a problem that Flakes is tied to the evaluation cache. There’s a proposal here to untangle this, and I have another proposal myself in the works too.

However I don’t think Flakes is ready to be stabilized even considering this. There’s a number of significant issues with Flakes yet unresolved:

The fact that flake.nix is not actually a Nix file: What language is flake.nix written in? · Issue #4945 · NixOS/nix · GitHub
The whole problem with system: Provide builtins.currentSystem as a input to flakes or make system a parameter · Issue #3843 · NixOS/nix · GitHub
Not being able to pass arguments to flakes: Flake arguments · Issue #2861 · NixOS/nix · GitHub
The problem of recursive input locking causing multiple incompatible Nixpkgs versions to be used: 1000 instances of nixpkgs

I think these are the major ones at least

NobbZ · May 30, 2023, 5:32pm

As much as I agree that flakes are not ready yet, and in the danger to fully derail this thread, I might have an unpopular and extreme opinion:

Either do a breaking change that ruptures contintents within the remainer of the year, or just realease them as they are. The “expermintal” status became a farce and everyone uses them as a stable tool. We are at a point where the “experimental” thing became basically just that thing that you enable anyway right after install to be able to use nix proper, or alternatively you use the determinate installer which by default enables flakes and nix-commands…

TL’DR: Break them now or just release flakes…

roberth · May 30, 2023, 5:32pm

Also some serious lock file issues:

Updating of individual inputs is broken
The lock file will grow indefinitely through mutual dependency updates
Inputs aren’t lazy when locking, penalizing development-only inputs, test-only inputs and other use case specific inputs. Not scalable.

I think all three should be solved by first un-flattening the lock file (ie getting transitive deps from dependency locks by default) and resolving follows at evaluation time.

(EDIT: my own thoughts, not decided by the Nix team)

roberth · May 30, 2023, 5:34pm

Changing the locking may well be such a change. You may need the newest Nix to read new lock files, unless we keep copying the transitive stuff for a migration window.

(EDIT: hypothesizing and, again, speaking for myself)

srghma · May 30, 2023, 5:42pm

More tests

Test 1 (throws `expected a string but got a thunk`)

$ git status
On branch main

No commits yet

Changes to be committed:
 (use "git rm --cached <file>..." to unstage)
       new file:   asdfasdf.nix
       new file:   flake.nix
       new file:   shell.nix

$ cat flake.nix
{
description = "my project description";

inputs.flake-utils.url = import ./asdfasdf.nix;

outputs =
 { self, nixpkgs, flake-utils }:
 flake-utils.lib.eachDefaultSystem
   (system:
     let pkgs = nixpkgs.legacyPackages.${system}; in
     {
       devShells.default = import ./shell.nix { inherit pkgs; };
     }
   );

}
$ cat asdfasdf.nix
"github:numtide/flake-utils"
$ nix develop .
warning: Git tree '/home/srghma/projects/hello' is dirty
error: expected a string but got a thunk at /nix/store/ny5h4254c0x95xv6nniqq89q83vpw0a7-source/flake.nix:24:1
(use '--show-trace' to show detailed location information)

Test 2 (throws errors, unless all used nix files are in repo and git added)

flake.nix

{
  inputs.flake-utils.url = "github:numtide/flake-utils";

  # WILL THROW WITH ERROR
  #
  # ✘  ~/projects/hello   main ±✚  nix develop .
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # error: anonymous function at /nix/store/pphykv9fdgj8vb6zsd1vn1mam1am0y8y-source/asdfasdf.nix:1:1 called without required argument 'nixpkgs'
  #
  #       at /nix/store/pphykv9fdgj8vb6zsd1vn1mam1am0y8y-source/flake.nix:3:19:
  #
  #            2|   inputs.flake-utils.url = "github:numtide/flake-utils";
  #            3|   outputs = args: import ./asdfasdf.nix args;
  #             |                   ^
  #            4| }
  #
  # outputs = args: import ./asdfasdf.nix args;

  # WORKS (but all files should be `git add`ed)
  #
  # If they are not added, then
  #
  # ✘  ~/projects/hello   main ✚  nix develop .
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # warning: creating lock file '/home/srghma/projects/hello/flake.lock'
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # error: getting status of '/nix/store/20w3fp4x5v37z8waql91vvq77rmbpw5n-source/asdfasdf.nix': No such file or directory
  # (use '--show-trace' to show detailed location information)
  #
  # ✘  ~/projects/hello   main ✚  ga asdfasdf.nix
  #
  # ~/projects/hello   main ±✚  nix develop .
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # error: getting status of '/nix/store/wcdk1h7x83km048qmnxqys2ghaiisa3g-source/shell.nix': No such file or directory
  # (use '--show-trace' to show detailed location information)

  outputs = { self, nixpkgs, flake-utils }@args: import ./asdfasdf.nix args;

  # THROWS ERROR
  #
  # ✘  ~/projects/hello   main ✚  nix develop .
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # warning: creating lock file '/home/srghma/projects/hello/flake.lock'
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # error: access to absolute path '/nix/store/asdfasdf.nix' is forbidden in pure eval mode (use '--impure' to override)
  # (use '--show-trace' to show detailed location information)
  #
  # outputs = { self, nixpkgs, flake-utils }@args: import ../asdfasdf.nix args;

  # THROWS ERROR TOO
  #
  # ~/projects/hello   main ✚  nix develop .
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # error: access to absolute path '/home/srghma/projects/asdfasdf.nix' is forbidden in pure eval mode (use '--impure' to override)
  # (use '--show-trace' to show detailed location information)
  #
  # outputs = { self, nixpkgs, flake-utils }@args: import /home/srghma/projects/asdfasdf.nix args;
}

Summary

1.

IF output is trying to import ../some-file-from-outside-git-repo.nix
THEN nix build . will throw error access to absolute path is forbidden

This is a meaning of self-contained / hermetic evaluation _{(taken from frase Note that any file that is not tracked by Git is invisible during Nix evaluation, in order to ensure hermetic evaluation)}

2. (based on test 2) it seems like nix is using the git database as a cache to find anwser on question

Do I need to evaluate flake.nix at all? OR I will just use flake.lock?

(seems weird that nix cannot do this without git)

Am I right?

ElvishJerricco · May 31, 2023, 12:20am

This one specifically bothers me such an enormous amount. We can’t have arbitrary expressions in flakes because it could lead to complex computations? Can you imagine if we’d said that about nixpkgs? The whole reason Nix is better than JSON is because you can write arbitrary expressions.

srghma · May 31, 2023, 9:53am

created question nixos - nix flakes: what is the difference between `nix build git+file:.` and `nix build path:.` - Stack Overflow

NobbZ · June 12, 2023, 12:17pm

After having watched the question on SO for nearly 2 weeks now, I am kind of curious how you managed to not get downvoted much, as the question in prose is quite different from what to expect when reading the subject…

Anyway…

what is the difference?

The difference between path and git+file is quite obvious… path just treats the full filesystem location as the base of the flake, git+file tries to read the flakes content from a git repository. This is explained in the manual. nix flake - Nix Reference Manual which has already been linked.

what is self-contained / hermetic evaluation?

In quite simple terms, this means, that everything required is included, a declared input or by definition always available.

So a US military MRE is “self-contained” as you just need to add some tap-water and due to some chemical reaction in the outer bag, it will heat up the contents of the inner bag. Tap-water is considered to be always available.

A can of beans though is not self contained, as you require to have a can opener and a camp fire.

git-flake-evaluation is hermetic, but is path-flake-evaluation hermetic?

Yes, both are hermetic and pure. But a git-type is easier to send and reason about, and to check if you are really looking at the same code.

does nix build git+file:. have some additional caching, that nix build path:. doesnt have?

Yes. Evaluation cache can only work on “snapshottable” flake types.

why not make nix build . just equal to nix build path:., instead of this find type automatically behavior, described above (Because: what problem its trying to solve? The problem, solutions to which require all these weird behaviors, like unsolicited git add behind the scenes? But I can easily get around/silence it with path:.? This design is so weird, reason is so shady)

To avoid surprises when you share your code. In general defaulting to the version control system is sane, as one can assume that you want to use the version control to share your code.