Can I use flakes within a git repo without committing flake.nix?

Help
32

At least the flakes are tied to the evaluation cache. That should have been two separate features. Flakes could already be stable for several NixOS releases, but instead “the evaluation cache feature” is still not ready and so is flakes.

33

I agree that it’s a problem that Flakes is tied to the evaluation cache. There’s a proposal here to untangle this, and I have another proposal myself in the works too.

However I don’t think Flakes is ready to be stabilized even considering this. There’s a number of significant issues with Flakes yet unresolved:

I think these are the major ones at least

10 Likes
34

As much as I agree that flakes are not ready yet, and in the danger to fully derail this thread, I might have an unpopular and extreme opinion:

Either do a breaking change that ruptures contintents within the remainer of the year, or just realease them as they are. The “expermintal” status became a farce and everyone uses them as a stable tool. We are at a point where the “experimental” thing became basically just that thing that you enable anyway right after install to be able to use nix proper, or alternatively you use the determinate installer which by default enables flakes and nix-commands…

TL’DR: Break them now or just release flakes…

9 Likes
35

Also some serious lock file issues:

  • Updating of individual inputs is broken
  • The lock file will grow indefinitely through mutual dependency updates
  • Inputs aren’t lazy when locking, penalizing development-only inputs, test-only inputs and other use case specific inputs. Not scalable.

I think all three should be solved by first un-flattening the lock file (ie getting transitive deps from dependency locks by default) and resolving follows at evaluation time.

(EDIT: my own thoughts, not decided by the Nix team)

1 Like
36

Changing the locking may well be such a change. You may need the newest Nix to read new lock files, unless we keep copying the transitive stuff for a migration window.

(EDIT: hypothesizing and, again, speaking for myself)

37

More tests

Test 1 (throws `expected a string but got a thunk`) 
$ git status
On branch main

No commits yet

Changes to be committed:
 (use "git rm --cached <file>..." to unstage)
       new file:   asdfasdf.nix
       new file:   flake.nix
       new file:   shell.nix

$ cat flake.nix
{
description = "my project description";

inputs.flake-utils.url = import ./asdfasdf.nix;

outputs =
 { self, nixpkgs, flake-utils }:
 flake-utils.lib.eachDefaultSystem
   (system:
     let pkgs = nixpkgs.legacyPackages.${system}; in
     {
       devShells.default = import ./shell.nix { inherit pkgs; };
     }
   );

}
$ cat asdfasdf.nix
"github:numtide/flake-utils"
$ nix develop .
warning: Git tree '/home/srghma/projects/hello' is dirty
error: expected a string but got a thunk at /nix/store/ny5h4254c0x95xv6nniqq89q83vpw0a7-source/flake.nix:24:1
(use '--show-trace' to show detailed location information)
Test 2 (throws errors, unless all used nix files are in repo and git added)

flake.nix

{
  inputs.flake-utils.url = "github:numtide/flake-utils";

  # WILL THROW WITH ERROR
  #
  # ✘  ~/projects/hello   main ±✚  nix develop .
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # error: anonymous function at /nix/store/pphykv9fdgj8vb6zsd1vn1mam1am0y8y-source/asdfasdf.nix:1:1 called without required argument 'nixpkgs'
  #
  #       at /nix/store/pphykv9fdgj8vb6zsd1vn1mam1am0y8y-source/flake.nix:3:19:
  #
  #            2|   inputs.flake-utils.url = "github:numtide/flake-utils";
  #            3|   outputs = args: import ./asdfasdf.nix args;
  #             |                   ^
  #            4| }
  #
  # outputs = args: import ./asdfasdf.nix args;

  # WORKS (but all files should be `git add`ed)
  #
  # If they are not added, then
  #
  # ✘  ~/projects/hello   main ✚  nix develop .
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # warning: creating lock file '/home/srghma/projects/hello/flake.lock'
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # error: getting status of '/nix/store/20w3fp4x5v37z8waql91vvq77rmbpw5n-source/asdfasdf.nix': No such file or directory
  # (use '--show-trace' to show detailed location information)
  #
  # ✘  ~/projects/hello   main ✚  ga asdfasdf.nix
  #
  # ~/projects/hello   main ±✚  nix develop .
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # error: getting status of '/nix/store/wcdk1h7x83km048qmnxqys2ghaiisa3g-source/shell.nix': No such file or directory
  # (use '--show-trace' to show detailed location information)

  outputs = { self, nixpkgs, flake-utils }@args: import ./asdfasdf.nix args;

  # THROWS ERROR
  #
  # ✘  ~/projects/hello   main ✚  nix develop .
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # warning: creating lock file '/home/srghma/projects/hello/flake.lock'
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # error: access to absolute path '/nix/store/asdfasdf.nix' is forbidden in pure eval mode (use '--impure' to override)
  # (use '--show-trace' to show detailed location information)
  #
  # outputs = { self, nixpkgs, flake-utils }@args: import ../asdfasdf.nix args;

  # THROWS ERROR TOO
  #
  # ~/projects/hello   main ✚  nix develop .
  # warning: Git tree '/home/srghma/projects/hello' is dirty
  # error: access to absolute path '/home/srghma/projects/asdfasdf.nix' is forbidden in pure eval mode (use '--impure' to override)
  # (use '--show-trace' to show detailed location information)
  #
  # outputs = { self, nixpkgs, flake-utils }@args: import /home/srghma/projects/asdfasdf.nix args;
}

Summary

1.

IF output is trying to import ../some-file-from-outside-git-repo.nix
THEN nix build . will throw error access to absolute path is forbidden

This is a meaning of self-contained / hermetic evaluation (taken from frase Note that any file that is not tracked by Git is invisible during Nix evaluation, in order to ensure hermetic evaluation)

2. (based on test 2) it seems like nix is using the git database as a cache to find anwser on question

Do I need to evaluate flake.nix at all? OR I will just use flake.lock?

(seems weird that nix cannot do this without git)

Am I right?

38

This one specifically bothers me such an enormous amount. We can’t have arbitrary expressions in flakes because it could lead to complex computations? Can you imagine if we’d said that about nixpkgs? The whole reason Nix is better than JSON is because you can write arbitrary expressions.

7 Likes
39

created question nixos - nix flakes: what is the difference between `nix build git+file:.` and `nix build path:.` - Stack Overflow

1 Like
40

After having watched the question on SO for nearly 2 weeks now, I am kind of curious how you managed to not get downvoted much, as the question in prose is quite different from what to expect when reading the subject…

Anyway…

  1. what is the difference?

The difference between path and git+file is quite obvious… path just treats the full filesystem location as the base of the flake, git+file tries to read the flakes content from a git repository. This is explained in the manual. nix flake - Nix Reference Manual which has already been linked.

  1. what is self-contained / hermetic evaluation?

In quite simple terms, this means, that everything required is included, a declared input or by definition always available.

So a US military MRE is “self-contained” as you just need to add some tap-water and due to some chemical reaction in the outer bag, it will heat up the contents of the inner bag. Tap-water is considered to be always available.

A can of beans though is not self contained, as you require to have a can opener and a camp fire.

  1. git-flake-evaluation is hermetic, but is path-flake-evaluation hermetic?

Yes, both are hermetic and pure. But a git-type is easier to send and reason about, and to check if you are really looking at the same code.

  1. does nix build git+file:. have some additional caching, that nix build path:. doesnt have?

Yes. Evaluation cache can only work on “snapshottable” flake types.

  1. why not make nix build . just equal to nix build path:., instead of this find type automatically behavior, described above (Because: what problem its trying to solve? The problem, solutions to which require all these weird behaviors, like unsolicited git add behind the scenes? But I can easily get around/silence it with path:.? This design is so weird, reason is so shady)

To avoid surprises when you share your code. In general defaulting to the version control system is sane, as one can assume that you want to use the version control to share your code.

2 Likes
41

To put that even more succinctly:

  • path:<path> ~= cp -r <path> /nix/store
  • git+file:<path> ~= git clone file:///<path> /nix/store

After that, the flake is evaluated in its new path in /nix/store. This is how it actually works today, in the future there will be some fancy mechanism to avoid doing this copy while pretending it has happened.

cp -r obviously copies everything, while git clone only copies files that have been git add-ed.

The path variant is generally git-unaware, so you can’t ask nix to use a specific commit or branch instead of what the repository is currently at, and you can’t ask it to exclude the .git directory.

You typically want to exclude the .git directory for caching (because otherwise any change will bust the cache), so nix defaults to the git protocol if at all possible. Same with other VC systems. The heuristic is deterministic, and very stable, so there’s not really a reason not to do this.

42

@NobbZ I didn’t uderstand anything new, the philosophical meaning of self-contained was evident without comparing to MRE, I would better see examples and links to the code

E.g. "we need to copy whole dir, bc IF we would not copy THEN … "

But thank you for trying, no insulting, I understand that you cannot see what is in my brain

Thx @TLATER Now I understand more

But, probably, you should change

git+file:<path> ~= git clone file:///<path> /nix/store

To

git+file:<path> ~= git clone file:///<path> /nix/git (because it’s probably behavior similar to builtins.fetchGit)

Question:

Why the full directory is copied to nix store ? cp -r <path> /nix/store

Why not? … not copy anything at all? I mean, nix build path:. will just read flake.nix and flake.lock and imported nix files (if they exist) to build required derivations.

43

Wer would not be able to ensure it exists at all, nor that path literals would point at the same location.

nix build (or any flake related thing in general) does not imply that the flake is actually local by any means.

Also, if we wouldn’t copy the flakes content to the store, then ./. would be some /home/srghma/… for you, /home/nmelzer/… for me, and /Users/foo/… for someone running on the mac.

Even worse, how would you resolve a ./. in a “remote” flake without copying first? Fully rely on a temporary folder created by mktemp -d with a random suffix?

All of these do not sound quite reproducible to me, as the state of the system influences what you get out of the build.

TLater was talking about “conceptually” which skips away a lot of the details… What actually happens for git+file is much more involved than a simple git clone (would take far too long for huge repos), instead in reality a git ls-files or so is ran in the repo and all files listed get copied over. But as said, TLater did simplify for getting concepts over, rather than implementations. They would be in the way here.

Because the lazytree hasn’t been merged yet, as it causes much more troubles than you might think it would.

See above.

2 Likes
44

@NobbZ

I think i now understood (more) the answer on "Why to copy whole repo? What else it needs except flake.nix and flake.lock? " (after reading edolstra post about flakes on tweag once again)

flake is simply a source tree (such as a Git repository) containing a file named flake.nix

Because flake is a repo by desing

45

Whatever is refered to by the flake. Which potentially is everything in its location. Even with lazytree, you will not be able to know in advance what exactly a derivation needs to be able to build. So when you hit a src = "${self}" or src = ./. you have to copy everything to satisfy the build.

1 Like
46

To clarify- under the new abstraction, this can be resolved by triggering after such dynamic paths are encountered, and won’t necessitate performing a full copy regardless of such encounters, right? I just want to make sure we aren’t losing the primary aspect of lazy trees I was looking forward to.

Otherwise, it could be worth looking into a “temporary” nix-store entry that read-only-links the original source until it is sure it was actually utilized.

47

Note that https://devenv.sh lifts the restriction for files needing to be tracked by git.

The downside is that if there are many/big files in the repo, loading the flake takes a lot longer.

One use case is reading .env files from Nix, but that file shouldn’t be committed to git, since it contains secrets.

1 Like
48

One thing I’m not entirely clear on after reading this thread:

Is there any downside in using path:. when your flake only contains devShells?

I have several projects at work that has nothing to do with Nix, but I like to use flakes to install development tools, and I don’t want to add that to Git.

1 Like
49

You give up on the evaluation cache. For just a handfull of devshells, that’s likely negligible.

50

So for the problem where you want to keep a flake.nix to provide a dev environment for a repo. Why not put that repo in a subdirectory and keep your flake.nix/flake.lock/.envrc in the directory above?

Like this:

embassy-rs-dev/  
   flake.nix      <-- my flake just for `nix develop`
   flake.lock
   .envrc
   embassy/    <-- repo I'm working on

Could also make embassy-rs-dev into a repo that has embassy as a submodule. Then the versions of the flake.nix and the embassy repo are linked.

Edit: It appears that it matters a lot whether embassy-rs-dev is a repo or not. If it is not a repo, I get

copying ~/bburdette/op-code/embassy-rs-dev

Every time I nix develop. If I specify nix develop path:., it still does the copy.

But if embassy-rs-dev is a repo, then nix-develop is instant.

2 Likes
51

Thanks! Creating a parent git repo does the trick indeed!

1 Like