Locally excluding nix flakes when using nix independenly of upstream

Help
1

Is it possible to use flakes without tracking them in Git? I want to use one for an existing work project, but I’m not yet ready (if ever) to upstream it to my colleagues. I’ve excluded the relevant Nix files (.envrc, flake.nix, flake.lock, shell.nix) from git, but when I try to build it, I get:

error: source tree referenced by 'git+file:///...' does not contain a '/flake.nix' file

Am I going about this wrong? Should I just be using shell.nix or something like that? Or should I install the tools I need globally? Or make a local repository that includes my work project as a git submodule?

Is there a guide I that covers development with nix (and possibly Nix with flakes) where upstream does not use Nix? I’ve done some searching but failed to find anything - perhaps I’m missing some search terms.

3 Likes
2

git add -N flake.nix might get Nix to cooperate with git. If it’s
gitignore’d, you might need to force-add it with -Nf. A bit awkward,
but it might work.

3

Thanks! Yeah, seems like only a partial solution, alas, as it shows up in my git status. Meaning if I want to do other things in Git I’ll need to constantly be adding/removing flake.nix. I’d be super interested if there is a way around this, or a different approach I could be taking for local development.

4

For my work projects where I can not risk accidental commits of nix related files, I use path:$(pwd) as flake ref.

That works pretty well.

You loose the nice stuff though that you’d normally get with SCM based flakes.

5

Oh this sounds promising! Could you elaborate a little on what this workflow looks like? Do you store your flake out of tree?

6

No, I have the flake.nix exactly where I’d put it usually.

And when I’d usually do nix build .#foo I now do nix build path:$(pwd)#foo.

Though most of the time it’s just use flake path:$(pwd) in a .envrc for use with direnv.

5 Likes
7

Cool, that seems to work, at least from the terminal!

It does take a good deal time to load the shell using nix develop – it says it is copying '<path-to-project' for a decent chunk of time – if my repo is gigabytes in size would that cause this to take a while? Or might it be something in how I set up my flake?

Also wondering if this can be hooked into my .envrc or shell.nix? I’ve been using direnv or Nix Environment Selector to help my editor (VS Code + rust-analyzer) to find the appropriate tools.

8

I managed to do this by adding this as my .envrc:

# reload when these files change
watch_file flake.nix
watch_file flake.lock
# load the flake devShell
eval "$(nix print-dev-env path:"$(pwd)")"

Still takes a good while to load. :thinking:

9

You can of course use .envrc, I do as well as I said. I am using nix-direnv addon for direnv through home manager. It does not only provide the use flake, but also creates GC roots for shells (flake shalles as well as regular ones).

Sadly yes, every invocation of a nix command will copy the flakes folder to the store. And if the folder is huge, then that will take a while.

10

Ah, woops, yeah I somehow missed the .envrc bit in your response, sorry. :sweat_smile:

Yeah it would be interesting if you could somehow do it incrementally at least? I guess that’s the only thing at the moment that’s a bit of a pain.

Anyway, thanks a bunch for your help!

11

Is this just for flakes invoked using path:, or for ones using git as well?

12

It is true that for all kind of flakes the full content of the flake will be copied to the store. For SCM based flakes, this is just what has been “staged” and therefore is part of the worktree according to the SCMs knowledge.

For path flakes, there is no other instance that could be used to “stage” or “hide” certain files, therefore the full path has to be copied.

There is some WIP though, that allows for lazy copy of the flakes content, when working with local flakes. Not sure how well it works currently. I’m using nix from master, updated at least once a week, and so far I haven’t really recognized any enhencements in that regard, though my stuff is usually small, such that I do not have any problems anyway.

13

Oh right! Now it makes more sense why it needs things to be tracked in source control! I thought it was just about it being finicky with making sure stuff was checked in (which seemed a bit annoying in lieu of a workaround, to be honest).

Yeah this might make things hard for now, so I might have to abandon the flake approach for the time being. I’d be potentially concerned about the wear on my SSD with all the copying! Lots of the size of the repository is in the git history, a large volume of test data, and the rust target directory (used for intermediate build artifacts).

Perhaps shell.nix is a better way to go for me at this point. :thinking: Does that run into the same issue with disk copying? I’m thinking I’ll just build stuff statefully in the nix-shell, as opposed to packaging stuff decoratively for now. At the very least it means that I can scope my dependencies to the repository, which is a nice win, even if I miss out on some of the other benefits of flakes.

14

It would definitely be cool to see the local workflow fleshed out more. I’m not sure if it’s possible, but it might also be useful to have support for ignoring certain non-essential files, like build artifacts, repository information, and test data, to keep disk traffic down, especially in large projects.

15

Oh, I’ll also add (for any Nix developers reading) that as somebody who has attempted to learn Nix in the past, despite some of the teething problems I’ve had with flakes they do seem like a nice improvement over the old approach! The addition of a lockfile, and having the configuration all in one place is very handy. Looking forward to future improvements, even if there is a way to go!

1 Like
16

I am far from an expert, but one out-of-the-box technique that might be useful is to keep your flake in an independent repository, and use it to generate a dev shell. You could then enter the shell environment by invoking the flake via its github “URI” (right from the clone of the target project), or clone the repository locally and invoke it from there (then move over into your target project directory). I’m sure this approach drops some of the advantages of keeping your flake alongside the software you’re flakifying, but as most things in life, we’re forced to engage in a tradeoff.

17

The excessive copying at least should, eventually, be fixed by Copy local flakes to the store lazily · Issue #3121 · NixOS/nix · GitHub.

1 Like
18

Oh thanks for the link to the relevant issue, that’s super useful!

In the interim I’ll try to figure out how to put the bits together to make a solution like @roni described.

1 Like
19

Ok, so I’m trying to implement this with the following (somewhat awkward) project structure as a workaround:

  • my-project - clone of upstream git repository, for local development and editing
    • .envrc - loads the flake from ../my-project-flake:
      # reload when these files change
watch_file "$(pwd)/../my-project-flake/flake.nix"
watch_file "$(pwd)/../my-project-flake/flake.nix"
# load the flake devShell
eval "$(nix print-dev-env "$(pwd)/../my-project-flake")"
  • my-project-flake local git repository containing the flake

    • flake.lock

    • flake.nix

      {
  inputs = {
    my-project = { url = "path:../my-project"; flake = false; };
  };
  outputs = { ... }
}

      Based on the wording here regarding path: inputs, I’m assuming that nix will automatically detect that "path:../my-project" is a git repository and update files incrementally as I work on my project locally.

But yeah, now I run afoul of:

error: relative path '../my-project' points outside of its parent's store path '/nix/store/<hash>-source'

I think I might be running into this issue? Not sure what would be a good workaround for this would be.

Edit: I ended up just doing:

{
    my-project = { url = "/absolute/path/to/my-project"; flake = false; };
  };
  
  output = { };
}

This isn’t ideal as it hard-codes my project to a specific location, but I guess seeing as this is a workaround on top of a workaround it should be ok for now.

1 Like
20

You can get rid of the awkwardness of having the absolute path in your flake.nix by using —override-input my project $(pwd) from your .envrc. The url in the flake input would be the upstream vcs. The idea is that for your local development you are overriding the upstream vcs with your own working copy. NB I haven’t actually tried this on my own so it might turn out there are some gotchas with the interaction of flake = false and —override-input