Ah, woops, yeah I somehow missed the .envrc bit in your response, sorry. 
Yeah it would be interesting if you could somehow do it incrementally at least? I guess that’s the only thing at the moment that’s a bit of a pain.
Anyway, thanks a bunch for your help!
Is this just for flakes invoked using path:, or for ones using git as well?
It is true that for all kind of flakes the full content of the flake will be copied to the store. For SCM based flakes, this is just what has been “staged” and therefore is part of the worktree according to the SCMs knowledge.
For path flakes, there is no other instance that could be used to “stage” or “hide” certain files, therefore the full path has to be copied.
There is some WIP though, that allows for lazy copy of the flakes content, when working with local flakes. Not sure how well it works currently. I’m using nix from master, updated at least once a week, and so far I haven’t really recognized any enhencements in that regard, though my stuff is usually small, such that I do not have any problems anyway.
Oh right! Now it makes more sense why it needs things to be tracked in source control! I thought it was just about it being finicky with making sure stuff was checked in (which seemed a bit annoying in lieu of a workaround, to be honest).
Yeah this might make things hard for now, so I might have to abandon the flake approach for the time being. I’d be potentially concerned about the wear on my SSD with all the copying! Lots of the size of the repository is in the git history, a large volume of test data, and the rust target directory (used for intermediate build artifacts).
Perhaps shell.nix is a better way to go for me at this point. 
Does that run into the same issue with disk copying? I’m thinking I’ll just build stuff statefully in the nix-shell, as opposed to packaging stuff decoratively for now. At the very least it means that I can scope my dependencies to the repository, which is a nice win, even if I miss out on some of the other benefits of flakes.
It would definitely be cool to see the local workflow fleshed out more. I’m not sure if it’s possible, but it might also be useful to have support for ignoring certain non-essential files, like build artifacts, repository information, and test data, to keep disk traffic down, especially in large projects.
Oh, I’ll also add (for any Nix developers reading) that as somebody who has attempted to learn Nix in the past, despite some of the teething problems I’ve had with flakes they do seem like a nice improvement over the old approach! The addition of a lockfile, and having the configuration all in one place is very handy. Looking forward to future improvements, even if there is a way to go!
I am far from an expert, but one out-of-the-box technique that might be useful is to keep your flake in an independent repository, and use it to generate a dev shell. You could then enter the shell environment by invoking the flake via its github “URI” (right from the clone of the target project), or clone the repository locally and invoke it from there (then move over into your target project directory). I’m sure this approach drops some of the advantages of keeping your flake alongside the software you’re flakifying, but as most things in life, we’re forced to engage in a tradeoff.
The excessive copying at least should, eventually, be fixed by Copy local flakes to the store lazily · Issue #3121 · NixOS/nix · GitHub.
Oh thanks for the link to the relevant issue, that’s super useful!
In the interim I’ll try to figure out how to put the bits together to make a solution like @roni described.
Ok, so I’m trying to implement this with the following (somewhat awkward) project structure as a workaround:
my-project - clone of upstream git repository, for local development and editing
.envrc - loads the flake from ../my-project-flake:# reload when these files change
watch_file "$(pwd)/../my-project-flake/flake.nix"
watch_file "$(pwd)/../my-project-flake/flake.nix"
# load the flake devShell
eval "$(nix print-dev-env "$(pwd)/../my-project-flake")"
my-project-flake local git repository containing the flake
flake.lock
flake.nix
{
inputs = {
my-project = { url = "path:../my-project"; flake = false; };
};
outputs = { ... }
}
Based on the wording here regarding path: inputs, I’m assuming that nix will automatically detect that "path:../my-project" is a git repository and update files incrementally as I work on my project locally.
But yeah, now I run afoul of:
error: relative path '../my-project' points outside of its parent's store path '/nix/store/<hash>-source'
I think I might be running into this issue? Not sure what would be a good workaround for this would be.
Edit: I ended up just doing:
{
my-project = { url = "/absolute/path/to/my-project"; flake = false; };
};
output = { };
}
This isn’t ideal as it hard-codes my project to a specific location, but I guess seeing as this is a workaround on top of a workaround it should be ok for now.
You can get rid of the awkwardness of having the absolute path in your flake.nix by using —override-input my project $(pwd) from your .envrc. The url in the flake input would be the upstream vcs. The idea is that for your local development you are overriding the upstream vcs with your own working copy. NB I haven’t actually tried this on my own so it might turn out there are some gotchas with the interaction of flake = false and —override-input
What exactly would that .envrc look like? Is this a feature added by the direnv-nix integration thing or am I unaware of a nix environment variable for input overrides?
Asking because I have this exact workflow all over the place, when depending on my own flakes, and being able to skip the constant --override-input invocations would be really handy.
I think it cannot (easily) be done using nix-direnv since it doesn’t have a way to pass flags to the various nix commands used behind the scenes. However, @brendanzab has this in their .envrc:
So a change to
eval "$(nix print-dev-env "$(pwd)/../my-project-flake" —override-input my-project "$(pwd)")"
would apply the override each time the environment is computed.
Ahh yeah, it seems I run into:
error: path '/absolute/path/to/my-project' is not a flake (because it doesn't contain a 'flake.nix' file)
Was a cool idea though! Would be nice if you could have a way to override non-flakified inputs somehow.
How do you read the contents of my-project ? for example I want to use flakes for python projects and I am using mach-nix to read requirements.txt file.
pyEnv = mach.mkPython {
requirements = builtins.readFile ./requirements.txt;
};
Here I am getting error that
error: getting status of '/nix/store/zh6zsdpwrcysbiqjm6mripkjjf9wq6wc-source/my-project/requirements.txt': No such file or directory
git add requirements.txt, nix flakes require the file to be in version control if you’re using version control. It does this for reproducibility and so that it can exclude .gitignored files from the store import.
There’s some issues about getting better error messages for that.
the flake.nix is in my my-project-flake
my-project = {
url = "/Users/user_name/Developer/my-project";
flake = false;
2 };
and the requirements.txt in my-project is under version control.
Here the problem is I don’t know how to reference the requirements.txt file from my input to flake.
Ah, then I think you are looking for something like
pyEnv = mach.mkPython {
requirements = builtins.readFile "${my-project}/requirements.txt";
};
I think this is technically IFD, so naughty, but it should work.
Just to suggest another possible workaround. You can also place the flake.nix in another directory and refer to that directory in envrc: use flake ~/projects/myflakes#myproject. Allows me to reuse devshell modules for different projects more easily, but it doesn’t allow me to refer to local project files (afaik).