Flake-compat as alternative to flakes or npins

Development
1

I’ve been using flakes since I started using NixOS about 4 years ago so it’s been a lot about how I’ve interfaced with Nix, but when I was implementing a POC of generating derivations for all terraform providers(and versions) by mangling the data from the OpenTofu registry i hit a roadbump since that repo is 276MB without history so every time i changed one byte in a Nix file and reevaluate it’ll copy 276MB to store and I thought that was really stupid so I worked around it by writing a default.nix with npins instead that could parse nix and json from the filesystem instead. While 276MB doesn’t sound like much, imagine that every time you press save, and I was doing this from a Chromebook which has eMMC SSD so it is both small, slow and has useless write endurance so I thought I’ve got to start unflaking my system.

But since I’ve been using flakes I just couldn’t bother unflaking my system right now and it’s also quite practical to have a flake to give others so i put it off for awhile. But today I looked into flake-compat as a way of getting around flakes and it was really close to doing what I wanted straight away, I just had to apply a tiny patch to it like this and i was evaluating my config from the filesystem!

diff --git a/flake-compat2.nix b/flake-compat2.nix
index df5128d9..e33aa94c 100644
--- a/flake-compat2.nix
+++ b/flake-compat2.nix
@@ -5,7 +5,7 @@
 # containing 'defaultNix' (to be used in 'default.nix'), 'shellNix'
 # (to be used in 'shell.nix').
 
-{ src, system ? builtins.currentSystem or "unknown-system" }:
+{ src, impureEval ? false, system ? builtins.currentSystem or "unknown-system" }:
 
 let
 
@@ -126,8 +126,20 @@ let
     isShallow = builtins.pathExists (src + "/.git/shallow");
 
   in
-    { lastModified = 0; lastModifiedDate = formatSecondsSinceEpoch 0; }
-      // (if src ? outPath then src else tryFetchGit src);
+    (
+      if !impureEval then
+        {
+          lastModified = 0;
+          lastModifiedDate = formatSecondsSinceEpoch 0;
+        }
+        // (if src ? outPath then src else tryFetchGit src)
+      else
+        {
+          lastModified = 0;
+          lastModifiedDate = formatSecondsSinceEpoch 0;
+          outPath = src;
+        }
+    );

This way I can use the flakes CLI to output and created a default.nix like this:

let
  # Our own patched flake-compat that evaluates impurely from disk
  flake-compat = import ./flake-compat.nix;
  flake = flake-compat { src = (builtins.toString ./.); };
in
  flake.outputs

Now sure this still evaluates all flake inputs from store and nixpkgs, but since we easily can craft our own nixpkgs instance within a flake and I could point that import at any path i want and it should be able to eval nixpkgs from filesystem too.

I just figured this out today but I’m quite excited because it means we can be compatible with flakes without evaluating from store. I can publish as a flake and with this default.nix so flake users can consume my packages (if i had any of value), my nixosConfig and whatever while being able to iterate on my machine without flakes and the iteration speed of evaluation from filesystem.

If you wanna reap the benefits you must add some arguments to your nixos-rebuild so it looks like this:

nixos-rebuild build --file $flakepath --attr "nixosConfigurations.\"$hostname\"" # nixos-rebuild applies the last bit of the attrpath

I don’t really have any outstanding issue with flakes that isn’t related to evaluating from store and this pretty much solves it.

I imagine that flake.lock format is pretty stable and unlikely to change an awful lot so it’s unlikely flake-compat would break one day either, the code to extract is is complicated but it’s not impossible considering I solved the source eval from store patch in no time. I must say I was really impressed to se nvd diff not have any changes when using flake-compat vs building with flakes.

I would love some thoughts and feedback it feels like I must’ve missed something, otherwise I think iterating on flake-compat could be a pretty valid community effort (so we can implement our “follows” in Nix instead of “flake inputs nix”). I don’t really mind the nix flake cli experience to be honest I think it’s good enough and if i must use it to update my lockfile (until someone implements a compatible tool) I’m fine with that.

Appreciate if we stay away from the flakes vs non-flakes discussion here since this pretty much seems to splat two flies with one smack as we say in Swedish :smile:

Cheers

EDIT: lix-project/flake-compat: Turns Nix flakes into normal Nix expressions - Lix Systems Lix has a flake-compat implementation with argument copySourceTreeToStore already :smile:

3 Likes
2

I mean, surely this means the community should iterate on the flakes concept so we don’t need to jump through ridiculous hoops just to avoid copying around gigabytes’ worth of data for no real reason.

Your workaround is cool, but this just points to the implementation of flakes still being unusable in practice. Lazy trees remains essential, and I’m honestly not convinced that feature by itself solves the usability problem.

I wonder if a nix fork that simply foregoes the copy to the store and just accepts the potential for impurity would be feasible. I’d certainly use that over the current implementation until a more workable solution is found…

8 Likes
3

I still dont understand why we even need lazy trees. It seems like a solution to a self-inflicted problem. Flakes should’ve never copied nix code to the store in the first place. It makes zero sense to me.

6 Likes
4
5

I agree and I would love to use such a thing, however I don’t think flake-compat is ridiculous, it’s just some Nix expressions to traverse the flake.lock file. If some community fork enabled this usecase without flake-compat I’d be happy to use it if maintenance looks bareable but this is old and tried code so it’s not too bad imho.

I really like that I didn’t have to do anything but add a patched flake-compat.nix and a default.nix rather than sort out all my legacy cruft that’s accumulated over the years.

It’s definitely not the “end stop” solution but I thought it to be quite elegant, I will iterate further and see if I can implement some smart way to implement overrideAttrs for default.nix and verify that “path” types aren’t fetchGitted so we can eval nixpkgs from FS easily too.

I encourage the 0 people who use both flakes and feel like hostages by using flakes to try this out considering how non-invasive it is.

I’ll post an example command to build your nixosConfiguration with flake-compat and compare the result with your current system.

Final note: I agree we shouldn’t have to do this but I’m happy to use a workaround while waiting for the “real” solution :slight_smile:

6

That looks very similar to what I did, I’m happy to see someone thought about this before me :smile:

7

I think it would make sense if a source is of the path type to not copy to store, but if it’s a git or any other type it’s copied to store, would make flakes a lot more user-friendly

8

Sorry, wasn’t my intent to say that. I like your hack, and I may well give it a shot myself (though I think my setup is a little too integrated to get much of a benefit, plenty of tools will still copy repos).

I just don’t think we should dump a bunch of effort into a workaround when an actual solution is possible.

1 Like
9

Haha, I didn’t mean to sound like I I interpreted as “i/this thing is stupid”. I agree that it’s pretty stupid to have to bend over backwards like this.

However I do disagree with your sentiment to “do the right thing and fix the source of the problem”. If Lix people haven’t done so already there must be a reason(?). What I’m trying to say is that the current situation is what it is and since this solves my issue and considering the git history of flake-compat and how little flakes change 2025 this fits my “goodenoughometer” :slight_smile:

Thanks for the clarification however!

10

I do think flake-compat is absurd inasmuch as somehow we can’t write performant c++ code so we’re using nix? The language which is known to be slow and memory hungry, and has the slowest-possible JSON parsing impl by any benchmark, to parse large amounts of JSON? And we can’t necessarily use other languages for fear of IFD, which is also a perf bottleneck nix created for itself with single-threaded eval. Though builtin fetchers including those for flakes are also single-threaded, so maybe it’s moot to even worry about IFD being slow.

(My view also isn’t a knock on your resourcefulness, but the existence and design of flake-compat in the first place.)

Also @azuwis it’s cool that lix folks tried to address the perf concerns, but their readme is incorrect :slight_smile: See README: fix node name look-up by Gerg-L · Pull Request #61 · edolstra/flake-compat · GitHub for the issue and fix.

1 Like
11

I would like to question your interpretation of “large amounts of JSON”, it might be slow as a turtle but my flake.lock is pretty big and it’s 29KB. :person_shrugging:

I have full understanding that the current Nix implementation is mostly single-threaded and sequential, parallelism is really hard. I hope svix graduates and changes that some day :smile:

I found that lix maintains their own flake-compat which I’ve opened not one but two pullrequests to while chasing this goose :smile:

12

s/svix/snix/

agreed, this looks promising, but a concern i have is that it is just another “fork” (okay, re-implementation) when really, the right solution is to adapt some of the concepts that flakes rushed in to nix directly.

  1. lockfiles
  2. a better command line experience

there’s no reason we can’t do this, and IMHO, it’s where the community should put its focus instead of 600 new forks of nix or flake-related tooling.

that said, i really like what snix is doing and prefer rust over cpp, so, i also wouldn’t mind if we shifted dev effort to snix and replaced the cpp impl with it :slight_smile: