when copying something to the nix store, make a hard link between the existing location and the store one.
Problems with this:
it would only work if the source and destination are on the same filesystem, such as where everything is in a single global /. This might be common enough to be worth trying, except that:
it would allow manipulation of the store contents through the hard link; the store is mounted read-only but the original file is outside that
@uep hard links and reflinks are different things. Reflinks are usually only possible on copy-on-write file systems, and can be made with cp --reflink=<always|never|auto>. Reflinks are indistinguishable from regularly copied files, but they can be created instantly and consume no space at first. They share all their blocks on disk with the original, but when either is modified, the new data is written to new locations on disk, and only the modified one is updated to point to that new data.
Yeah as @ElvishJerricco said, basically a reflink is equivalent to a copy, just faster. I have my nix store on a large magnetic disk array. I know I can probably spend some time tuning it for small copies. Or I could have nix do a reflink, which is a fast, basically instant copy.
As for the same filesystem… yes that is true. IIRC, the filesystem ioctl (FICLONE) that does this will fail if the filesystems are different, and then you fallback to normal copying.
cp is a shell utility and Nix is not a shell script. “Copy” is an abstract concept, it’s not something that you can do at a low level without breaking abstractions and making code more expensive to mantain.
Considering the portion of nix users who use it on linux, a fair number of whom have a CoW filesystem, that maintenance burden seems entirely justifiable. It’s really not difficult to try a reflink first before initiating a normal copy, but only on linux.
Unfortunately reflinks are not even support on ZFS openzfs/zfs#405. I have seriously considered switching to btrfs because of it, but haven’t so far. If Nix could take advantage of this that might push me over the edge.
I want to try btrfs , but i read some many bad things about it, maybe the bad things i read about it are not true, and as with all software, it gets better over time.
Maybe keeping things the can be easily rebuild on btrfs , and more inmportant stuff on zfs?
At least for btrfs, this has been fixed in recent kernels which allow reflinks inside the same FS even across VFS boundaries, so this is something we can actually start to actively use now with 6.1 becoming the new LTS kernel. Would be great to have Nix use reflinks wherever possible for 23.05.
Remember when the Linux API became stablized and we got all those games ported over and performance was better and we didn’t have to use Wine anymore? This is going to be like that.
Interesting idea, but how would this interact with the upcoming source tree abstraction? From my understanding that PR eliminates the need to even copy many inputs to the store since they are fetched lazily. Things like patches and zip extractions are done in-memory which also saves a ton of disk space. I suppose that shallow copies could still be used for situations where store paths are materialized but a more generalized solution might be desirable - as discussed this would only really apply to BTRFS.
Source tree abstraction only helps in certain cases. If you set src = self; in your derivation, the whole flake is getting copied anyway, and it would still be nice if it was done with a reflink if possible. Same issue if you pin a copy of self in your registry in a nixos config.
There is nothing you’d need to do for NAR serialization/deserialization. I’m not proposing Nix dedup anything on the fly. I’m merely asking that, if we’re copying something to the local store and it’s from a local file, then we try the various clone/reflink operations (via the system calls / ioctls), and fall back to regular behavior if they fail.
The source path abstraction stuff is interesting, but my use case here is for quick flake iteration, and the flake sources are copied to the nix store as far as I can tell. It’s a small pain (few hundred milliseconds).