I’ve seen both cp and mv used in custom installPhase scripts for putting files under $out. Is there any reason to prefer one over the other?
As far as I can tell, the nixpkgs docs use both in examples.
3 Likes
It depends.
I want to say upfront that on most filesystems ( ext4 for example ) if you use cp -pr --reflink=auto -- . "$out"; you should just end up creating hardlinks; so really the performance hit you take is just syscalls rather than disk IO. But there are of course exceptions.
If you’re packaging something for your own usage that you don’t intend to PR into Nixpkgs you can absolutely use mv.
Just keep in mind that a handful of setup hooks related to tests, “source only” hooks, and some dist hooks ( scarcely used ) would likely fail since they often run after the install phase. For Nixpkgs we wouldn’t want to break these niche use cases so cp is preferred.
Personally a lot of the time I’ll just run builds in $out to avoid copying or moving anything; but if I were planning to PR Nixpkgs I would drop that optimization.
For Nixpkgs also keep in mind that in practice people consume built artifacts from caches, so the performance difference between a move vs copy can end up being a wash. What really matters is that a build is reproducible in this context.
2 Likes
If /tmp and /nix/store are on different file systems, including different ZFS datasets or btrfs subvolumes, then there is no difference between moving and copying, is there? I don’t even think reflink works across file systems.
1 Like
If everything is on the same filesystem then mv is faster because it does not need to duplicate the data.
There is nothing forbidding using mv.
I can’t think of any hooks that would do that when manually specifying the install phase. Most places where such things occur use hooks for everything and the package managers commands.
This is dirty because it leaves temporary files. Don’t do that.
2 Likes
I always assume that /nix/store is on a different file-system so I copy. And if it wasn’t I still prefer to copy because then the files in $out are allocated at the same time, rather than some allocated during the build and some not. Don’t take my word on that, that’s me being superstitious about fragmentation.
Also, for the sake of being functional its best to leave the artifacts of the buildPhase where they are and not mutate the build state after buildPhase. That allows someone to add a postInstall or fixupPhase hook that can work on the build artifacts seperately from what happened in installPhase.
2 Likes
Wouldn’t that leave changes in the build directory which really should be in $out?
It’s not something you would normally do, I just think it’s good practice in theory.
I think packagers from more “traditional” distros might suggest that both are wrong and tell you to use install.
But I have personally used all 3 just sort of at random so far 
4 Likes
All of my builders have nix-daemon’s TMPDIR pointing to a tmpfs filesystem. There is a very large performance benefit from this – with enough RAM incomplete build artifacts never hit the disk, and even if they do (go to swap) it’s still faster because swap doesn’t need to be journaled/sync’d/crashsafe.
Plus with install you don’t need mkdir -p:
installPhase = ''
install -D frobnicate -t $out/bin/
'';
3 Likes
No you won’t. That command will create reflinks on certain filesystems which support them. Btrfs, XFS and bcachefs are the only Linux ones I know of.
There’s a catch here that /nix/store and $TMPDIR are usually behind different mounts in the VFS which prevents copies but with recent kernels, that has been alleviated.
On regular old filesystems like EXT4, it’ll create regular old copies. No hardlinks.
Unlike ZFS, you can move files between btrfs subvolumes.
1 Like
Why does the installPhase put files under “$out” in the first place? Isn’t “$out” the path to the store, i.e. /nix/store/...-hello...? The install phase is after the build phase and often installs to the prefix, which is $out, right?
It seems like your last sentence is characterizing this correctly, so I’m not sure where the confusion is. Can you maybe expand on why it doesn’t seem right?
My confusion arises from the fact that you would cp or mv something to $out during the installPhase. Maybe, because the buildPhase produced output outside of $out so cp or mv is used to bring the output (i.e. binaries) back into the store?
I think you’re imagining that the build is taking place in the store, when it’s taking place in a temporary directory (which the source has normally been unpacked to, first).
1 Like
Yes, that was my assumption, I’m glad to get this explained, thank you. So if I have a custom build script, I do actually have to cp or mv my result to $out within the installPhase (now the term makes more sense, too). So the pwd of the builder is the temporary location and not $out. Is this properly documented? It seems to not be mentioned on the Nix Pills, but maybe I missed that essential key information?
It’s a bit over half way into the Derivations section of the Nix manual:
The builder is executed as follows:
- A temporary directory is created under the directory specified by
TMPDIR(default/tmp) where the build will take place. The current directory is changed to this directory.
2 Likes
I think the Derivation section should not be under “Nix Language” (same goes to the builtin constants / functions). A derivation is not a language feature, but rather part of the framework. I could possibly program the behavior of a derivation in another language.
I think I missed that crucial information, because I was expecting only language specific properties like syntax, data types, etc. under “Nix Language”. Maybe it was better to split this section into two?
Derivations and built-in functions are also on the same level, although a derivation is is a built-in function according to these docs.
Hm, depends on your viewpoint. The thing with the Nix Language is that it’s a 100% pure functional language with no side-effects except derviations. The derivation function is thus a built-in, otherwise Nix as a language would be practically useless.
You also have to consider that a derivation is only the *.drv file in the store. Building a derivation results in a realisation, and actually doing this build is implemented by the Nix Daemon, but deeply integrated into the Nix Language as well, as evaluating the outPath of a derivation must cause it to be built, else that path could not be known.
You are very correct that the daemon could be re-written in a different language, but that doesn’t change the fact that derivations are a feature of the Nix Language.
The framework, by which I assume you mean nixpkgs and its lib and stdenv, only builds on top of that functionality, and so should not document its behavior.
1 Like
I think what they meant was that the .drv files could be created without Nix. And they’d be right because that’s exactly what GUIX was(/is?) doing. It’s just data in a weird format.
1 Like
There are some differences between install and cp/mv. However I don’t know how to recursively use install yet, so I prefer to use cp -r.