I’m running into Option in nix-build, to force rebuild, even if package exists in store · Issue #493 · NixOS/nix · GitHub where changes to the input files of my derivation aren’t always replacing the content in nix-store and the hash is staying the same. My workaround has been to simply add a new attritube to the hashset e.g
temp = "one"; and then just keep adding a character at each
nix-build to force a new hash.
Regardless, I’d like to understand how the hash of the derivation is calculated.
Important: the hash of the out path is based solely on the input derivations in the current version of Nix, not on the contents of the build product. It’s possible however to have content-addressable derivations for e.g. tarballs as we’ll see later on.
But what does “input derivations” mean?
The documentation states:
Each output path is a concatenation of the cryptographic hash of all build inputs, the
name attribute and the output name.
But what is the “build input”?
Anybody have a link to documentation or code that clear this up?
Nix never “replaces the content in the nix-store”. It only ever puts new stuff in there. So changing your input files will always cause the things depending on them to rebuild. The inputs to a derivation are literally anything that it refers to. Derivations in Nix are quite simply just a set of environment variables, a builder script to run with them, and a list of other derivations it depends on. If any of these things is different, you have a whole new derivation
The fact that you’re observing your derivation not changing when you change your input files indicates either 1) your derivations aren’t dependent on the files you think they are, 2) a bug in nix, or 3) you aren’t changing the hash on a FOD.
#3 is the one gotcha. Fixed-output-derivations (FODs) are derivations like
fetchFromGitHub that have a
hash argument. Nix uses that hash to determine the output path instead of the derivation’s inputs. So if two derivations are completely different, but have the same fixed output hash, you only ever need to build one of them, and the result will count for the other. So if you e.g. update the
rev argument to
fetchFromGitHub but not the
hash argument, Nix will think you’re still trying to refer to the old version of the source, even though the
rev changed, because the hash you have for
fetchFromGitHub is incorrect (if you deleted the existing one and tried to rebuild, it would fail and tell you that the resulting output doesn’t match the expected hash)
The nix language evaluation creates
.drv files in the nix store, which are named by a hash of their contents, and contain every aspect of how to create a derivation. What to run, with what environment, with what nix store paths available in the sandbox, etc. You can see a pretty-printed form of the data in a
nix show-derivation. For normal derivations (not fixed-output), iirc (I might be missing some details here, but it’s the rough idea) the final output path’s hash is calculated with the drv file’s hash followed by
!out, hashed again (or more generally
! followed by the output name in question, since derivations can have multiple outputs).
For fixed-output derivations, the hash is the specified content hash that the result is supposed to have, truncated and encoded as nix store paths normally are. Since the derivation is always considered to have failed if the hash doesn’t match, it’s then the hash of the actual content.
As was mentioned above, this means nix will happily reuse an existing store object for a FOD so long as the hash is the same, even if other things change.
Today we just merged the first section of the upcoming architecture documentation in the Nix manual. We call it “build inputs” there, because “derivation” is a very specific concept that needs further clarification. You can check the recent pull requests on closely related concepts, which I’m currently trying to explain more consistently or introduce gently if they are still undocumented.
Feedback is highly appreciated, either here or as PR comments, as well as issues or pull requests.
Thanks for updating the docs.
I’ve got a related question: does the same apply for a nix-shell? I’ve noticed that derivation paths for nix-shell (even pure ones) often vary. Any insights on this?
This can happen easily if the src in your shell.nix is, say, the full unfiltered source of the local directory. Diffing two store paths may help identify what’s shifting.
(you might, for example, have a script or dev tool logging somewhere inside the project, writing build artifacts, regenerating documentation copies with the updated date in them…)
Thanks for the link @fricklerhandwerk . I’m having trouble understanding what a “build input” is though and how a hash is made from that.
When building a derivation, is this the process?
- copy everything that’s a path to a temporary directory (or mount the path in an overlay?)
- hash the copied/overlayed contents in nix/primops.cc at 15341334b50b2b432830a8e78abc331d0a47de13 · NixOS/nix · GitHub
- create a location in the store:
- copy the path content to the location in the store
.drv contents where each referenced path is replaced with a string path to the location in the store
- evaluate what is described in Architecture - Nix Reference Manual in
If so, then built input is then a nix expression or a path in the nix store, correct?