There are two common problems I run into when I’m either fetching some source or importing something that fetches some source. The examples are for overlays which I highly recommend using.
First, I don’t know what the attribute I want is called or where they have put it.
This can be figured out by loading up the source in nix repl and using the tab completion to explore the content of the derivation. I do this when the source is either too complex to work out manually or they set the attribute name to “${something.attribute} = thing.I.want” and working out something.attribute requires a bit of digging. Here is an example of how to do this with an overlay:
Save this as a file (tmp.nix), where overlay.nix is the filename of the overlay you want to explore:
Second, I know what attribute is called, but I might get “error: undefined variable”.
Sometimes you just need to change the sha256 of the source by one letter to get Nix to pull it down since Nix doesn’t realize the source has changed. I’ve run into situations where I’ve tried to build a derivation, it fails because I was using the wrong commit, but upon changing the rev the derivation still fails with the same issue since it is still using the old downloaded commit. The fix is to change the sha256 so Nix downloads the new commit and then when Nix fails the download it will tell you the correct sha256 for that commit.
Note: Nix may download the source even if it fails on something unrelated, so it may not be obvious that you are using something other than what you think you downloaded. Changing the sha256 can be a good sanity check.
This is by design. It’s called a fixed-output derivation and it’s identified by its hash and not how it’s build. If you pass the same sha256 to two derivations Nix will think they are just two ways to build the same thing.
It’s like a ‘break’ in the build-time dependency tracking and hashing in that no matter how it’s built, as long as the specified hash is the same, they are considered identical. I think if you understand the existence of such breaks (which is necessary to establish trust in these network-going derivations), you can understand this: Take Nix’s perspective: Given two derivations X and Y with the same sha256, if I’ve already built X, then building Y is either going to
Build something with sha256 identical to the output of X, so why bother, or
Fail to complete or build something with the wrong sha256, which means a build failure in general.
I agree that this is not the most intuitive behavior and can be difficult to communicate. See for example this StackOverflow question where the OP confused Git hash and output hash and considered it unacceptable to change the hash because a specific version is wanted.
So basically always take into account the sha256. Or rather, just think in terms of how Nix works.
I personally either just use nix-prefetch-git (or other related stuff) to get output hash for me and just treat it like any other argument when I fill it into a fetch*, or just simply change the sha256 not just as a sanity check but a deliberate action to make Nix tell me the hash.
The way I look at it (perhaps incorrectly) is that in principle Nix is a pure language (does not have side-effects). This means that an expression should always evaluate to the same value. Network I/O introduces impurity in Nix. The most straightforward example being that you use a fetcher to fetch a Nix expression to evaluate it. Since it is not guaranteed that the server always returns the same expression, evaluation would be impure. Fixed-output derivations bring back purity in such cases. Again taking the example of fetching a Nix expression – by specifying the hash, you can ensure that the same expression is always returned and thus that the evaluation is pure.
That said, there are impurities in Nix, some of the fetchers will work without a hash, you can reference local files, etc.
This is by design. It’s called a fixed-output derivation and it’s identified by its hash and not how it’s build. If you pass the same sha256 to two derivations Nix will think they are just two ways to build the same thing.
Actually, I think including the URL in the hash would have the same effect, since it would cause new builders to download the new files when the URL changes since the expression hash no longer is in the cache.
I think doing the above would help prevent a lot of confusion. I’m considering simply opening a PR to do this. Any objections?
None whatsoever. Looking at what is discussed in this thread the solution is to either include the URL in the hash (clean and simple) or create a new wrapper for fectch* commands as described by jonringer. Both would alleviate my second step in fetch* troubleshooting and remove this common cause of confusion.
None whatsoever. Looking at what is discussed in this thread the solution is to either include the URL in the hash (clean and simple)
It is not that clean, as it breaks all compatibility with mirrors… Being able to fetch the same content from an alternative source is sometimes needed.
My personal feeling is that these tarballs and other files are identified by their hash and the URLs are merely suggestions about where to get them. I don’t particularly like that this is given up, but if it’s causing so much trouble it might be worth the switch.
I’m thinking of other ways in which handling of such fixed-output derivations can be improved. For a start, I wonder if OfBorg can detect that a new derivation’s output already exists, but it was built by a different derivation, and when it happens build the new derivation anyway as a check. This might mean that we need to track in a database somewhere all these fixed-output derivations, but I feel like finding these could be just a database query for the closure of the new derivations and wouldn’t cost too much. It could also be a basis for a periodic check for dead links in Nixpkgs, which we apparently lack to some extent.
A similar thing might be possible for Nix itself, but I can’t think of a nice way to catch exactly the mentioned ‘forgot to change hash’ problem without causing false positives for legitimate use. Some kind of ‘known alternate deriver’ mechanism might be needed, but it would also need to be communicated through the binary cache, which is largely immutable… so that’s a problem.
I agree with @7c6f434c, breaking mirroring is a bad idea and @dramforever is right that the URLs are merely a suggestion about where to get a source. I jumped the gun advocating for a PR to include the URL in the sha256 for everything.
I’ve been trying to think of a way to handle ‘forgot to change hash’ or ‘copied hash from other derivation’ issues that are especially confusing for those not familiar with how Nix works. However, I’m thinking the right solution is to use an external program to handle sources, like Niv, or simply use nix-prefetch-git and never write a fetch* command that does not import a source description.
Note: I noticed I cannot edit a post after a day has passed. Is there a way to unlock a post for editing so I can change the HowTo to reflect what has been discussed? I’m guessing you need to be an Admin in order to do this.
I’ve been trying to think of a way to handle ‘forgot to change hash’ or ‘copied hash from other derivation’ issues that are especially confusing for those not familiar with how Nix works.
I think copying hash from another derivation instead of using all-zeros is a mistake people normally do not repeat, and probably not the biggest part of the learning curve in Nix.
Updating the version without updating the hash is a more frequent problem, and can happen again if one is a bit too tired.
We actually have an option of encouraging setting the name attribute that would incude version; this requires to update the name when source archive format changes, but that is rare and mistakes normally lead to a quick failure to unpack. Then the version is inside the hash for derivation path, together with content hash, but the true origin and true file name is not.
Another option is a check (ofborg check? git commit hook would be slow if it needs to implement this properly) that complains in case of version changes without any hash changes. I guess it is enough if it is «neutral», as hopefully people do look at neutrals.