Multiple outputs and cycle detected in build error

Help
1

I’m confused at how to make a derivation with multiple outputs, where the bin output points to the out output.

Nix (at both 2.25 and 2.26 versions) reports that I have a cycle, which I totally don’t understand. It should be a DAG: files in $out, files that refer to them in $bin.

What am I missing?

2

Looks like a nix bug. I don’t see any references to bin from out.

1 Like
3

Adding -vvvv even tells you where it finds the references:

$ nix build -L -vvvv --keep-failed
...
scanning for references for output 'bin' in temp location '/nix/store/fanwb23f2w12fv3020sfkvv04qxxywdp-test.drv.chroot/root/nix/store/w7v7gkpl25drm28kck39xjxw0j8mnf04-test-bin'
found reference to 'fd118hwh7d1ncib4mdw56ylv3g9k0iyj' at offset '14'
found reference to 'bp5fphhp0g023w16qiwqlaky5pkrfziv' at offset '88'
scanning for references for output 'out' in temp location '/nix/store/fanwb23f2w12fv3020sfkvv04qxxywdp-test.drv.chroot/root/nix/store/bp5fphhp0g023w16qiwqlaky5pkrfziv-test'
found reference to 'w7v7gkpl25drm28kck39xjxw0j8mnf04' at offset '12'
note: keeping build directory '/tmp/nix-build-test.drv-1/build'
lock released on '/nix/store/bp5fphhp0g023w16qiwqlaky5pkrfziv-test.lock'
lock released on '/nix/store/w7v7gkpl25drm28kck39xjxw0j8mnf04-test-bin.lock'
building of '/nix/store/fanwb23f2w12fv3020sfkvv04qxxywdp-test.drv^bin' from .drv file: goal destroyed
error: cycle detected in build of '/nix/store/fanwb23f2w12fv3020sfkvv04qxxywdp-test.drv' in the references of output 'bin' from output 'out'

I think found reference to 'w7v7gkpl25drm28kck39xjxw0j8mnf04' at offset '12' is a reference to bash, but for some reason it attributes it to the current derivation instead of attributing it to the input. No, $out contents does not have any outer references. But maybe (temporary) file path itself does?

4

It appear to reproduce all the way back to Nix 2.3:

let
  sources = import ./npins;
  pkgs = import sources.nixpkgs {};
in

pkgs.stdenvNoCC.mkDerivation {
  name = "test";
  outputs = [ "bin" "out" ];
  dontUnpack = true;

  preInstall = ''
    mkdir $out
    touch $out/foo
    mkdir $bin
    ln -s $out/foo $bin/bar
  '';
}
1 Like
5

I hit this. So far as I can tell, there is no --keep-failed which lets us get our hands on these files to determine where the reference is, which makes this tricky to debug. What’s happening is that reference scanning is running on the nar for these paths.

It looks like the reference creating the cycle is that $out is getting a file $out/nix-support/propagated-build-inputs which points at $bin. Not obvious where this is coming from.

6

Add a "dev" output and it works.

This is the result of one of the standard fixup hooks. I’m not really sure why it does what it does.

1 Like
7

OK, well, this is happening to openssl from nixpkgs in my setup for reasons that are hard to determine, so I feel that there is something in nixpkgs which needs fixing, but I haven’t put my hand on it yet.

For example, I compared the standard openssl build (which works) against my own one using nix-diff to compare derivations. There is nothing material different between the two, only the compiler. One gives a reference cycle during the build, and the other does not.

8

Observation: openssl has a dev output (of course).

I’ve been diffing the NIX_DEBUG = 7 logs, which include traces of the shell script.

The writing of this file is among the last things that happen and frustratingly the pipe target paths aren’t shown in the trace so it is difficult to determine where this is going wrong. The trace shows essentially nothing interesting different between the two:

+++ local 'po_dirty=bin dev out'
+++ set +o pipefail
++++ echo 'bin dev out'
++++ tr -s ' ' '\n'
++++ grep -v -F dev
++++ sort -u
++++ tr '\n' ' '
+++ propagatedBuildOutputs='bin out '
+++ set -o pipefail
+++ '[' -z 'bin out ' ']'
+++ mkdir -p /nix/store/5ll3kccsgh0giw51gqby8135yjba8a1s-openssl-aarch64-unknown-linux-gnu-3.3.2-dev/nix-support
+++ for output in $propagatedBuildOutputs
+++ echo -n ' /nix/store/a5nwxf802jfh0p55cd919scxm69dzlpb-openssl-aarch64-unknown-linux-gnu-3.3.2-bin'
+++ for output in $propagatedBuildOutputs
+++ echo -n ' /nix/store/rbw39g87zd8j4nm93680yin708glxrd0-openssl-aarch64-unknown-linux-gnu-3.3.2'
+++ return 0

I think it’s the second echo here creating the cycle. Perplexingly, it is present in the ‘good’ case as well as the bad case.

That it’s present in the good case can be seen with something as simple as:

nix-build --arg overlays '[(self: super: {
  openssl = super.openssl.overrideAttrs { NIX_DEBUG = 7; };
})]' -A openssl '<nixpkgs>'

(I don’t have a public reproducer for the bad case currently)

The key part of the log is right at the end, shortly before fixupPhase completed is printed.

Edit:

OK, that’s a red herring for openssl. For the original reproducer (shared by @PhilipTaron) we see mkdir -p $out/nix-support, but for openssl, it’s always $dev/nix-support (in both good and bad cases).

This is leading me to speculate that in my case there could actually be a cycle being introduced by switching out the compiler somehow. (Even though the compiler I’m interested in, pkgsLLVM.openssl isn’t always creating this situation).

Edit 2:

Argh, finally found it. I had a wrapper which was writing some files out, and one of those files was introducing a reference. It would be great if there was some way to get nix to tell you where the reference creating the cycle was; it was difficult to find buried inside a nontrivial build.

(edits rather than posts because discourse won’t let me make more replies currently).

1 Like
9

I agree! See this 10-year old issue: "Cycle detected in the references of '/nix/store/...'" message could be more useful · Issue #481 · NixOS/nix · GitHub

10

One interesting takeaway from that issue is that after the error, the paths actually do exist in the store after failure. This isn’t something I had realized or expected, because usually a build error implies the build result not being present in the store. This one fact would have gone some way to making it easier to diagnose what was happening.