Transient issue building CMake-based package

Help
1

I’m getting an issue periodically in a handful of CMake-based gnuarm firmware packages where I get a build failure that looks like this:

Assembler messages:
Fatal error: can't create /build/build/foo_firmware/devel/share/foo_firmware/std_periph/misc.o: No such file or directory
make[3]: *** [Makefile:138: /build/build/foo_firmware/devel/share/foo_firmware/std_periph/misc.o] Error 1
make[2]: *** [CMakeFiles/foo_firmware_bootloader.dir/build.make:70: CMakeFiles/foo_firmware_bootloader] Error 2
make[1]: *** [CMakeFiles/Makefile2:955: CMakeFiles/foo_firmware_bootloader.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
Failed   <<< foo_firmware [21.1s, exited with code 2]

I’m using NixOS, and the issue goes away if I build unsandboxed, in a develop shell, or with --keep-failed. If I use breakpointHook (as suggested in Build breaks, but only *without* --keep-failed) then sure enough the path is missing. The issue seems to also just go away on its own sometimes, even retrying the build immediately afterward. So it seems like it must be something racy, but I’m at a bit of a loss for how to make headway on it.

2

My first impression is that the build doesn’t expect to use a build directory.

The nix cmake setupHook assumes that you want to use the standard build directory paradigm that most cmake projects use.

To disable the build directory behavior, add dontUseCmakeBuildDir = true; to your package build.

Here’s the setup hook: https://github.com/NixOS/nixpkgs/blob/2a42aa90298cba13d302605d00d4331b134ace98/pkgs/development/tools/build-managers/cmake/setup-hook.sh

Official docs: Nixpkgs 24.05 manual | Nix & NixOS

I tried to document this better in the manual in this PR, but lost interest due a lack of a clear way forward to getting it merged.

3

Unfortunately, this doesn’t change anything for me, since my package isn’t actually using the cmake-provided configurePhase. I’m using a custom Nix package builder which builds these packages using colcon, which is helpful for my scenario since colcon is also the workspace development tool, so there’s greater development/packaging consistency without forcing my users to directly work with Nix.

Since it definitely does seem to come up specifically in the cmake-within-cmake case, I may try to put together a MWE of that scenario and see if I can stimulate the issue to manifest.

4

I personally believe this is an anti-pattern, and that the cmake logic should be able to handle finding dependencies through find_package — CMake 3.31.0 Documentation

5

Agree in general! This is a bit of a special case as it’s an x86 “package” that wraps up some scripts and helper binaries with an ARM Cortex firmware— and the firmware is what’s built in the inner project.

It may well be possible to do this purely with add_subdirectory, but at this point it’s what’s worked for my org, and we’re just confused why it would fail to build on Nix.

6

is this publicly available?

I’m assuming there’s just some “we have a path which does …/…/…/…/…/some/path” from some deep file, which is “bumping” into the smaller directory structure.

7

It’s possible? But this is all working intermittently— even just kicking it a few times on the same machine, it eventually goes through. So I assume it’s got to be something racy.

Unfortunately it isn’t available publicly, but if I can’t come up with a parallel MWE, I may see about stripping down what we have to the point where it would be shareable.

8

Time to close the loop here! Turns out this wasn’t a CMake issue at all, but rather one with the Makefile that came with the bootloader we were using. Which, as it turns out, is open source— the problem is here:

The creation of the BUILDDIRS directories is a dependency of the final elf target, but not of the intermediate .o targets. Thanks for the suggestions and patience!