So it doesn’t find libstdc++.so.6. Actually I find it more surprising that it does find all the other libraries, given that Bazelisk probably doesn’t patch the binaries that it downloads to use Nix store paths?
Anyway, does somebody know how this is supposed to be used? Create a FHS user env?
This is not going to work either, if someone succeed that would be awesome to read about it and have access to the code though.
The best solution I found for using bazel on NixOS is to use distrobox to get into your HOME but from a docker image handled by distrobox, it’s super easy to use:
nix-shell -p distrobox
distrobox create
distrobox enter
In this environment, you can use bazelisk and build bazel projects that aren’t using rules_nixpkgs
It’s actually ldd that’s finding all those libraries. Or more specifically, ld-linux.so is. I’m not sure why, since LD_LIBRARY_PATH is not set, but it will search the lib directories of all installed packages for libraries with the same name. The path doesn’t have to match.
The only path that has to match is that of the dynamic linker itself, so the real problem is:
This is only found since ldd directly calls your binary with that linker. That can be solved with patchelf --set-interpreter.
As for libstdc++, it’ll most likely exist if you’re in a shell with stdenv.cc.cc.lib, since that should add it to LD_LIBRARY_PATH. If not, you can add it to that variable using lib.strings.makeLibraryPath.
You’re not wrong though. This will only tape over the problems of running specifically the bazel binary on NixOS. Un-sandboxed builds, or any additional tools that bazel downloads, will fail too, and you’ll have to patchelf more things…
Alternatively, you can consider steam-run (yes), or that unholy flake that creates a symlink to a dynamic linker in /lib.
I use nix-ld (integrated in nixos now, even if I would like to slightly change the way it work to semi-enable it by default without breaking reproducibility… but it’s another question) to deal with this kind of issues to recreate the dynamic linker in /lib without using another OS (really sad to install nixos to use ubuntu) and it works great. I give here (goto nix-ld section) a list of libraries that is cool to have to run most softwares (I still need to complete that list).
Ahah, technically you can enable nix-ld only in the nix-shell by configuring the variables inside it this way you still have clear boundaries, reproducibility (better than in VM) and better integration with OS (that is now only a kind of “mirror” reflecting.the rix-shell configuration).
Nix-ld provides a shim layer for these types of binaries. It is installed in the same location where other Linux distributions install their link loader, ie. /lib64/ld-linux-x86-64.so.2 and then loads the actual link loader as specified in the environment variable NIX_LD .
Ah! Interesting approach, much better than I’d thought. That still relies on host configuration being the same though, your shell will not be reproducible on NixOS systems that don’t use this. It certainly isn’t “better than in VM” from a reproducibility perspective though. You can build those with nix too.
I wonder what the repercussions of enabling this in general would be, while leaving NIX_LD unset? It could then even print a useful error for unsuspecting binary users, instead of that ridiculously cryptic “file not found” message that the kernel spits out.
OTOH it would encourage more use of precompiled binaries without going through proper integration, which I don’t think is a good thing.
Anyway, I digress, I’m sure there has been discussion about this before.
Yes it’s really neat. Well for a VM the VM might download different versions of a package when it gets upgraded and it’s hard to write the full configuration of the VM in a single file to distribute (isn’t why you use nixos?), so it’s not great for reproductibility. With a nix-shell, you only need to ensure that nix-ld is enabled, and then the nix-shell can pin the loader/used libraries.
I completely agree about this misleading error, and NIX_LD definitely provides a clearer error (that could even be improved). In my opinion it would be a huge step ahead for beginner users that are not aware of the solutions to avoid this issue (see e.g. this post). My only issue with default integration is that I’d like first to solve this issue (should be fairly trivial) to add an additional environment variable to toogle nix_ld (and potentially other binaries like /bin/bash…) without the need to touch the annoying to configure NIX_LD_LOADER variable : Decouple configuration from activation · Issue #29 · Mic92/nix-ld · GitHub
Regarding the fact that it may encourage users not to package properly the packages… I’m not sure. As soon as you want to collaborate with non-nixos (but nix) users or distribute a package (nixpkgs/flake) you still need proper packaging. On the other hand it encourages people to stay on NixOs as we provide now an easy solution (and a clear error message) for running packages that can’t be patched easily (I know people that quitted NixOs for that reason and I would not encourage developers to use NixOs without that as npm/bazzel/… are close to impossible to patch easily… just try to install electron with some js library). And I feel really bad to recommend an ubuntu/whatever vm first because containers have limitations (no setuid, not possible to run sandbox/fuse commands inside needed for nix-build/appimage, poor os/direnv integration… see more reasons here GitHub - Mic92/nix-ld: Run unpatched dynamic binaries on NixOS), also it makes NixOs depend on other distributions, people will also ask why they should use nixos then and finally it will not provide any sort of reproducible environment (your environment may break any time you upgrade the vm and it’s hard to write in a single file how you setup the VM… isn’t it why you are using NixOs?).
Anyway maybe we should discuss this elsewhere. I’ll try to create a discussion on that wgen I have some time.
Thanks for posting this! I’m still having trouble getting nix-ld to work with bazel.
I’m using the bazel_5 nix package rather than bazelisk. It’s complaining about NIX_LD not being set even though it is set in my shell. I’ve tried setting the variable directly before the command as well as passing it through to the build process with --action_env and it still gives me an error saying that NIX_LD isn’t set.
Running bazel command
$ NIX_LD=$NIX_LD bazel build --symlink_prefix=/ //:gazelle --action_env=NIX_LD=$NIX_LD
INFO: Analyzed target //:gazelle (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/iain/.cache/bazel/_bazel_iain/ec25bac409c7a6e71e62e8d8714aed0b/external/go_sdk/BUILD.bazel:51:15: GoToolchainBinaryBuild external/go_sdk/builder failed: (Exit 1): bash failed: error executing command /nix/store/5ynbf6wszmggr0abwifdagrixgnya5vy-bash-5.2-p15/bin/bash -c ... (remaining 1 argument skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
cannot execute external/go_sdk/bin/go: You are trying to run an unpatched binary on nixos, but you have not configured NIX_LD or NIX_LD_x86_64-linux. See https://github.com/Mic92/nix-ld for more details
Target //:gazelle failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.533s, Critical Path: 0.37s
INFO: 3 processes: 3 internal.
FAILED: Build did NOT complete successfully
Running bazel downloaded binary without nix-ld env vars set
$ /home/iain/.cache/bazel/_bazel_iain/ec25bac409c7a6e71e62e8d8714aed0b/external/go_sdk/bin/go version
cannot execute /home/iain/.cache/bazel/_bazel_iain/ec25bac409c7a6e71e62e8d8714aed0b/external/go_sdk/bin/go: You are trying to run an unpatched binary on nixos, but you have not configured NIX_LD or NIX_LD_x86_64-linux. See https://github.com/Mic92/nix-ld for more details
Running bazel downloaded binary with the env vars set
$ /nix/store/i6a9j315dl95az9a9vjpgya90y6j9z66-go-1.20.2/bin/go version
go version go1.20.2 linux/amd64
So it works fine in the shell environment with nix-ld enbaled but bazel is either refusing to see the NIX_LD variable or not passing it through when it executes the binary.
I’ve had to resort to using distrobox for now as I need to be able to run bazel for work but I’d love to not need it and just the the nix-shell I have setup.
You need to pass more things to Bazel (especially Bazel 5):
to the action_env you need to pass NIX_LD (done) and NIX_LD_LIBRARY_PATH
to the host_action_env you also need to pass NIX_LD and NIX_LD_LIBRARY_PATH
You don’t need to set it to itself in the command line, though.
There’s another possible failure mode: this requires that rules pass the shell env to their actions, and some don’t. See https://github.com/bazelbuild/bazel/issues/12049 for more info on that. In some cases, you may have to patch rules directly to pass use_default_shell_env = True to actions.
There are other possible failure modes as well, but these should solve the bulk of issues.
Thanks for that! I’ve tried running it with those flags but I still get the same error.
I’ve changed the command to just build a go package which has no other dependencies so it should just be invoking the go compiler to rule out any other rules causing problems.
$ echo $NIX_LD
/nix/store/76l4v99sk83ylfwkz8wmwrm4s8h73rhd-glibc-2.35-224/lib/ld-linux-x86-64.so.2
$ echo $NIX_LD_LIBRARY_PATH
/nix/store/2w4k8nvdyiggz717ygbbxchpnxrqc6y9-gcc-12.2.0-lib/lib /nix/store/76l4v99sk83ylfwkz8wmwrm4s8h73rhd-glibc-2.35-224/lib
$ bazel build @com_github_benbjohnson_clock//:go_default_library --action_env=NIX_LD --action_env=NIX_LD_LIBRARY_PATH --host_action_env=NIX_LD --host_action_env=NIX_LD_LIBRARY_PATH
INFO: Analyzed target @com_github_benbjohnson_clock//:go_default_library (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/iain/.cache/bazel/_bazel_iain/ec25bac409c7a6e71e62e8d8714aed0b/external/go_sdk/BUILD.bazel:51:15: GoToolchainBinaryBuild external/go_sdk/builder failed: (Exit 1): bash failed: error executing command /nix/store/5ynbf6wszmggr0abwifdagrixgnya5vy-bash-5.2-p15/bin/bash -c ... (remaining 1 argument skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
cannot execute external/go_sdk/bin/go: You are trying to run an unpatched binary on nixos, but you have not configured NIX_LD or NIX_LD_x86_64-linux. See https://github.com/Mic92/nix-ld for more details
Target @com_github_benbjohnson_clock//:clock failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.561s, Critical Path: 0.39s
INFO: 3 processes: 3 internal.
FAILED: Build did NOT complete successfully
From what I can tell from the rules_go code, it sets use_default_shell_env = "PATH" not in go.env and if I run go env I don’t see any PATH variable so I believe that should be set to true?
Sorry, I was a bit unclear. You need to pass them as, e.g., --action_env=NIX_LD=$NIX_LD. You can use a tools/bazel helper script to set these dynamically without having to pass them in like this.
You can verify that they’re getting passed in correctly by running the build command with -s, which should show the environment as it is being executed.
Sorry I misunderstood! I’ve tried again passing them like that but I still get the error =\
$ bazel build @com_github_benbjohnson_clock//:go_default_library \
--action_env=NIX_LD=$NIX_LD \
--action_env=NIX_LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH \
--host_action_env=NIX_LD=$NIX_LD \
--host_action_env=NIX_LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
INFO: Analyzed target @com_github_benbjohnson_clock//:go_default_library (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/iain/.cache/bazel/_bazel_iain/ec25bac409c7a6e71e62e8d8714aed0b/external/go_sdk/BUILD.bazel:51:15: GoToolchainBinaryBuild external/go_sdk/builder failed: (Exit 1): bash failed: error executing command /nix/store/5ynbf6wszmggr0abwifdagrixgnya5vy-bash-5.2-p15/bin/bash -c ... (remaining 1 argument skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
cannot execute external/go_sdk/bin/go: You are trying to run an unpatched binary on nixos, but you have not configured NIX_LD or NIX_LD_x86_64-linux. See https://github.com/Mic92/nix-ld for more details
Target @com_github_benbjohnson_clock//:clock failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.095s, Critical Path: 0.39s
INFO: 3 processes: 3 internal.
FAILED: Build did NOT complete successfully
Update: run with --sandbox_debug, I can see that it’s running a bash command and only passing PATH and TMP_DIR
edit: actually python/starlark dict.update doesn’t return the new dict, does it? so you’ll have to have a separate line just above the return that’s just env.update(ctx.configuration.default_shell_env)