User environments / direnv / lorri / bazel / buildfhsuserenv

Help
1

I’ve only been using NixOS for the last month or two, but I’ve gotten to the point where I’m relatively comfortable. I’ve migrated dotfiles, I have my machine built the way I want (doing things I hadn’t managed before NixOS, and it’s fantastic!).

There’s one part that I haven’t yet managed to get into place, and that is development environments. I’m used to using tools like pyenv and goenv to manage per-project configs, and so I’ve settled on direnv, and I’m using lorri to make that process a bit more efficient.

Where I’m running into problems, however, is supporting the build systems of the projects I work on. I can’t make everyone else migrate to NixOS, so I have to find ways of enabling my system to behave closer to how other linux systems work.

My main problem at this point, however, is building software with bazel. For those who don’t know, bazel attempts to bootstrap its build tools, to make it independent of the surrounding environment. It fails to be hermetic, however, because some of the most common build rules (e.g. go) download binaries that depend on the outer environment.

So far, I’ve run into two problems, the first is with binaries (like protoc) which depend upon libstdc++ and the second is the go binary, which seems to have incompatible linking.

For the first, I created a direnv where I update LD_LIBRARY_PATH to include stdenv.cc.cc.lib/lib, which allows protoc to run.

For the second, there’s a way to tell bazel to use the host go toolchain instead of downloading, which works.

So far, so good, but sometimes bazel runs programs inside of a sandbox (using the linux-sandbox binary), and now my carefully crafted LD_LIBRARY_PATH is lost, and protoc fails to execute again.

When I add a package to environmentPackages, it is symlinked into my environment. Is there a way to symlink stdenv.cc.cc.lib into my environment so that I can successfully build using bazel despite the linux-sandbox command? (I know, ironic name, since it’s not a fully functioning sandbox)

2 Likes
2

Personally, I use shell.nix to set up per-project environments, and buildFHSUserEnv in the shell.nix’s of projects that expect libraries to be in such specific places.

I still run into trouble from time to time, but haven’t had the bandwidth to explore the other available tools (direnv/lorri/flakes/devshell/…) yet, happy to hear about others’ experience :wink:

3

Do you have examples of your shell.nix files? Lorri actually uses shell.nix to build an environment, and then direnv to enter that environment when you change to that directory. (so your project must have a shell.nix and a .envrc in the root of the project)

Theoretically, then, I have the same setup as you, I just don’t seem to have configured my shell.nix properly.

4

For example https://git.sr.ht/~raboof/nix-shells/tree/main/item/github.com/akka/akka-grpc/shell-fhs.nix (though that particular project doesn’t need it anymore on the latest version :wink: )

(note this is not a bazel project - I have no experience with bazel specifically)

5

if protoc is needed in the sandbox then it should appear in the bazel graph (via tools or data keywords), you shouldnt have to craft LD_LIBRARY_PATH, especially is strictenv is enabled in next bazel versions.
GitHub - tweag/rules_nixpkgs: Rules for importing Nixpkgs packages into Bazel. may help you in that regard so that even your colleagues not on nixos can reap some nix benefits.

6

protoc is in the bazel graph, and it’s properly downloaded… Only it’s dynamically linked and expects to be able to find libstdc++.so.6

My problem is making sure that libstdc++ is available, even during the sandbox.

I’m using bazel 3.7.2, but I can try with newer versions (4.0) if that changes the way the dependencies are downloaded and/or compiled

7

That’s located with your C compiler, generally people use stdenv.cc.cc to locate it. stdenv.cc is the wrapped compiler, which is “nix aware”, stdenv.cc.cc is the unwrapped one with raw libraries.

8

@jonringer I’ve actually figured that out (mentioned above with regards to LD_LIBRARY_PATH).

I added to my .envrc: export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$(nix eval --raw nixpkgs.stdenv.cc.cc.lib)/lib"

That allows protoc to work when it’s not sandboxed. Once it’s sandboxed, however, my custom LD_LIBRARY_PATH is not available, and so the linker fails to find it.

Having looked at the man page for ld.so, it appears that the linker checks two different sections in the binary, LD_LIBRARY_PATH, /etc/ld.so.cache, /lib, /usr/lib, and potentially /lib64 and /usr/lib64.

That would mean that I need to either patch the binary using patchelf --set-rpath={????} or find a way to add symlinks to /lib or /usr/lib

Patching the binary that bazel downloads is a bit of a mess, so I’m going to try to add a symlink using configuration.nix

9

…or, figuring out how /etc/ld.so.cache works

10

the quick hack would be to use steam-run

nix-shell -p steam-run --run 'steam-run <command>'

This will run your command in an FHS environment.

If you want to stream-line it, you can use buildFHSUserEnv to create an environment with only relevant dependencies

11

@jonringer I’ll give a shot to steam-run on Monday and report back here (have to head home for covid curfew). As to buildFHSUserEnv, I know about it but still haven’t figured out how to use it. This was my shell.nix:

{ pkgs ? import <nixpkgs> {} }:

(pkgs.buildFHSUserEnv {
  name = "plezentek-projects-bazel";
  targetPkgs = pkgs: with pkgs; [
    bashInteractive
    glibc
    stdenv.cc.cc
  ];
  # profile = ''
  #   export LD_LIBRARY_PATH=${pkgs.stdenv.cc.cc.lib}/lib:$LD_LIBRARY_PATH
  # '';
  runScript = "bash";
}).env
12

That looks sane to me. What didn’t work about it? Perhaps you can share a small project demonstrating the problem?

13

@douglas At the risk of not being super useful, but I want to weigh in the reference to

That tool is definite part of the developer environment space.

14

@raboof I’ve put up a git repository at GitHub - dmayle/nixos-bazel-userenv-example: Example repo with different ways to setup a bazel user environment for development . It has a very simple bazel project in it which works on debian. I haven’t yet added in the nixos customization since my NixOS computer is at the office. I’ll do that on Monday morning.

On my system, that shell.nix seemed to work to create en environment, but bazel still wouldn’t build the proto library.

@blaggacao that looks really cool. On Monday, I’ll try building my enviroment with devshell, nix-shell, and lorri

15

One more advice: drop lorri for flakes * (or something else) if possible. Lorri behaves odd in non-obvious ways at times.

On the other hand: if it works for you, why not stick with it, just be prepared.

* flakes, as many other things in the nix ecosystem, are still “unstable” and who knows if they will one day become “stable” :wink:

16

Even though flakes are a pretty nice thing I don’t think it’s a good idea to start using it productively now, especially as a beginner, because

  • it’s unstable software (hence the name nixUnstable) which will have regressions, changed CLI interface etc. This is totally normal for unstable software, but especially when being new to an ecosystem, it’s probably not a good idea to start right off with it. I already had to debug a segfault (with unknown cause) twice on nixUnstable for instance.
  • the entire API is subject to change. There was an RFC, but it was closed. The fast adoption is kinda controversial considering that there’s no actual process to get out something standardized.
17

I really think your comment here deserves to be addressed more thoroughly, and while I was about to do so, I realized it might be more appropriate to start a new thread, rather than hijack this one. So here it is.

1 Like
18

Thanks to help from @raboof the nix-shell version works fine. Using the same shell.nix, lorry fails to properly setup the FHS environment. I’m not going to try the same with devshell

19

That was supposed to say “now try”, instead of “not try”. After looking at devshell, it’s not clear what it buys me other than using toml instead of nix. (I suppose the nag screen about ‘–add-root’ is a plus) Since I’m not providing dev environments to others, though, I think I’ll just try to pursue with lorri the question of why it isn’t set up properly.

20

After further research, it turns out that it’s a current limitation of direnv. direnv spawns a sub-shell to evaluate the environment, and then copies environment variables and updates the current shell. buildFHSUserEnv requires a subshell to properly setup the chroot that allows installation of directories like /lib and /usr/lib. Fixing lorri would require making changes to direnv to allow it to enter and exit a shell on directory change

1 Like