I’ve been using NixOS at home for several months now, and I think I’m getting close to convincing my employer to take it on as well, which will be excellent both for us and for Nix. I think I need a bit of advice from the gurus here though to help my case.
For background, my employer has a set of very large codebases, organized as a few monorepos (plus a huge number of scattered micro repos); we are thinking that a good fit for the builds of that will be Bazel, to give us all the caching, autofixing, cross language goodness. However that does nothing for the control of the runtime, or for deployment/distribution; plus quite a lot seeps into the hermetic builds. Hence Nix, which solves this side of life extremely nicely. Then further, we’ll want to use a decent IDE with built in syntax highlighting, jumping to definition/declaration, refactoring, etc etc - let’s say CLion as an example.
(Aside - I’m open to other IDEs of course; I’m fine with saying that only one IDE is supported out of the box as long as it covers C++/Java/JavaScript (and Rust would be great, not that we use it yet but a person can hope!).)
The question is how best to get all these tools to play with each other nicely? Both Bazel and Nix will want to orchestrate the build and control dependency versions; CLion will need to read the Bazel BUILD/WORKSPACE files (which should be fine, I know they’re working on that) and the nix definitions, understanding how they play together. Basically I want everything to be smooth as silk, and to just work out of the box, integrated all the way up the development chain, with all the beautiful features that you guys have written. Oh and this all needs to work on Windows as well, since we have end-user UI apps there. I’m figuring that WSL will be my saviour there as then I’ll be able to use Nix everywhere, but please let me know if I’m being overly optimistic…
Note that this whole thing will be basically from scratch (ported across from a current bit-of-a-mess-but-works), so we have a great deal of scope to get it right from the beginning. We are willing to entertain solutions that require real work, as long as the end result is stable and sustainable. Please help me get it right!
Thanks in advance for your help, I know I’m asking a lot.
Another approach would be to build a profile or shell.nix and first load the external dependencies using nix. One of the advantages with that approach is that the Bazel version can be controlled precisely. And CLion or another editor be brought in as well using that method. In that case the developer would first run a command to load the nix environment before invoking Bazel.
Regarding Windows support, this is still a challenge. If the need is to provide a developer experience then WSL is pretty good. But it doesn’t help if Windows executables need to be generated. In that case cross-compilation might save the day and both might be combined. See:
Excellent thank you for the info. It’s interesting - I wonder if the best approach is nix-wraps-bazel-wraps-nix: so we have an outer Nix env that controls the Bazel version, the IDE and other things for the dev env. Then inside that there’s WORKSPACE and BUILD stuff driving Bazel, pulling in Nix deps via nixpkgs. And then Bazel uses Nix envs both for the actual builds (so we use a Nix-defined toolchain), for the runtime of the tests, and as a final product (i.e. it generates a Nix definition). Does that make sense?
So re Windows support, it actually looks not-too-insane: we use Nix to drive WSL for setting up PATH etc, but the actual compiler/linker/etc are native Windows Visual stuff. Am I missing what would be wrong with that? Of course on Windows, by far the best debugger is Visual Studio… Pulling that into the solution seems like it’d be way more difficult though, right?
I’m one of the developers of GitHub - tweag/rules_nixpkgs: Rules for importing Nixpkgs packages into Bazel. and I think the best approach is nix-wraps-bazel-wraps-nix as you say. The two approaches that @zimbatm mentions are not mutually exclusive, and there are advantages to not putting all Nix dependencies required during a Bazel build inside a shell.nix. The two main ones are:
if you use rules_nixpkgs then you won’t have to invalidate the entire Bazel build cache each time you change the shell.nix file.
if you use rules_nixpkgs then Bazel knows exactly which parts of your build depend on which Nixpkgs package, so Bazel can interleave building some of the code with Nix downloading more packages. In this way, Bazel does not need to wait for Nix to be completely finished for it to start building code.
So putting non-build related tooling, as well as Bazel itself, in a shell.nix file sounds legit. But keep the Bazel build hermetic by having it declare the Nix build-specific dependencies explicitly in the WORKSPACE file.
I gave a lightning talk about this at BazelCon’18. The videos aren’t up yet but should be very soon. Tweag has been bazelifying large repos for a number of folks since the beginning of the year, so if you want to collaborate on this give us a ping. We’re documenting best practices as we go along. See e.g. here for how we use Bazel+Nix in the context of rules_haskell.
Awesome. Much to read and think about, thanks again! We may well be pinging you for collaboration. This approach makes total sense to me; I eagerly look forward to the videos of your talk!
I see from this how you pull dependencies into Bazel from Nix, which gives you more hermeticity for sure and is a great start. You don’t seem to go the extra mile and generate nix-based runtimes for your built artefacts - unless I’m misunderstanding? My thought was that I would want to generate nix environments from my Bazel build for both the tests and the final runtimes, thus ensuring that the thing I test is a) exactly the thing I run in production, and b) fully driven by my developer environment. Do you see what I mean, and am I correctly understanding what you’ve done?
Nice thanks, I’ll take a look. The trick would be (for example) to make the IDE understand the whole setup so well that when you change your Nix/Bazel files, it figures out how to reindex all the symbols seamlessly. Not trivial! But that’s what makes for a truly lovely dev experience, right?
Another complementary tool is direnv which I wrote a while ago.
Direnv is a shell extension that can change the environment variables depending on the current directory. It can change the PATH and load nix-shell environments without branching into a sub-shell. That way the developer is automatically using the right tooling when entering the project directory.
There is an ecosystem of IDE extensions so it’s possible to benefit from those properties as well. I haven’t checked for CLion, if it’s missing the integration typically takes bewteen 10 lines to a page of code.
Just to chip in here - direnv is a absolutely invaluable for our workflow here. Without nix it’s nifty enough on its own but it absolutely shines with nix. Try it out if you’re not using it.
PS. I have no horse in the direnv race - just a very happy user.
Slight update here, though not much to report - we are noodling around with nix + bazel, bootstrapping ourselves up the learning curve. Steep! Rewarding though. Further bulletins as events warrant.