Hey everyone,
I have been digging around in the core stdenv infrastructure recently and have come up with a robust solution for generating compile_commands.json files on nix.
First, let me explain some background. In the C++ world, the most popular LSP server is clangd. For clangd to function correctly, it needs a compile_commands.json, which is essentially just a list of all the compilation commands which were executed during a build process. This sounds simple, but because of the complexity of modern build systems, it can sometimes be hard to generate a compile_commands.json. Currently, there are three approaches:
- Cmake will generate a
compile_commands.jsonif asked. Unfortunately, this only works for cmake projects, and in my experience, is not super reliable. I have had projects in the past where cmake just refuses to generate acompile_commands.json. With nix it doesn’t work great either because we hide some of our compile flags inside the compiler wrappers. This can be worked around by reading environment variables fromnix-support. - The project bear will track system calls to try and figure out exactly which compile commands are being used. This is the only choice for build systems other than
cmakeThis works quite well on debian based systems, but not on nix, again because of the compiler wrappers. It is something they are actively working on, but it is just a difficult problem to solve generally. -
clanghas the pretty much undocumented flag-MJwhich generatescompile_commands.jsonfiles, but they need to be spliced together. I have never managed to get this approach working correctly.
On nix we religiously wrap our C compilers. This interferes with cmake and bear’s ability to capture compile commands accurately. I personally think the compiler wrappers are great, because they allow up to set up custom compiler environments very easily. It is also fairly straightforward to instrument the compiler wrappers so they they can be used to generate compile_commands.json files reliably! Here is a PR doing that:
https://github.com/NixOS/nixpkgs/pull/192694
It works as follow: It adds support for a post-wrapper-hook.sh in the nix-support of the wrapper. The post-wrapper-hook can be used to generate a compile_commands.json for any project using:
You use mini_compile_commands_client.py in the post-wrapper-hook.sh to extract the compile commands, which get sent through a unix socket to a running mini_compile_commands_server.py, that stores them and then writes them to an output file. The client server architecture is required here because usually, multiple compiler invocations are happening in parallel.
All of the code linked above is still very rough, but it is working. I have tested it on several large projects. Here is an example shell.nix which creates an environment with mini_compile_commands_client.py hooked into the compiler wrapper.
with (import <nixpkgs> {});
let llvm = llvmPackages_latest;
in (mkShell.override {stdenv = ( mini-compile-commands.wrap llvm.stdenv );}) {
buildInputs = [ cmake gtest ];
}
When the compiler is called, if a mini_compile_commands_server.py is running, it will send through the compiler commands.
Note, if you do want to try it out, you will need to rebuild large chunks of nixpkgs locally, so be careful!
I am writing this post because I think this is a very nice feature. In particular, it would allow us to have IDE integration with clangd for working on nix from nixos! I am pretty committed to getting it merged and I would love some community feedback about implementation details. As I mentioned above, this is still a work in progress. There a lot of obvious nits, and nothing is well documented right now. I am more interested in feedback about the approach in general.
Thanks for reading 