I’m finding Nix’s approach to boost python a little baffling. In Debian, a single Boost source package produces many binaries, among them a libboost-pythonX.YY which permits packages to depend on only the portions of Boost they need without inheriting random other dependencies, such as on Python:
https://packages.debian.org/bullseye/source/boost1.74
I expected to see this in Nix implemented with outputs, but instead it’s options. I tried overriding this in my overlay:
boost = prev.boost.override {
python = prev.python3;
enablePython = true;
};
But then when I went to build, it was suddenly rebuilding systemd, dbus, and even postgresql. Eek! Do I need to do this as a namespaced my.boost instead so that it doesn’t get picked up by all these other derivations? Or just make the override locally when needed, for example like this:
https://github.com/NixOS/nixpkgs/blob/8284fc30c84ea47e63209d1a892aca1dfcd6bdf3/pkgs/applications/networking/p2p/twister/default.nix#L16
Doesn’t this approach lead to a lot of copies of libraries like Boost, with different permutations of their options? Is this a Nix wart, or is this one of those things that’s actually good for me and I’ll love it when I understand it better?
1 Like
If you globally modify the Boost derivation in your system, then you need to rebuild everything that uses Boost since the versions in the Nix cache are likely not built with that Boost variant. I’ve had a quick look into the Boost derivation and one reason for this option seems to be that cross-compilation gets disabled when Python support is enabled. So there’s already a good reason to have this as a separate option.
Whether or not this is good design is probably a matter of taste. A boost-derivation in the Nix-store clocks in at 7MB, so I personally don’t mind having several of them around. From what I can judge at a quick glance, most of the available options seem alter the build quite a bit. Debug/Release, Static/Shared, etc. I tend to like this kind of design, but I admit, it takes some getting used to (which I am also still in the middle of).
What is the reason that you want to enable Python support in Boost? Is it for a project of yours that requires this? The general vibe I get from Nix is that this should always be done on the lowest possible level (i.e. in the derivation for your package or your dev-shell). Because, if you pass your derivation along to someone else, you cannot rely on them having the same global system configuration. That would defeat the whole purpose of Nix where your derivation is fully describing how to build your project or how your development-environment looks like.
Doesn’t this approach lead to a lot of copies of libraries like Boost, with different permutations of their options?
Yes, but in practice, most permutations aren’t needed.
Is this a Nix wart, or is this one of those things that’s actually good for me and I’ll love it when I understand it better?
More of a wart of trying to give python integration. Problem with python is that there’s more than one interpreter version. What would happen if I did python39Packages.python-mapnik, but it has a boost-python from python38… it would likely fail. In that case, you would need to specify the interpreter version anyway.
A boost-derivation in the Nix-store clocks in at 7MB, so I personally don’t mind having several of them around
If you have nix.autoOptimiseStore = true;, nix will only ever store a single copy of duplicated files:
If set to true, Nix automatically detects files in the store that have identical contents, and replaces them with hard links to a single copy. This saves disk space. If set to false (the default), you can still run nix-store --optimise to get rid of duplicate files.
It’s required for the python bindings https://github.com/miguelfreitas/twister-core/blob/323314ebf3e08bb2dd3e307e664a4ec055fa9dcc/configure.ac#L909
If the author doesn’t care about exposing the python bindings, they can just do:
configureFlags = [ "--disable-python-bindings" ];
Okay, thanks all! In my case, I’m not actually using twister-core— that was just an example to demonstrate ways I had found of trying to address this, while looking in nixpkgs. My situation is actually related to ROS use on Nix, where boost-python is fairly prevalent, eg:
https://github.com/ros-perception/vision_opencv/blob/906d326c146bd1c6fbccc4cd1268253890ac6e1c/cv_bridge/CMakeLists.txt#L9-L15
One reason to dislike this is situations like what I just encountered, where I had a package that uses PCL (pointcloud library), but also offers a python interface via boost-python. Using only the upstream packages resulted in a link conflict because of the multiple boosts. In order to get it to build, I had to override the PCL definition to also use the same boost. This resulted in a much more extensive rebuild than if PCL could have continued using its original non-python Boost, and my later boost-python-needing package could have simply mixed it in.
Looks like this has indeed been under discussion for some time:
https://github.com/NixOS/nixpkgs/issues/45462