When I install python packages with withPackages, all packages are built from source. Either binaries from pypi nor from nix are used. It takes a very long time to install some packages. Is there anyway not to build them from source? Thanks!
1 Like
Maybe try mach-nix?
https://www.reddit.com/r/Python/comments/npu66t/reproducible_python_environment_with_nix_flake/
1 Like
In theory they can come from the cache, as long hydra has built them.
If you aren’t seeing non-python packages build from source and you don’t need a very specific python, you may be able to just switch to an interpreter they are built for. I’m not sure exactly what gets built/cached, but I just tried to open a nix-shell with nix-shell -p 'python<x>.withPackages(ps: with ps; [anyjson])' with several versions and found:
- python: built (used python27)
- python3: download from cache (used python38)
- python36: built
- python37: built
- python39: download from cache
If it’s more than just python, you may not be on a rev that is built, or you may be overriding a common dependency.
3 Likes
Thanks! I was using pkgs.python3Full.withPackages packages. It seems the whole python is built from source everytime with lots of tests. I change it to pkgs.python3.withPackages packages and now everything is downloaded from cache.
2 Likes
python3Full.pkgs aren’t build by hydra (which populates the cache), so yea, you will get the full gentoo experience 
2 Likes
Where can you get this information upfront if something have prebuilt binaries?
1 Like
If it’s listed here: https://hydra.nixos.org/eval/1723694#tabs-still-succeed or https://hydra.nixos.org/eval/1723716#tabs-still-succeed , then it’s available
2 Likes
Is there some tool to query it? like nix-index?
If you take unmodified packages from python38.pkgs or python39.pkgs, then they should be in the cache. If they’re not then either there was a false negative failing build.
With python ok, there is Full suffix. Same for git. So these are not built by hydra.
But there is also mongodb which is not build since some version bc mongodb change licence and hydra does not build packages under OSI unapproved licences.
So how can I check with some cli tool uprofront that mongodb-4_2 will be compiled ?
Python3Full is the standard python for many users, as it corresponds to the standard python on other systems.
It would be really great to have these prebuilt by hydra as well.
What is the process to get them included in hydra?
Is there a reason why they aren’t included in hydra?
@jonringer can you help?
In the meantime, you might want to try out “cachix watch-store” while you build your packages; that will put them into a semipublic cache… see https://github.com/mcdonc/.nixconfig/blob/561f42c8b01d387bc35948ab326abdb189224536/videos/cachix/script.rst#caching-new-store-paths-created-during-a-nixos-rebuild
1 Like
Adding those would double the work Hydra needs to do for the Python package sets and we need to choose what we can afford to build and what not.