Petition to build and cache unfree packages on cache.nixos.org

1

I’m all about FOSS but for some things there are simply no sensible FOSS alternatives. In particular, anything and everything that touches CUDA is unfree and therefore not built by Hydra or cached on cache.nixos.org. Can we relax this constraint, at least for some packages?

I’m really running into trouble trying to use jaxlibWithCuda and tensorflowWithCuda right now. Building tensorflowWithCuda takes 80 CPU-hours. I maintain my own cachix cache, and build a subset of packages in GitHub Actions, automatically pushing them to my cache. But tensorflowWithCuda is so nasty that it times out the GitHub Actions limits (see eg this run).

How are users supposed to use CUDA-related software on NixOS?

Relevant PRs:

12 Likes
2

There’s some recent activity here: https://github.com/NixOS/nixpkgs/issues/83884

4 Likes
3

You could try using nixbuild.net - it even has a GH action. Making the resulting cache publicly available could be an alternative to your proposal, in a way.

4

My cachix is actually public already. I’m aware of nixbuild.net but unfortunately I don’t have the means to pay for all these builds myself.

2 Likes
5

Have you checked btw if you’re legally allowed to redistribute this? IIRC that’s one of the reasons why we don’t do that on cache.nixos.org .

1 Like
6

I would imagine that even if cudatoolkit/cuDNN are not redistributable, packages that compile against them ought to be redistributable?

7

Then beware of keeping non-redistributable stuff in closures. Otherwise the current substitution algorithm wouldn’t work well I think, even if you somehow convinced the cache to keep some holes in its closures.

8

Ok, now I’m even more confused… It appears that python3Packages.pytorch is cached on cache.nixos.org even though it links against CUDA/cuDNN as well. Why is PyTorch cached but tensorflowWithCuda and jaxlibWithCuda are not?

For example,

$ nix-shell -p python3Packages.pytorch
...
copying path '/nix/store/klmb0q1kyd8k00n0mfzyhjmvkz54vzvf-python3.9-pytorch-1.9.0' from 'https://cache.nixos.org'...
copying path '/nix/store/f5ghfax668nswhh1r60j4c5dyxbyhjps-python3.9-pytorch-1.9.0-dev' from 'https://cache.nixos.org'...
...
1 Like
9

From the pytorch-bin package:

    # Includes CUDA and Intel MKL, but redistributions of the binary are not limited.
    # https://docs.nvidia.com/cuda/eula/index.html
    # https://www.intel.com/content/www/us/en/developer/articles/license/onemkl-license-faq.html
    license = licenses.bsd3;

I agree, it’s odd though.

10

So presumably tensorflowWithCuda and jaxlibWithCuda could be made redistributable as well?

11

Well, I included the license bit, to demonstrate that it’s odd. It has a bsd3 license, when it should probably have a unfreeRedistributable license.

One option would be to have a config.acceptCudaLicense, and just have the meta check it before someone attempts to install.

2 Likes
12

The way we handle licenses in Nixpkgs is just not correct. The license that is added in meta is typically the license of the source. The license of the built artifact is not necessarily the same.
https://github.com/NixOS/nixpkgs/issues/106471

8 Likes
13

The meta.license system was never designed to handle such details. It was just a dumb string not too long ago.

14

Ok, my solution for now is just to give up on the source builds entirely (related commit). This may not be the nix-iest way but these massive builds make it impossible for me to be productive otherwise. I’ve already spent way too much time on this…

It’s a shame, because I really believe that Nix/NixOS has a lot to offer the reproducible research community. But without a “just works” solution for CUDA packages I can’t recommend Nix to other ML people in good conscience.

8 Likes
15

I think that’s fair. But I would also like to point out that cuda and similar frameworks have always been awkward when used in tandem with FOSS. If nvidia had cuda under mpl where they reserved patent rights, but allowed for a permissive license, this would be a non-issue.

6 Likes
16

Yeah, NVIDIA is not without fault here. They haven’t been the best community members in Linux/OSS world to say the least…

Unfortunately dealing with CUDA is just a necessary reality for the kind of work that I do. And as much as possible I’d like to do that work with NixOS :stuck_out_tongue:

3 Likes
17

@samuela if this is solvable for your particular use case with resources, and a binary cache of a few libraries is the last barrier to making reproducible research in (some area of) ml much better (using Nix), there are certainly those of us in academia who would be able to help out

What is expensive for an individual to front can often be really negligeable in an academic setting (especially given free bandwidth and spare servers).

Please DM or email me if you want to see if we can make this happen!

14 Likes
18

I will be a bit egotistical here, but why should us cache unfree software? Our lives are already harder because they are not open source.

Why should us make the lives of unfree software projects easier by giving them bandwidth, disk space and processor cycles for free?

3 Likes
19

@AndersonTorres How would you suggest that users do machine learning on NixOS?

(Also not to nitpick but in many cases unfree != open source != non-redistributable.)

4 Likes
20

Do not ask me. However, you have asked, so I will answer.

I am on the same side of Linus and his advice to Nvidia here. We should not bow to them merely because they monopolize the market.

If you are taking a more pragmatical way, just try to convince people with bandwidth and disk space and money to pay the eventual lawyers and money to pay the eventual sysadmins about your idea. I am only a humble user of an AMD Ryzen while I am trying to save some money for a RISC-V desktop…

1 Like