These are excellent questions and I’ve been meaning to look into how Nix can/could be better than Conda.
I’d start by emailing your audience in advance (if you have that option) and ask them 1-3 problems they find in Conda. Or google around for blog posts complaining about Conda. That should give you good insight into what to talk about in context of how Nix can do it better.
I’m working on macOS funding campaign to which a few companies already committed to contribute. The goal is to fund Nix development related to macOS so that with time we’ll have excellent support, which as you said is really important for developer environments.
Most good libraries select optimized kernels based on the CPUID instruction, but there are exceptions. You can also build derivations with specific march or mtune flags. There was a recent discussion about this:
IANAL, but I think the MKL license would allow us to redistribute it, the only icky part is that it requires that MKL is not modified and the question is whether patchelf’ing is modification. The other roadblock is that Hydra does not build non-free software packages.
It’s pretty much the same story with CUDA. The CUDA Toolkit is distributed through Anaconda:
Licensing is more complicated for the whole CUDA stack, as far as I remember you cannot redistribute the driver libraries outside very specific circumstances (through the NVIDIA-provided Docker container). But I think most of the toolkit can be redistributed.
Same here. I strongly prefer Nix and I think it is better, but it requires an investment. It’s like writing machine learning code in Rust. I love it, but I would still recommend most of my colleagues to stick with Python + PyTorch for the time being. Doing machine learning in Rust is going off the beaten path and as a result, you have to do a lot of the plumbing yourself.
The are many paper cuts, but our current biggest problem [1] is that we do not have packages built against MKL and CUDA in our binary cache. This makes the user experience miserable. With conda or plain old pip, you are up and running in a few seconds. With nixpkgs, you are first off to one or two hours of builds. That is if the packages build at all, since we don’t build them in Hydra regressions are not always noticed.
CUDA is pretty much essential to any kind of high-performance numerical computing. MKL is still the dominant BLAS/LAPACK library, because it is faster in a lot of scenarios. E.g. a few month ago I benchmarked some of my transformer models again with various BLAS libraries, MKL was ~2 times faster than the best competition (OpenBLAS), due to having specific optimizations (e.g. batched GEMM). Another issue is that OpenBLAS is not usable in some cases, because it cannot be used in multi-threaded applications.
Another issue is that even without MKL and CUDA, builds of some libraries frequently fail, because there are still some pre-SSE4.2 machines in the Hydra cluster. So, even if you intend to use e.g. our PyTorch builds without MKL or CUDA, it frequently gets built locally because it is not in the binary cache.
[1] In my experience, machine learning + NLP pays my bills.
As others already said there isn’t a general yes / no answer to this. One data point: we run a lot of numerical code and for us nix is great because:
some people in team know nix
we almost exclusively run Linux
we run a lot of other stuff that’s not python/numerical code
But I know that one of our competitors tried nix and it was universally reviled:
bad experience on OSX
no windows support
often difficult to run software where you don’t have source (ubiquitous /bin/bash)
if you don’t have someone championing nix in the team and willing to iron out rough edges it’ll be a downward spiral of frustration and failure to bend nix to your will
IMO the last point is probably the most important. Just dumping nix on a team will almost certainly fail, but being there and helping with whatever weird situation people get stuck in could work. If you can’t provide that go with conda.
One thing to remember about conda is they changed their licensing last year:
“We clarified our definition of commercial usage in our Terms of Service in an update on Sept. 30, 2020. The new language states that use by individual hobbyists, students, universities, non-profit organizations, or businesses with less than 200 employees is allowed, and all other usage is considered commercial and thus requires a business relationship with Anaconda.”
This can be mitigated by either using some of the existing machine features such as big-parallel or we add another one like SSE42 to skip such hardware.
The good news all of these are solvable problems! The biggest unknown is windows support, the rest is something we’re making progress on.
I agree, the biggest blocker for Nix adoption is that knowledge spreads from person to person rather through text, audio or video.
As far as I understand, unfreeRedistributable is not built on Hydra, since it sets free = false, with the exception being unfreeRedistributableFirmware? The same applies to issl, the license used for MKL.
If we would permit building issl and/or unfreeRedistributable, that would be awesome! We could ship all the relevant libraries with MKL enabled, or at least offer a withMKL flavor.
CUDA licensing seems to be more complicated, since the license is really oriented towards including CUDA in an application. I guess someone from the foundation could contact NVIDIA to ask them about redistribution as part of a Linux distribution (nvidia-compute-license-questions@nvidia.com).
I guess a technical workaround would be to allow CUDA/cuDNN to be built on Hydra, but not uploading the output paths to Hydra (preferLocalBuild?). Then all the dependencies could be built, but CUDA would still be built locally (which is fast anyway).
Oh, that’s a great idea. How does this work? Can we add
requiredSystemFeatures = [ "sse42" ];
to the relevant derivations and then convince a Hydra admin to add this feature to several appropriate machines in the cluster? Or does the feature need to be added to Nix, like there is a kvm feature now?
I get questions about unfree packages and poetry2nix from time to time, and the lack of precompiled MKL packages is a major hassle for these users, to the point where some has gone with Conda instead of Nix.
We are not doing anyone any service by sticking to not pre-building these packages, and we are hampering adoption in a space where Nix could truly shine.
As I’ve noted in the linked issue, we also incentivize packaging unfree software from binaries instead of from source, which is probably the opposite of what we want.
About 6 years ago I switched from conda to nix for several reasons. Reproducibility, because I found it easier to write expressions, because the scientific code I wrote had to run on several machines (unfortunately I could not work with it on our cluster), and because it “made sense” to package all packages, regardless of language, with the same manager. It took quite some effort at the time: cleaning up the spread of Python packages across the tree mixing 2 and 3, packaging Jupyter packages, and so on, but we’ve come a long way since and I’d argue thinks are a lot easier nowadays, especially also with poetry2nix.
For myself, the performance improvement with MKL is no longer relevant, however, given all the effort that’s been put in to make MKL work with our packages, together with a package set of packages that work, not requiring “solving”, we should really take that leap and provide MKL compiled packages as well.
Having a binary cache and a build machine to build scientific non-free packages would be great. The package set could be relatively small yet still provide a lot of value.
Seeing also the activity surrounding pytorch and tensorflow in Nixpkgs, I think it would be really good having a separate Nix Scientific Computing community/organization that has the resources for building packages, of course with MKL.
Would it make sense for such an organization to provide a nix version recompiled with a different nix store location, that could be deployed on HPC clusters whithout root access, and a binary cache to go with it, rather than everyone rolling their own?
The store issue is a bigger issue than MKL (for me at least). Unless you have a good relationship with the cluster administrator, this makes nix a non-starter on a lot of HPC environments (or you have to run it via a singularity container)
Yes, I found this one earlier. Not too long, and uses the time available rather well. Decent short demonstrations of Conda’s instability, in the first 6 minutes.
I would also like to add that the devops experience is a lot nicer in a nix stack as opposed to a docker/oci based configuration. I’m not sure how relevant this is to HPC scenarios, as I don’t work in that space. But if you have a user-facing inference service, then these are still relevant.
You can share build outputs, so even if you target a docker file, you don’t have to deal with recomputing expensive docker layers.
Dockerfile imperative steps vs Nix’s dockerTools.buildImage { contents = [ a b c ];}
If you access to bare metal, you can also export your package as a service/nixos module. Especially nice with nixops.
You can create a [private] binary cache of commonly shared packages (dev environments, deployment artifacts)
Development / CI/CD / Deployment environment agreement
Nix as a “build aware” configuration language also allows some freedom to be able to configure multiple aspects of your configuration stack. (e.g. ports between machines, domains, etc)
For me, the freedom of being able to change, pin, or upgrade any dependency when building a package is not something I’ve ever found to be easy with other package managers. This comes at some complexity with using nix, but at the very least it’s possible and a first class citizen.
Also, with all the talk about centOS in the news, I’m not sure how the “stable/LTS distros” deal with situations like needing the latest stable rustc compiler or python (docker image?). As with nix, you can “opt in” pulling certain packages from unstable at pinned points in time.
When I was working at Microsoft, the ability to run tests against many version of python (even pre-releases) was appealing as well. Painful when you distribute a client sdk, but don’t have good ways to test for breakages in up-coming releases.
I concur. Having something like /opt/nix is much, much more palatable than /nix for the traditional IT but this has been brought up before even for non-HPC cases. For instance I have a mac which is managed centrally so I can’t use the Catalina workarounds to install Nix because I can’t create volumes.
I don’t have any experience with Singularity. It still requires good relationship with administrator, doesn’t it? [“Singularity must be installed as root to function properly.” though these aren’t the most recent docs, perhaps.] Or have you found that it’s easier to persuade administrators to install Singularity than to install Nix?
Do you have anything noteworthy to report on your experience with Singularity?
