Comparing Nix and Conda

there have been moments when (mini)conda with channel conda-forge was good and useful
but that is years ago

current issues make it almost useless:

(I have to check out mamba)

1 Like

After reading this topic top to bottom again, I am trying to connect the dots but I may be wrong doing so…
By no means I have any experience in HPC, but it seems that Nix Friday - [interview] floxdev - YouTube Flox might be able to fit in nicely in a HPC environment?
As mentioned also in this topic that Centos has been dropped by Redhat, now there is a window of opportunity for Nix(OS) as a great replacement for all those HPC environments running old Centos versions installed!
And with all the focus on Cuda arn’t we overlooking other options?
FOSDEM 2021 - GPU Computing Made Simple with the C++ Vulkan SDK & the C++ Kompute Framework (AMD, Qualcomm, NVIDIA & Friends)

https://towardsdatascience.com/machine-learning-and-data-processing-in-the-gpu-with-vulkan-kompute-c9350e5e5d3a

2 Likes

People are on PyTorch/Tensorflow/etc. on CUDA. It’s great that there are alternatives, but it’s useless to tell people who are heavily invested in an existing framework that there is an alternative. If we can’t support the libraries that they use + CUDA, then they will just go somewhere else.

People will only switch if it supports their existing frameworks and the alternative to CUDA as an option that works out of the box and is in the same ballpark performance-wise.

[tanget]
ROCm/HIP on AMD GPUs/accelerators probably comes closest, but anyone who has used those in the real-world will tell you that it is a miserable experience. You’ll encounter lots and lots of bugs, bug reports apparently go to /dev/null, and there are a lot of performance regression compared to NVIDIA. My previous employer purchased some Radeon VIIs. But we ran into so many issues that even the ardent AMD fans in the group were swearing by our NVIDIA GPUs.
[/tangent]

I think we should consider setting up sort-of-a SIG that collects money (or finds sponsors) to rent a build server and provide an additional binary cache for scientific computing. I am willing to put time in this, but I know that @tbenst and @zimbatm have done prior work on a scientific computing overlay, so I am hoping that they’ll also give weigh in on this topic.

2 Likes

import nixpkgs { allowUnfree = true; …} and it’s ad hoc again

If you don’t want to allow all unfree packages you can whitelist specific packages with allowUnfreePredicate or licenses with whitelistedLicenses, as described in NixOS - Nixpkgs 20.09 manual.

Probably I don’t really understand what you mean by ”ad hoc“ here, but wouldn’t the only other solution be to allow all unfree packages by default?

I think, the overlaying is really handy but the mechanism via the fixed-point recursion mixed with the different callPackages and mixed with the non-recursive .override* overriding is confusing, also to many other people. Therefore, in order to not damage things one doesn’t fully understand, it is sometimes recommended to do things in a certain way which then seems ad hoc to me, and somehow defeats the argument for nix that I understand what happens in general because it’s based on principles. In practice it makes no large difference if things are indeterminstic or only fragile. At least, if one’s hobby is not to investigate such problems.

3 Likes

Thank you, I didn’t know about the allowUnfreePredicate!

What I had in mind saying “ad hoc” (and it may’ve been not exactly the appropriate language) was rather that flakes themselves discard the licensing info. There’s no way to mark a flake that it imports nixpkgs with allowUnfree, nor is there a way to configure the system to allow building/installing “unfree” flakes on it.

1 Like

Are you aware of anyone successfully running Nix inside Singularity? (So far, we [one other person on our project: I have not had the time to take a serious look yet] have failed to get it to work.)

… and I wonder whether it wouldn’t be better to use singularity-tools.buildImage to create a Singularity image with all our software, rather than trying to create a Singularity image containing Nix and build our software within it.

I’m only aware of the existence of docker/singularity buildImage, but have never used them.

As a grid manager myself I completely agree with @mjlbach that asking a beleaguered administrator to manage Nix installations across hundreds of machines is a very hard sell. I’d also agree with @teh that it takes the strongest of Nix advocates to provide the leadership required to successfully introduce Nix-based workflows into an organization.

I would surely have given up on Nix myself if not for the promise of correctness, hermetic reproducible builds, and the incredibly vast and fresh set of curated build expressions in the Nixpkgs collection. We created flox with the express goals of managing Nix at scale and lowering the adoption barrier for Nix in an enterprise setting, and for us it has addressed most of the issues raised in this thread. (Thanks @JosW for making the connection!)

We’re really interested to see how what we’ve built can be of use in other HPC environments, and to that end I’d invite everyone to sign up for our open beta and give it a go.

3 Likes

Amen.

FWIW, from my perspectives as

  1. Nix hobbyist
  2. contributor to a relatively small publicly-funded scientific collaboration

I am unlikely to find the time and motivation to take a deeper look at flox unless and until it is open-sourced.

8 Likes

Thanks @jacg for the feedback - flox wouldn’t be possible without Nix and countless other open source projects, so we’re thrilled to give back to the open source community by sharing flox. For now we’re just asking for everyone’s patience and feedback as we navigate the [mostly legal] intricacies of a new project. With respect to open sourcing, all options remain on the table as we work to figure out the next steps in our journey.

Whatever form flox may take, I hope it will at least provide a fresh perspective on architectural issues that I think may be hampering Nix adoption. In my experience these were the key design choices that made it possible for us to deploy Nix at scale:

  1. Centralizing all Nix processing and state (including profiles) on a single pair of fault-tolerant “storehouse” servers
  2. Creating a “nixexprs” framework for managing/simplifying Nix expressions in an enterprise setting
  3. Providing automatic and effective CI/CD

The first of these is simply an idea/architecture, and it’s not even unique - I recall listening to Pierre-Antoine Bouttier at NixCon 2018 as he described the GRICAD HPC center’s approach of sharing out /nix as managed from “head” nodes over NFS, and wondering why more people didn’t deploy Nix in that way. The second is available for all to explore on the flox beta, and the third is just Hydra as driven by our “nixexprs” framework.

… and that’s it really. Of course, while this approach has worked for us it may not work for everyone, which is why we want to work with the community to figure out how flox can help Nix gain a foothold in more companies.

2 Likes

I think the only machine that doesn’t have SSE4.2 instructions in wendy, which should be retired by the end of the year.

For distributing CUDA and other packages that are unfree but redistributable, see Consider building unfreeRedistributable Packages · Issue #83884 · NixOS/nixpkgs · GitHub

I’m confident that arguments for supporting science and some firmware bring enough benefits that we can get this merged, but someone will have to do the diligence work and create PRs.

2 Likes

I’m happy to say that Graham made wendy build only big-parallel jobs so SSE4.2 issue is resolved!

5 Likes

Thanks! That is great news!

Indeed :slight_smile:

It seems that the biggest two blockers are:

a) licencing issues
b) getting started tutorial

I’d like to make a case for a) if someone can help. For b) I’d like to hear how data scientists use Nix today so that we find a simple way for newcomers to join!

1 Like

I’d like to invite those that are willing to help or needing help to join us on slack at GitHub - nix-community/nix-data: Standard set of packages and overlays for data-scientists [maintainer=@tbenst]

4 Likes

The Slack link returns “This link is no longer active. To join this workspace, you’ll need to ask the person who originally invited you for a new link.”. I would also suggest having a dedicated SIG here on Discourse. Slack could be problematic on corporate networks/laptops/etc.

2 Likes

Conda has cross platform dependency files. Something like this should be used in the future for all languages. But nix is a deterministic distribution system avoiding side effects.
What to use depends on your target. Sometimes packaging something for nix takes some time cause you’re missing N dependencies.
So eventually use best tool for your task at hand.

Beside deterministic sandboxed builds nix allows to copy built packages with dependencies to other machines etc.

Thanks, there’s a new link: https://join.slack.com/t/nix-data/shared_invite/zt-ca8csgcz-N9Fyh~tnoZPY8x5lE_slFA

We also have #data-science channel on discord now!

3 Likes

Blasting valuable information into the slackhole seems like an unnecessary sacrifice. I’d rather have those conversations in a more durable place like Discourse or Zulip that have permanence, tags, search, etc.

1 Like

There’s a search feature in discord with unlimited history plan.

Setting up Discourse is quite some work and it’s not possible to have it freely run anymore as NixOS did back in the days.

I don’t think the current slack setup is perfect, but the goal is to get people talking and document things into more permanent forms like READMEs.

1 Like