I don’t have the answer, by no means an HPC expert, but it is a very strange coincidence you ask this at this time.
Just a week ago I strumbled onto the following European project EESSI, European Environment for Scientific Software Installations https://www.eessi-hpc.com/
The European Environment for Scientific Software Installations (EESSI, pronounced as “easy“) is a brand new collaboration between different European HPC sites & industry partners, with the common goal to set up a shared repository of scientific software installations that can be used on a variety of systems, regardless of which flavor/version of Linux distribution or processor architecture is used, or whether it’s a full size HPC cluster, a cloud environment or a personal workstation.
A nice read can be found here, https://onlinelibrary.wiley.com/doi/10.1002/spe.3075
In this article they mention Prefix Gentoo also, in use with Cern CVFMS which is mentioned here in discourse also, Distributing the Nix store with CVMFS+Nix? - #4 by siscia
At thirst I thougt the EESSI was trying to reinvent the wheel because Nix would be able to achieve just what they are searching for?
And just as @toraritte in the above discourse topic was thinking about Flox reading about the CFMFS file system, so was I thinking Flox (@limeytexan @bpiv400) might fit in nicely into the EESSI project wishing.
Turns out Nix is even mentioned as an potential option in the article:
Other potential options for the compatibility layer included Nix38 (and GNU Guix39 which is based on the Nix package manager), which had previously been evaluated for this purpose by Compute Canada,40 though they also currently use Gentoo Prefix (see Section 8.1 for more on the collaboration between EESSI and Compute Canada).
But Gentoo Prefix was choosen for the Compute Canada, it’s overview can be read overhere,
https://www.researchgate.net/publication/334778484_Providing_a_Unified_Software_Environment_for_Canada’s_National_Advanced_Computing_Centers
4.1
In general, Nix provides an environment that is internally consistent but is best developed using Nix itself, through tools such as nix-shell.
Such tools can only be used if the Nix store is writable, an impossibility with CVMFS. Attempting to use Nix in conjunction with other development tools using environment variables can be fragile at times. Moreover, the dependency of the hash on the whole installation recipe, instead of only the dependencies (as, for example, implemented in Spack [8]), causes the store path to change even
if a bug fix or security update needs to be applied, aggravating this problem. We must note however, that some clusters in Compute Canada provide Nix with a writable Nix store, which is completely separate from the software stack described here.
There’s also a talk about the Compute Canada endeavour on YT, https://www.youtube.com/watch?v=n6rqjf3dmI4
By providing the above info I hope that follow up answers on your question can also take this info in account. I for one would be very interested why using Nix in their attempt was no success, and if the more experienced community members or the NixOS foundation could be beneficial in trying to get it working for them? Assuming they still want to try to use Nix though.
Sorry for rambling…