nix-scheduler-hook is very easy to use. After building it, a binary named `nsh` is available. Simply set `build-hook` to the path to this binary, and create a `nsh.conf` file in a Nix configuration directory containing your Slurm JWT token (`slurm-jwt-token = token`) and the API host of the endpoint running slurmrestd (`slurm-api-host = host`). Full usage details are available in the README.
Because it depends on a job scheduler for scheduling builds, the value of the Nix `builders` setting / your `machines` file is ignored when using it as your build hook. This means the nodes in the cluster must be configured to be accessible by hostname in your SSH configuration file rather than just your machines file. I hope to eventually add a feature where builds are forwarded on to the regular build hook in decline cases (such as system mismatch) rather than simply declining.
It is currently in a very early stage of development, and thus there are a number of unresolved issues and missing features. For instance, `requiredSystemFeatures` is not yet supported, and the GC roots of the build results are not properly automatically cleaned up. However, it is in a basically usable state. Bug reports and enhancements are welcome!
This makes SLURM basically function like a custom remote builder, right? So it picks which node to run it on? (I assume it’s a single node and you can’t use the SLURM distributed features that dispatch the same job to multiple nodes to run as parallel processes?)
Yes, slurm decides what node to run each job on. In this case, I’m not sure what having multiple nodes per job would mean, since you normally can’t distribute a nix derivation build across multiple machines. In the case where there are multiple derivations to be built at the same time, multiple jobs will be submitted in parallel, like normal remote building.
I’ve been wanting to make something like this, but without assuming the slurm cluster has nix or a nix daemon available. For this i planned to use bwrap or apptainer/singularity.
My working idea is to somehow setup an environment with singularity/apptainer/bwrao bindmounts and overlays such that each slurm job can do its build in a tmpfs local-overlay-store, then with a post-build-hook upload the result back to the underlying store in a way that is concurency-safe.
For this to be efficient it makes sense to invoke this tool as a remote --store, and have the custom remote store fetch all paths available in binary caches before queuing slurm jobs.
Since the remote --store shim will have access to the full build graph, it also becomes possible to queue the jobs with --dependency on each other, allowing the later jobs to increase in priority while waiting for long-running preceding builds.
What you’re proposing sounds like it would require a modification to how Nix dispatches builds, since normally one hook invocation = one leaf dependency build, so it doesn’t have access to the entire build graph. At that point you might instead want your own nix build entrypoint that can queue up the jobs with --dependency on each other. How would the dependent jobs get the newly built dependencies from jobs that just completed on different machines?
The bwrap idea sounds awesome, and would definitely make the tool more accessible. If you ever build it, feel free to build off of what I’ve already done and I’d be happy to integrate it.
with `–eval-store local –store ssh-ng://login1.hpc.uni.edu` the remote becomes responsible for planning the full build graph afaiu.
This is what the post-build-hook is for, it is to upload the resulting store paths back to the underlying nix store in the home directory of the user, which then becomes the underlying fs of the overlayfs in the next dependant slurm job. (i assume the user has a network-mounted shared folder between all the worker machines, i have not used slurm in any other setting)
Thanks for the reference - based on feedback I got from the bionix people, I’ve added the ability to specify additional job submission parameters at the derivation level. The README has an example of how to do this.
A bit of a simpler solution I came up with for this problem is to just have the user install nixStatic to their home directory / shared location on the cluster. Just added support and documentation for this.
As of the latest release (0.5.0), you can now also use it with PBS clusters, and Slurm clusters that aren’t running slurmrestd (via job-scheduler = slurm-native). I plan to add support for other schedulers too, like Spectrum and SGE. Update: SGE is a non-starter because it doesn’t have the necessary API to retrieve the host a job is running on, and I can’t even obtain the community edition of Spectrum without having a customer account with IBM, so… RIP that.
Putting a nix store on your shared filesystem can come with its own problems, e.g. exhausting your file count quota. The naive solution here is to let the submitter be a bottleneck for transferring all dependencies between dependent jobs, which is what currently happens. A better solution would be to allow the submitter to just coordinate the copying directly between nodes, by remotely invoking nix-copy-closure or possibly even just copyPaths(sshStore1, sshStore2, ...) (I’m not sure if that would funnel everything through the local host or not). The submitter needs to at least coordinate this dynamically because we don’t know what nodes we’ll need to copy the results to at job submission time.
This only works if run as a daemon or a store, as I don’t think there’s a way to send build-hooks the full build graph.
Hello! Just wanted to post an update about the future of nix-scheduler-hook, for those who are interested. I am the mentor of a NixOS Google Summer of Code project with the focus to remove the build-hook mechanism entirely from Nix. This means that the way nix-scheduler-hook is currently invoked will not continue to be supported in the future. However, there is a plan in place to migrate it to supporting two alternative methods of invocation:
as a Store plugin, so you could use it like nix build --store nsh://slurm (the build results would still be automatically copied back to your local store)
as a daemon-like program wrapping a Store implementation, so you could use it like nix build --builders 'ssh-ng://localhost?remoteProgram=/path/to/nsh' (easier to configure, does not require messing with plugins)
As such, the future of nix-scheduler-hook (as well as all custom build hooks) is to transform into custom Store implementations, and extensions to the Store class in Nix are in active development to make this more ergonomic. The work done to migrate NSH to this format will form the basis of a general-purpose build-hook migration guide, for any other custom build hooks out there in the wild (although I do not personally know of any others besides mine). This guide would also be useful for future efforts to implement other custom remote building logic.