Automatically upload built derivations to cache?

SKyletoft · February 13, 2023, 1:54pm

I’m currently uploading my built packages to my binary cache manually after I’ve built them (overridden emacs, gnome, patched kernel). This is annoying and easy to forget. Is there a way to configure my machines to always upload derivations it builds (but not copies) to a binary cache?

wamserma · February 13, 2023, 4:39pm

cachix watch-exec mycache nix-build or cachix watch-store mycache

nrdxp · February 13, 2023, 5:42pm

While the above answer does work for Cachix, the OP didn’t specifically mention the use of Cachix so just in case you are using something else, you can more generally define a post-build-hook to run after a build.

There are some caveats with this method that I tried to outline in this feature suggestion:

github.com/NixOS/nix

feature: add `upload-caches` setting

opened 06:31PM - 06 Oct 22 UTC

nrdxp

feature

**Is your feature request related to a problem? Please describe.** Right now, i…f one wishes to automatically upload build results to a given cache, they must rely on setting a `post-build-hook`. This is difficult for a few reasons: one because the environment of this script is incredibly minimal and it runs under `/` so you don't know where your source lives or which flake you are building from, etc. There is also the fact that the calling user depends on whether you are running nix in singler user or daemon mode, which affects where Nix looks for s3 credentials, for example. Then there is the issue where if the hook ends uncleanly that the entire build chain will stop unless `keep-going` is passed, meaning you have to carefully capture every failure condition in your script and ensure it never exits with anything but `0`. It's also suboptimal because if one calls `nix copy` from this script, the next nix build will not proceed until the upload has finished, which can substantially increase total build time. I worked around this in a work project by using `taskspooler` in the post-build-hook (i.e. `nix run nixpkgs#ts`) to send the `nix copy` command to a queue so that the next build could proceed. This is, however, overly convoluted and error prone. **Describe the solution you'd like** We already have all the copy logic available to us, so why not just add a new config setting `upload-caches` set to valid cache uris such as `upload-caches = s3://bucket?profile=my-profile&endpoint=some.endpoint.com`. The we can just have Nix do the equivalent of a `nix copy` after each build result. Ideally, the daemon could handle this in the background as it continues on to the next build. **Describe alternatives you've considered** We can just keep the post-build-hook and try to improve it some, but I think the overwhelming majority of folks would want a `post-build-hook` that just uploads build results to some cache, so why make them implement it from scratch? A higher level interface for this seems warranted. We could maybe improve this some by just calling the post build hook once, only after the build for the derivation which was passed on the command line finishes instead on each dependency built. Instead of passing the `OUT_PATHS` of the current derivation, we pass all `OUT_PATHS` of the both the current derivation and the all the dependencies that may have been built as well. We still have to wait for `nix copy` to finish before nix exits though, and it still seems overly convoluted compared to a simple nix setting. **Additional context** Here is an example of a [post-build-hook](https://github.com/nrdxp/nrdos/blob/master/src/nrdos/upload-to-cache.sh) I'm using for my own NixOS flake. I already removed the taskspooler logic since it overcomplicated the script and just resigned myself to waiting after each build for now.

But if you set the hook in your flake.nix as nixConfig.post-build-hook then it will only run for a given flake, so that approach is decent for project specific caching.

I will say though, that I no longer rely on the post-build-hook myself due to some of the pain points, and instead rely on CI to upload to project specific caches using std-action.

SKyletoft · February 13, 2023, 6:38pm

Is there some equivalent for non-flake derivations and overrides?

kevinmehall · February 13, 2023, 7:07pm

See Untrusted CI: Using Nix to get automatic trusted caching of untrusted builds for details on using post-build hooks.

nrdxp · February 13, 2023, 7:18pm

The post-build-hook is a nix.conf setting so its not specific to flakes in general, but if you set it globally you might end up uploading too much to your cache (since absolutely everything you build will trigger the hook).

If you are not using flakes and you are using a recent enough Nix binary, you can alternatively set it in your project shell (if you are using nix-shell or nix develop for your project) by exporting something like:
NIX_CONFIG="post-build-hook = /path/to/hook.sh" in the shell.

SKyletoft · February 13, 2023, 8:08pm

Uploading everything I build was the goal. Or does build include just downloading?

I set it globally for now, I have a couple of hundred gigs to spare on my server. Worst case I’ll just have to GC a bit more often.

Thanks

nrdxp · February 13, 2023, 9:01pm

the post-build-hook only runs on local builds, so nothing substituted will trigger it. So if you want to just set it globally then yeah, it sounds like that would be a decent solution for your use case.