Doing builds on a binary cache

Hello all,

I’m trying to set up a binary cache for multiple users. I don’t particularly want to give ssh access to the server in question, which seems to rule out the users being able to populate the cache with the nix copy command. The instructions here: Binary Cache - NixOS Wiki mention that “one option is to build the packages directly on that [the binary cache] machine”, which seems like a pretty optimal outcome anyway. However, I cannot see any actual instructions on how to do this. Has anyone managed to get this up and working?

Thanks!

Looking at this again - I had first assumed that the instructions meant automatically, ie, when a user wants a particular package, it first gets built on the binary cache machine, then delivered to them, but maybe it means to manually build on the binary cache machine - hopefully not as this would be a bit cumbersome.

All of these various options are possible. Your decisions on where builds occur, who has permissions to write to the binary cache, what kind of cache it is, etc will define how to accomplish what you want.

In this case you seem to not want to give users access, but also want to allow them to populate the cache? Where exactly do you want to draw the line and with what mechanisms?

The easiest answer when it comes to running your own cache is often an S3-compatible one.

2 Likes

Thanks, maybe S3 is the way to go. At the moment the cache is on a server which has other useful stuff on, and is considered a little security-sensitive. What I was imagining might be possible is something like:

  • User A wants a new package, defined (let’s say) in a nix flake
  • After establishing that the package isn’t in user A’s local nix store, instead of building on user A’s local machine, Nix sends the derivation to the machine hosting the binary cache
  • The machine hosting the binary cache builds it instead
  • User A then retrieves it from the binary cache
    Is this something that can be automated?

With building-at-cache the issue seems to be the following:

  1. Everyone can run arbitrary code in namespace sandbox on that machine (by definition of the build). If it is sensitive, given that zerodays happen, you might not want it just for this reason.

  2. Source fetching also means that everyone can open connections from this machine’s IP.

If users can populate the cache from outside, there is of course no technical guarantee that someone doesn’t push a backdoored glibc build or something (which might also get used by a process on this machine if that version ends up used for the cache infrastructure).

S3-compatible storage (or whatever plain-file-storage thing you prefer) might indeed be your best bet if you believe the users won’t atttack each other via pushing malicious store paths. Whatever can serve static files and accept static files is good enough, the cache is just a bunch of narinfo files pointing to nar files.

1 Like

Thanks, you make a good point about the security issues there. I now have a cache up and running on S3, the documentation on how to do this was good. I do not expect everyone using the cache to understand how the cache works, and therefore I do not expect them to actively push to the cache using nix copy. What would be neat is if some hook could be added to the derivations they are using, so that after a local build has occurred (ie, in a situation where no cache existed), the built package gets pushed to the cache automatically. Is this common practice, or is there any pre-existing approach for this?

1 Like

I expect it is not a very common practice due to backdoor insertion concerns or something, so setting up dedicated build machines or giving people SSH on a single powerful machine for remote building is hopefully more typical.

Of course in many cases for social reasons you can be reasonably sure that a) people won’t do bad things on purpose, b) if anybody did, spreading malware via Nix cache would be among the lesser of the worries, c) lateral movement in case of attacker intrusion can be also achieved in simpler ways.

If your users do not know Nix, maybe you should just give them a wrapper script? Maybe the same script would manage flake repositories, but in any case when it builds stuff, it also tries to push.

1 Like