I am currently working on a project that involves building a large C++ codebase (around 7,000 files) using Nix. To speed up the build process(from 2 hours down to 10 mins), the project was already using Incredibuild, a tool that allows for distributed compilation across multiple hosts and cores. However, I’ve run into a problem related to Nix’s sandboxing features.
Incredibuild runs processes on remote machines in a secure sandbox. Everything each process requires to run properly is dynamically emulated by Incredibuild from the local host to the remote machine. Any output generated by the process – std output, errors, return codes, files generated, etc. – is automatically synched back to the local host, as if the process had been executed locally.
The Issue:
Nix’s build sandboxing disallows network access by default. Unfortunately, Incredibuild requires network access to distribute the build tasks to other machines. Disabling sandboxing entirely is not an ideal solution for me, as it introduces several issues with the local environment interfering with the builds. This interference affects reproducibility and causes other unpredictable behavior.
What I’ve Tried:
- Nix Build Flags: I’ve experimented with various Nix build flags, but couldn’t find a configuration that allows network access for Incredibuild while keeping other sandboxing features intact.
What I’m Looking For:
• Partial Sandboxing: Is there a way to selectively allow network access in the Nix build environment while retaining the other benefits of sandboxing?
• Incredibuild Integration Tips: Has anyone successfully integrated Incredibuild with a Nix build system? If so, any guidance on managing the network restrictions without completely disabling sandboxing would be greatly appreciated. My last resort would be to patch nix sources for this.