Hi,
I’ve NixOS installed on my system and I’m trying to build a package which is not in nixpkgs - it’s a golang project, but the (default) build instructions use make and there are a couple of targets in the repo. It’s probably possible to use the standard golang build process as the contents of the Makefile are quite simple but this might change and I’d prefer to build with the recommended approach if possible.
When trying to do this, I found that the build process had trouble downloading dependencies and then I found out about the whole sandboxing of the build process which I think is causing problems pulling in dependencies. I could not find many details on this - there is this post on discourse:
but it’s not clear to me when it applies in the build process and, more specifically, how this relates to downloading build dependencies (eg go mod vendor). Is it the case that all of the steps in the build process eg unpackPhase, configurePhase, checkPhase, buildPhase are run in a sandbox environment which is intended to have no network connectivity? (Yes, I understand the sandbox can be disabled but it is enabled by default in NixOS iiuc and I do think it’s sensible, so would like to figure out a sandbox compatible way of building this…)
Thanks, rgds,
Sean.
1 Like
Sandboxing is activated in all build stages and so there is no network access in any of them.
How do you fetch source code or other build dependencies then?
The answer is: fixed-output derivations.
Unlike normal derivations, fixed-output derivations (which are used by common nixpkgs helpers such as fetchFromGitHub, buildGoModule etc) have network access, but they have to provide a hash of the resulting store paths (tarball, directory, whatever is fetched) in order to guarantee purity.
The workflow to generate the hashes when creating nix packages is to either use a specific tool (like nix-prefetch-github) that downloads the URL and gives you the hash, or run the build with lib.fakeHash as placeholder and copy the actual hash from the hash mismatch error message.
If you look into some nixpkgs packages, the src attribute is usually a fixed-output derivation with a output hash, like fetchFromGitHub. The nix way is to fetch external sources in separate fixed-output derivations (for which you can calculate the hashes) and refer to those in the actual build derivation, instead of fetching dependencies inside the build derivation (for which calculating hashes is infeasible, since it would require running the whole build first; also the build would have to be binary-reproducible).
Go packages are usually built using buildGoModule (implementation/documentation). Internally it creates a fixed-output derivation for fetching the go modules, which is why you have to pass a vendorHash to it.
Now to your specific package, it really depends on what is going in the Makefile (it would help if you could share a link). If it is fetching dependencies from the internet during the build, that won’t work. I would recommend using buildGoModule and extending from that (using hooks like preConfigure, preBuild, preInstall, postInstall). You can, for example, fetch additional dependencies using fetchFromGitHub in a let ... in; statement and reference that in the build.
2 Likes
Thanks for your speedy response here @edwinistrator
Sandboxing is activated in all build stages and so there is no network access in any of them.
V helpful and unambiguous clarification - thanks!
How do you fetch source code or other build dependencies then?
The answer is: fixed-output derivations.
Unlike normal derivations, fixed-output derivations (which are used by common nixpkgs helpers such as fetchFromGitHub, buildGoModule etc) have network access, but they have to provide a hash of the | | resulting store paths (tarball, directory, whatever is fetched) in order to guarantee purity.
The workflow to generate the hashes when creating nix packages is to either use a specific tool (like | nix-prefetch-github) that downloads the URL and gives you the hash, or run the build with lib.fakeHash as placeholder and copy the actual hash from the hash mismatch error message.
Yes - I somehow got some of this having used standard buildGoModule builders in the past but I’m having issues in this slightly more complex scenario which highlighted the fact that I had no idea what’s going on under the hood.
If you look into some nixpkgs packages, the src attribute is usually a fixed-output derivation with a output hash, like fetchFromGitHub. The nix way is to fetch external sources in separate fixed-output derivations (for which you can calculate the hashes) and refer to those in the actual build derivation, instead of fetching dependencies inside the build derivation (for which calculating hashes is infeasible, since it would require running the whole build first; also the build would have to be binary-reproducible).
Right.
Go packages are usually built using buildGoModule (implementation/documentation). Internally it creates a fixed-output derivation for fetching the go modules, which is why you have to pass a vendorHash to it.
Now to your specific package, it really depends on what is going in the Makefile (it would help if you could share a link). If it is fetching dependencies from the internet during the build, that won’t work. I would recommend using buildGoModule and extending from that (using hooks like preConfigure, preBuild, preInstall, postInstall). You can, for example, fetch additional dependencies using fetchFromGitHub in a let ... in; statement and reference that in the build.
Great - thanks for the tip. The package I’m trying to build is this:
I didn’t see it in the nixpkgs search and I don’t think there are any PRs open for it (there is a different, unrelated package also called soci).
It’s not clear to me what’s possible in the preBuild, preConfigure etc hooks - are these run in a sandboxed environment also? I guess not if they can be used to pull in extra content…
More generally, I’m curious to know how the sandboxed environment actually works - is this documented anywhere?
Thanks, rgds,
Sean.
1 Like