How to override and build certain package from master/main branch?

and homebrew seems to run “$ go mod tidy” before compiling the code which is probably why it works.

2 Likes

I’m not sure if this is as complicated for all Nixpkgs or is sops just a very complicated example to stumble upon.

Even when I try to run it with the go mod tidy in the prebuild it fails.

This is probably because network requests are not allowed in nix builds?

$ nix-build --expr 'with (import <nixpkgs> { });
(sops.override {
  buildGo122Module = args: buildGo124Module ( args // { vendorHash = "sha256-anKhfq3bIV27r/AvvhSSniKUf72gklv8waqRz0lKypQ="; });
}).overrideAttrs {
  src = fetchFromGitHub {
    owner = "getsops";
    repo = "sops";
    rev = "2eb776b01df5df04eee626da3e99e9717fffd9e0";
    hash = "sha256-VB4/DyQoQnV/AAXteJPsD2vbtAilZcJPTCXk2nvUZU8=";
  };
  preBuild = "go mod tidy";
}'
...
error: builder for '/nix/store/bjzp60dr7h8kgq6ipmcwsdg3fcdiiwxn-sops-3.9.4.drv' failed with exit code 1;
       last 25 log lines:
       > go: github.com/getsops/sops/v3/version imports
       >     github.com/blang/semver: module lookup disabled by GOPROXY=off
       > go: github.com/getsops/sops/v3/version imports
       >         github.com/hashicorp/go-cleanhttp: module lookup disabled by GOPROXY=off
       > go: github.com/getsops/sops/v3 tested by
       >     github.com/getsops/sops/v3.test imports
       >        github.com/stretchr/testify/assert: module lookup disabled by GOPROXY=off
       > go: github.com/getsops/sops/v3/cmd/sops/subcommand/filestatus tested by
       >     github.com/getsops/sops/v3/cmd/sops/subcommand/filestatus.test imports
       >         github.com/stretchr/testify/require: module lookup disabled by GOPROXY=off
       > go: github.com/getsops/sops/v3/gcpkms tested by
       >    github.com/getsops/sops/v3/gcpkms.test imports
       >         google.golang.org/genproto/googleapis/rpc/status: module lookup disabled by GOPROXY=off
       > go: github.com/getsops/sops/v3/gcpkms tested by
       >       github.com/getsops/sops/v3/gcpkms.test imports
       >         google.golang.org/grpc/metadata: module lookup disabled by GOPROXY=off
       > go: github.com/getsops/sops/v3/gcpkms tested by
       >        github.com/getsops/sops/v3/gcpkms.test imports
       >         google.golang.org/protobuf/proto: module lookup disabled by GOPROXY=off
       > go: github.com/getsops/sops/v3/gcpkms tested by
       >       github.com/getsops/sops/v3/gcpkms.test imports
       >         google.golang.org/protobuf/types/known/anypb: module lookup disabled by GOPROXY=off
       > go: github.com/getsops/sops/v3/hcvault tested by
       >  github.com/getsops/sops/v3/hcvault.test imports
       >        github.com/ory/dockertest/v3: module lookup disabled by GOPROXY=off
       For full logs, run 'nix log /nix/store/bjzp60dr7h8kgq6ipmcwsdg3fcdiiwxn-sops-3.9.4.drv'.

Yes. The builder intentionally has no internet access. You could run go mod tidy in that repo manually, create a patch and provide that in the override as well.

1 Like

sops is just massively unlucky in this case; my overrides are generally far simpler.

It seems to work in my fork where I committed the results of go mod tidy:

nix-build --expr 'with (import <nixpkgs> { });
(sops.override {
  buildGo122Module = args: buildGo124Module ( args // { vendorHash = "sha256-anKhfq3bIV27r/AvvhSSniKUf72gklv8waqRz0lKypQ="; });
}).overrideAttrs {
  src = fetchFromGitHub {
    owner = "onnimonni";
    repo = "sops";
    rev = "2c3bb510ddb28531e32ac2286f1cf3ff69f0c2ac";
    hash = "sha256-VB4/DyQoQnV/AAXteJPsD2vbtAilZcJPTCXk2nvUZU8=";
  };
}'

I will create a PR into sops and ask about this.

Thanks for helping me to learn something today :+1:

2 Likes

Would you be able to show me an example of this by providing link to somewhere or code of how to do it?

I‘ve never done this myself so don‘t quote me on anything. But my understanding would be to use git to create the patch (only works if you haven’t committed your changes yet)

git diff > mypatch.patch

and reference that like

patches = [ ./mypatch.patch ];

as you do with vendorHash.

Also see: nixpkgs/pkgs/build-support/go/module.nix at 6509831a8861818b9b304bc2f16fe2e8aa5c012c · NixOS/nixpkgs · GitHub

As I said, I never tried this before. Something I can highly recommend for learning is reading code in nixpgs. A lot!

I was finally able to make even the original untouched main branch to work by looking again on the sops/package.nix file in nixpkgs.

It contained:

postPatch = ''
    substituteInPlace go.mod \
      --replace-fail "go 1.22" "go 1.22.7"
  '';

And now overriding golang to 1.23.0 and using that I was able to make it compile:

nix-build --expr 'with (import <nixpkgs> { });
(sops.override {
  buildGo122Module = args: buildGo123Module ( args // { vendorHash = "sha256-anKhfq3bIV27r/AvvhSSniKUf72gklv8waqRz0lKypQ="; });
}).overrideAttrs {
  src = fetchFromGitHub {
    owner = "getsops";
    repo = "sops";
    rev = "2eb776b01df5df04eee626da3e99e9717fffd9e0";
    hash = "sha256-VB4/DyQoQnV/AAXteJPsD2vbtAilZcJPTCXk2nvUZU8=";
  };
  postPatch = \'\'
    substituteInPlace go.mod \
      --replace-fail "go 1.22" "go 1.23.0"
  \'\';
}'

Seems interesting that the postPatch step worked but running go tidy mod in the preBuild step did not.

1 Like

Builds (except for fixed-output derivations) are network-sandboxed as mentioned earlier, so go mod tidy wouldn’t work unless you used a FOD.

2 Likes

I was able to compile sops from the main branch and use it with devenv too.

Here’s a example commit if someone in the future is wondering about this: Build unreleased sops 3.10 from main branch to support age-plugin-se … · midworkhq/midwork@c4d3dfa · GitHub

It would be really neat to be able to figure out how to get the current golang version from the buildGo123Module to be able to do this without hardcoding it:

postPatch = ''
    substituteInPlace go.mod \
      --replace-fail "go 1.22" "go 1.23.0"
  '';

If someone in the future will figure this out please let me know :+1:

The builder has a passthru.go attribute, so something like this should work:

nix-build --expr 'with (import <nixpkgs> { });
(sops.override {
  buildGo122Module = args: buildGo123Module ( args // { vendorHash = "sha256-anKhfq3bIV27r/AvvhSSniKUf72gklv8waqRz0lKypQ="; });
}).overrideAttrs (finalAttrs: _: {
  src = fetchFromGitHub {
    owner = "getsops";
    repo = "sops";
    rev = "2eb776b01df5df04eee626da3e99e9717fffd9e0";
    hash = "sha256-VB4/DyQoQnV/AAXteJPsD2vbtAilZcJPTCXk2nvUZU8=";
  };
  postPatch = '"''"'
    substituteInPlace go.mod \
      --replace-fail "go 1.22" "go ${finalAttrs.passthru.go.version}"
  '"''"';
})'

See https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs regarding the args passed to overrideAttrs.

1 Like

It worked beatifully!

Thanks :bowing_man:!