Can't update a Go package, getting `go: inconsistent vendoring`

doronbehar · April 7, 2023, 8:19am

I’m trying to update pistol in Nixpkgs. Here’s the link to upstream’s go.mod file:

doronbehar/pistol/blob/d27976f2320fea00040ca658ffcb86bd083c43c0/go.mod

module github.com/doronbehar/pistol

go 1.19

require (
	github.com/adrg/xdg v0.4.0
	github.com/alecthomas/chroma/v2 v2.7.0
	github.com/alessio/shellescape v1.4.1
	github.com/alexflint/go-arg v1.4.3
	github.com/doronbehar/magicmime v0.1.1-0.20211127135329-3de4ff29dc49
	github.com/dustin/go-humanize v1.0.1
	github.com/mholt/archiver/v3 v3.5.1
	github.com/nwaples/rardecode v1.1.3
	github.com/sirupsen/logrus v1.9.0
)

require (
	github.com/alexflint/go-scalar v1.1.0 // indirect
	github.com/andybalholm/brotli v1.0.5 // indirect
	github.com/dlclark/regexp2 v1.8.1 // indirect

This file has been truncated. show original

I’m trying to update it to 0.3.3 and I get when I run nix build -Lf. pistol in Nixpkgs:

building
Building subPackage .
go: inconsistent vendoring in /build/source:
	github.com/adrg/xdg@v0.4.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/alecthomas/chroma/v2@v2.7.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/alessio/shellescape@v1.4.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/alexflint/go-arg@v1.4.3: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/doronbehar/magicmime@v0.1.1-0.20211127135329-3de4ff29dc49: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/dustin/go-humanize@v1.0.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/mholt/archiver/v3@v3.5.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/nwaples/rardecode@v1.1.3: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/sirupsen/logrus@v1.9.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/alexflint/go-scalar@v1.1.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/andybalholm/brotli@v1.0.5: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/dlclark/regexp2@v1.8.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/dsnet/compress@v0.0.2-0.20210315054119-f66993602bf5: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/golang/snappy@v0.0.4: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/klauspost/compress@v1.16.3: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/klauspost/pgzip@v1.2.5: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/pierrec/lz4/v4@v4.1.17: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/ulikunitz/xz@v0.5.11: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	github.com/xi2/xz@v0.0.0-20171230120015-48954b6210f8: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	golang.org/x/sys@v0.6.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt

	To ignore the vendor directory, use -mod=readonly or -mod=mod.
	To sync the vendor directory, run:
		go mod vendor

Why is that? I also tried with buildGo119Module - I get the same vendorSha256 and the same errors.

A similar experience is reported in keybase: 6.0.2 → 6.0.4 · Issue #219045 · NixOS/nixpkgs · GitHub .

kirillrdy · April 8, 2023, 9:44am

Hi, While I am actually also confused on whats going on, I found at least one hint

git clone https://github.com/doronbehar/pistol
nix-shell
make

go build -ldflags "-X 'main.Version=v0.3.3-git'" ./cmd/pistol
go: finding module for package github.com/adrg/xdg
go: finding module for package github.com/alessio/shellescape
go: finding module for package github.com/alexflint/go-arg
go: finding module for package github.com/sirupsen/logrus
go: finding module for package github.com/doronbehar/magicmime
go: finding module for package github.com/alecthomas/chroma/v2/formatters
go: finding module for package github.com/alecthomas/chroma/v2
go: finding module for package github.com/alecthomas/chroma/v2/lexers
go: finding module for package github.com/alecthomas/chroma/v2/styles
go: finding module for package github.com/mholt/archiver/v3
go: finding module for package github.com/nwaples/rardecode
go: finding module for package github.com/dustin/go-humanize
cmd/pistol/main.go:10:2: cannot query module due to -mod=vendor
internal_writers/chroma.go:8:2: cannot query module due to -mod=vendor
internal_writers/chroma.go:10:2: cannot query module due to -mod=vendor
internal_writers/chroma.go:11:2: cannot query module due to -mod=vendor
internal_writers/chroma.go:12:2: cannot query module due to -mod=vendor
previewer.go:15:2: cannot query module due to -mod=vendor
cmd/pistol/main.go:9:2: cannot query module due to -mod=vendor
cmd/pistol/main.go:8:2: cannot query module due to -mod=vendor
internal_writers/archives.go:14:2: cannot query module due to -mod=vendor
internal_writers/archives.go:12:2: cannot query module due to -mod=vendor
internal_writers/archives.go:10:2: cannot query module due to -mod=vendor
internal_writers/archives.go:13:2: cannot query module due to -mod=vendor
make: *** [Makefile:16: build] Error 1

but

GOFLAGS="" make
go build -ldflags "-X 'main.Version=v0.3.3-git'" ./cmd/pistol

if you run go env you can see that something is setting GOFLAGS=“-mod=vendor”

doronbehar · April 8, 2023, 12:03pm

Hey thanks for the hint! Indeed:

github.com

doronbehar/nixpkgs/blob/c592167f7ce9eb972dfb8fc1b8a53394eb4c8487/pkgs/build-support/go/module.nix#L176


      
                outputHashAlgo = "sha256";
              })
          ) // overrideModAttrs modArgs) else "";
          
          
package = stdenv.mkDerivation (args // {
            nativeBuildInputs = [ go ] ++ nativeBuildInputs;
          
          
  inherit (go) GOOS GOARCH;
          
          
  GO111MODULE = "on";
            GOFLAGS = lib.optionals (!proxyVendor) [ "-mod=vendor" ] ++ lib.optionals (!allowGoReference) [ "-trimpath" ];
            inherit CGO_ENABLED;
          
          
  configurePhase = args.configurePhase or ''
              runHook preConfigure
          
          
    export GOCACHE=$TMPDIR/go-cache
              export GOPATH="$TMPDIR/go"
              export GOPROXY=off
              export GOSUMDB=off
              cd "$modRoot"

And the comment near the proxyVendor argument says:

# Whether to fetch (go mod download) and proxy the vendor directory.
# This is useful if your code depends on c code and go mod tidy does not
# include the needed sources to build or if any dependency has case-insensitive
# conflicts which will produce platform dependant `vendorHash` checksums.

Hence it might make sense in my case to set proxyVendor = true;, since my package does require some C libraries, but there are no dependencies with C code in them. Anyway, after resetting the vendorHash, I get different errors:

Building subPackage ./cmd/pistol
go: downloading github.com/doronbehar/magicmime v0.1.1-0.20211127135329-3de4ff29dc49
go: downloading github.com/adrg/xdg v0.4.0
go: downloading github.com/alexflint/go-arg v1.4.3
go: downloading github.com/alessio/shellescape v1.4.1
go: downloading github.com/sirupsen/logrus v1.9.0
go: downloading github.com/alecthomas/chroma/v2 v2.7.0
go: downloading github.com/mholt/archiver/v3 v3.5.1
go: downloading github.com/dustin/go-humanize v1.0.1
go: downloading github.com/nwaples/rardecode v1.1.3
cmd/pistol/main.go:10:2: github.com/adrg/xdg@v0.4.0: reading file:///nix/store/ma1v7gjshgrr0kbd6c31jrv7dsb1ldml-pistol-0.3.3-go-modules/github.com/adrg/xdg/@v/v0.4.0.zip: no such file or directory
internal_writers/chroma.go:8:2: github.com/alecthomas/chroma/v2@v2.7.0: reading file:///nix/store/ma1v7gjshgrr0kbd6c31jrv7dsb1ldml-pistol-0.3.3-go-modules/github.com/alecthomas/chroma/v2/@v/v2.7.0.zip: no such file or directory
internal_writers/chroma.go:10:2: github.com/alecthomas/chroma/v2@v2.7.0: reading file:///nix/store/ma1v7gjshgrr0kbd6c31jrv7dsb1ldml-pistol-0.3.3-go-modules/github.com/alecthomas/chroma/v2/@v/v2.7.0.zip: no such file or directory
internal_writers/chroma.go:11:2: github.com/alecthomas/chroma/v2@v2.7.0: reading file:///nix/store/ma1v7gjshgrr0kbd6c31jrv7dsb1ldml-pistol-0.3.3-go-modules/github.com/alecthomas/chroma/v2/@v/v2.7.0.zip: no such file or directory
internal_writers/chroma.go:12:2: github.com/alecthomas/chroma/v2@v2.7.0: reading file:///nix/store/ma1v7gjshgrr0kbd6c31jrv7dsb1ldml-pistol-0.3.3-go-modules/github.com/alecthomas/chroma/v2/@v/v2.7.0.zip: no such file or directory
previewer.go:15:2: github.com/alessio/shellescape@v1.4.1: reading file:///nix/store/ma1v7gjshgrr0kbd6c31jrv7dsb1ldml-pistol-0.3.3-go-modules/github.com/alessio/shellescape/@v/v1.4.1.zip: no such file or directory
cmd/pistol/main.go:9:2: github.com/alexflint/go-arg@v1.4.3: reading file:///nix/store/ma1v7gjshgrr0kbd6c31jrv7dsb1ldml-pistol-0.3.3-go-modules/github.com/alexflint/go-arg/@v/v1.4.3.zip: no such file or directory
cmd/pistol/main.go:8:2: github.com/doronbehar/magicmime@v0.1.1-0.20211127135329-3de4ff29dc49: reading file:///nix/store/ma1v7gjshgrr0kbd6c31jrv7dsb1ldml-pistol-0.3.3-go-modules/github.com/doronbehar/magicmime/@v/v0.1.1-0.20211127135329-3de4ff29dc49.zip: no such file or directory
internal_writers/archives.go:14:2: github.com/dustin/go-humanize@v1.0.1: reading file:///nix/store/ma1v7gjshgrr0kbd6c31jrv7dsb1ldml-pistol-0.3.3-go-modules/github.com/dustin/go-humanize/@v/v1.0.1.zip: no such file or directory
internal_writers/archives.go:12:2: github.com/mholt/archiver/v3@v3.5.1: reading file:///nix/store/ma1v7gjshgrr0kbd6c31jrv7dsb1ldml-pistol-0.3.3-go-modules/github.com/mholt/archiver/v3/@v/v3.5.1.zip: no such file or directory
internal_writers/archives.go:10:2: github.com/nwaples/rardecode@v1.1.3: reading file:///nix/store/ma1v7gjshgrr0kbd6c31jrv7dsb1ldml-pistol-0.3.3-go-modules/github.com/nwaples/rardecode/@v/v1.1.3.zip: no such file or directory
internal_writers/archives.go:13:2: github.com/sirupsen/logrus@v1.9.0: reading file:///nix/store/ma1v7gjshgrr0kbd6c31jrv7dsb1ldml-pistol-0.3.3-go-modules/github.com/sirupsen/logrus/@v/v1.9.0.zip: no such file or directory

doronbehar · April 8, 2023, 9:32pm

I also tried to export GOFLAGS="" in preBuild hook, that didn’t work either.

kirillrdy · April 8, 2023, 10:01pm

i kinda get whats going on, but I dont understand why it works for other go packages.

the issue is that vendor directory is in wrong place

here is my diff

diff --git a/pkgs/tools/misc/pistol/default.nix b/pkgs/tools/misc/pistol/default.nix
index 930554d1869..cc1613f5e47 100644
--- a/pkgs/tools/misc/pistol/default.nix
+++ b/pkgs/tools/misc/pistol/default.nix
@@ -8,16 +8,21 @@
 
 buildGoModule rec {
   pname = "pistol";
-  version = "0.3.2";
+  version = "0.3.3";
 
   src = fetchFromGitHub {
     owner = "doronbehar";
     repo = pname;
     rev = "v${version}";
-    sha256 = "sha256-DGbWlEDI/qGNpdT8N79RsLxnUm4Sw8lHFRIBFd1GmK0=";
+    sha256 = "sha256-6V3Du3+0YliwzsvZolapBAnyqbsv1WZhWQkaldDR+Ok=";
   };
 
-  vendorSha256 = "sha256-poTd0lXRaJeDxwcw+h76NPC0mFB9nwm2vLLB5UUK1dk=";
+  vendorSha256 = "sha256-TJio8YN3gmGikOEW4C8oqtY+8G+eToxv+0J7vlblB08=";
+
+  postConfigure = ''
+    chmod 777 vendor
+    cp -a vendor/vendor/* vendor
+  '';
 
   doCheck = false;

doronbehar · April 8, 2023, 11:00pm

Thanks a lot for your help again. I finally figured this out :

This happens (probably only to me) due to installManPage in postBuild. The go-modules derivation inherits the postBuild that one would expect to be used by the main builder, not the builder of the go-modules derivation[1]:

github.com

NixOS/nixpkgs/blob/7f3f7ec38154b9f4eb551a67bfe4214db43d4200/pkgs/build-support/go/module.nix#L86-L92


      
          prePatch = args.prePatch or "";
          patches = args.patches or [];
          patchFlags = args.patchFlags or [];
          postPatch = args.postPatch or "";
          preBuild = args.preBuild or "";
          postBuild = args.postBuild or "";
          sourceRoot = args.sourceRoot or "";

Hence since go-modules runs my postBuild, the $out directory is already created due to installManPage (and the man page is installed there as well, without use of course). This makes the following command act differently:

github.com

NixOS/nixpkgs/blob/7f3f7ec38154b9f4eb551a67bfe4214db43d4200/pkgs/build-support/go/module.nix#L145


      
            runHook postBuild
          '';
          
          
installPhase = args.modInstallPhase or ''
            runHook preInstall
          
          
${if proxyVendor then ''
            rm -rf "''${GOPATH}/pkg/mod/cache/download/sumdb"
            cp -r --reflink=auto "''${GOPATH}/pkg/mod/cache/download" $out
          '' else ''
            cp -r --reflink=auto vendor $out
          ''}
          
          
  if ! [ "$(ls -A $out)" ]; then
              echo "vendor folder is empty, please set 'vendorHash = null;' or 'vendorSha256 = null;' in your expression"
              exit 10
            fi
          
          
  runHook postInstall
          '';

It copies the vendor directory into the $out directory, resulting in $out/vendor which becomes vendor/vendor in the main derivation.

[1]: This hooks inheritance behavior is something to be discussed and changed, IMO. Will open an issue or PR soon.