Hi guys,
I am relatively new to Nix package building and Go language with it’s concepts (modules, vendoring etc.). I need to build Nix packages for the main go-graphite components.
So far, I was able to locally build go-graphite/carbonapi with no issues by running nix-build with this derivation code in default.nix:
with import <nixpkgs> {};
buildGoModule rec {
pname = "carbonapi";
version = "0.14.2.1";
src = fetchFromGitHub {
owner = "go-graphite";
repo = "carbonapi";
rev = "${version}";
sha256 = "04rghwww4z2fkakwkhsafqs1s21hb19wrarzvrp5svxgipryd288";
};
vendorSha256 = null;
deleteVendor = true;
runVend = false;
}
I am using buildGoModule because I see a go.mod file in the repo root, and it works as expected.
But I don’t have success using the same approach to build grafana/carbon-relay-ng.
So my steps begin with creating a derivation similar to above:
with import <nixpkgs> {};
buildGoModule rec {
pname = "carbon-relay-ng";
version = "v0.13.0"; # latest available tag at the moment
src = fetchFromGitHub {
owner = "grafana";
repo = "carbon-relay-ng";
rev = "${version}";
sha256 = "0bxbcd244dimkrlwigx782d28ng187w8qvl700v9r33dv5nxmgvj";
};
vendorSha256 = null;
deleteVendor = true;
runVend = false;
}
When running nix-build, it fails with this:
building '/nix/store/5f8jnl43fjmjwbxww4jaajsnl5xqiqd1-carbon-relay-ng-v0.13.0.drv'...
unpacking sources
unpacking source archive /nix/store/cn4fkb942kan5qgw0mvdh9bl7biaag1w-source
source root is source
patching sources
configuring
building
Building subPackage ./aggregator
statsmt/latencyhistogram12h32.go:6:2: cannot find package "." in:
/private/tmp/nix-build-carbon-relay-ng-v0.13.0.drv-0/source/vendor/github.com/Dieterbe/artisanalhistogram/hist12h
statsmt/latencyhistogram15s32.go:7:2: cannot find package "." in:
/private/tmp/nix-build-carbon-relay-ng-v0.13.0.drv-0/source/vendor/github.com/Dieterbe/artisanalhistogram/hist15s
aggregator/aggregator.go:10:2: cannot find package "." in:
/private/tmp/nix-build-carbon-relay-ng-v0.13.0.drv-0/source/vendor/github.com/Dieterbe/go-metrics
stats/metrics_wrapper.go:9:2: cannot find package "." in:
/private/tmp/nix-build-carbon-relay-ng-v0.13.0.drv-0/source/vendor/github.com/Dieterbe/go-metrics/exp
statsmt/meter32.go:9:2: cannot find package "." in:
/private/tmp/nix-build-carbon-relay-ng-v0.13.0.drv-0/source/vendor/github.com/dgryski/go-linlog
statsmt/process_reporter.go:7:2: cannot find package "." in:
/private/tmp/nix-build-carbon-relay-ng-v0.13.0.drv-0/source/vendor/github.com/prometheus/procfs
statsmt/out_graphite.go:10:2: cannot find package "." in:
/private/tmp/nix-build-carbon-relay-ng-v0.13.0.drv-0/source/vendor/github.com/sirupsen/logrus
builder for '/nix/store/5f8jnl43fjmjwbxww4jaajsnl5xqiqd1-carbon-relay-ng-v0.13.0.drv' failed with exit code 1
error: build of '/nix/store/5f8jnl43fjmjwbxww4jaajsnl5xqiqd1-carbon-relay-ng-v0.13.0.drv' failed
As far as I understand, at this point I need to use vendorSha256 so that it can get the dependencies, so to get the right value, I set that to vendorSha256 = lib.fakeSha256; and get this:
unpacking source archive /nix/store/cn4fkb942kan5qgw0mvdh9bl7biaag1w-source
source root is source
patching sources
configuring
building
vendor folder does not exist, 'deleteVendor' is not needed
builder for '/nix/store/wqp6igs60606smald4ssv5zlvhz9llny-carbon-relay-ng-v0.13.0-go-modules.drv' failed with exit code 10
cannot build derivation '/nix/store/nawqw84hqjn85vb9qsi61dsrm58ydif6-carbon-relay-ng-v0.13.0.drv': 1 dependencies couldn't be built
deleteVendor means to delete the vendor folder from the sources. Since that folder does not exist, I set it to deleteVendor = false; and run nix-build again. At this point, it downloads many dependencies and gives me the expected value:
go: downloading github.com/kr/text v0.2.0
go: downloading ...
go: downloading ......
installing
hash mismatch in fixed-output derivation '/nix/store/h2dhkg6k4kjcyij5a57xpmz1fqdjzl6j-carbon-relay-ng-v0.13.0-go-modules':
wanted: sha256:0000000000000000000000000000000000000000000000000000
got: sha256:1c0jdhrj5048s0062va30byxvcm5l5cmd4d9kfb39cg1lfs644ik
So I set vendorSha256 to that value and build again, getting this next error:
unpacking sources
unpacking source archive /nix/store/cn4fkb942kan5qgw0mvdh9bl7biaag1w-source
source root is source
patching sources
configuring
building
Building subPackage ./aggregator
go: inconsistent vendoring in /private/tmp/nix-build-carbon-relay-ng-v0.13.0.drv-0/source:
github.com/Shopify/sarama@v1.19.0: is explicitly required in go.mod, but vendor/modules.txt indicates github.com/Shopify/sarama@v1.27.2
github.com/eapache/go-resiliency@v1.0.1-0.20160104191539-b86b1ec0dd42: is explicitly required in go.mod, but vendor/modules.txt indicates github.com/eapache/go-resiliency@v1.2.0
github.com/eapache/go-xerial-snappy@v0.0.0-20160609142408-bb955e01b934: is explicitly required in go.mod, but vendor/modules.txt indicates github.com/eapache/go-xerial-snappy@v0.0.0-20180814174437-776d5712da21
github.com/eapache/queue@v1.0.2: is explicitly required in go.mod, but vendor/modules.txt indicates github.com/eapache/queue@v1.1.0
github.com/golang/snappy@v0.0.0-20170119014723-7db9049039a0: is explicitly required in go.mod, but vendor/modules.txt indicates github.com/golang/snappy@v0.0.1
github.com/kr/pretty@v0.0.0-20160325215624-add1dbc86daf: is explicitly required in go.mod, but vendor/modules.txt indicates github.com/kr/pretty@v0.2.1
github.com/kr/text@v0.0.0-20150905224508-bb797dc4fb83: is explicitly required in go.mod, but vendor/modules.txt indicates github.com/kr/text@v0.2.0
github.com/pierrec/lz4@v0.0.0-20161206202305-5c9560bfa9ac: is explicitly required in go.mod, but vendor/modules.txt indicates github.com/pierrec/lz4@v2.5.2+incompatible
github.com/rcrowley/go-metrics@v0.0.0-20151130033752-7da7ed577850: is explicitly required in go.mod, but vendor/modules.txt indicates github.com/rcrowley/go-metrics@v0.0.0-20200313005456-10cdbea86bc0
github.com/stretchr/testify@v1.2.2: is explicitly required in go.mod, but vendor/modules.txt indicates github.com/stretchr/testify@v1.6.1
golang.org/x/crypto@v0.0.0-20181015023909-0c41d7ab0a0e: is explicitly required in go.mod, but vendor/modules.txt indicates golang.org/x/crypto@v0.0.0-20200820211705-5c72a883971a
golang.org/x/net@v0.0.0-20180112015858-5ccada7d0a7b: is explicitly required in go.mod, but vendor/modules.txt indicates golang.org/x/net@v0.0.0-20200904194848-62affa334b73
golang.org/x/sys@v0.0.0-20181022134430-8a28ead16f52: is explicitly required in go.mod, but vendor/modules.txt indicates golang.org/x/sys@v0.0.0-20200323222414-85ca7c5b95cd
golang.org/x/text@v0.3.1-0.20171227012246-e19ae1496984: is explicitly required in go.mod, but vendor/modules.txt indicates golang.org/x/text@v0.3.3
run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory
builder for '/nix/store/yrpfgkxbs8vhvx5dg9zf0m1kmrpfjw6x-carbon-relay-ng-v0.13.0.drv' failed with exit code 1
At this point, I tried many different things and possible combinations of vendorSha256, deleteVendor and runVend values without success, always resulting in inconsistent vendoring.
I see no vendor/modules.txt in the repo, so I wanted to see what the source code looks like before the build, so I added this piece to the derivation:
preBuild = ''
ls -al
'';
Surprisingly for me, it showed a vendor folder among other source files (I assume that setting vendorSha256 to a non-null value does that):
drwxr-xr-x 4 sergeykranga admin 128 Jan 1 1970 validate
dr-xr-xr-x 8 sergeykranga admin 256 Jan 28 11:25 vendor
-rwxr-xr-x 1 sergeykranga admin 236 Jan 1 1970 vendor_health.sh
I decided to check the contents of go.mod and vendor/modules.txt with preBuild:
preBuild = ''
echo "From go.mod:"
cat go.mod | grep Shopify
echo "From vendor/modules.txt:"
cat vendor/modules.txt | grep Shopify
'';
and got this:
From go.mod:
github.com/Shopify/sarama v1.19.0
From vendor/modules.txt:
# github.com/Shopify/sarama v1.27.2
github.com/Shopify/sarama
github.com/Shopify/sarama/tools/tls
This was interesting so I checked go.mod in the repo at v0.13.0 tag:
github.com/Shopify/sarama v1.19.0
The question is: why in go.mod it is set to an expected value, but the vendors/modules.txt file has a different version?
So I am stuck at this point, I was thinking to:
- try to do what go says to do - try to use
-mod=readonlywhen building, but I don’t know how to do that withbuildGoModule
Any help is appreciated!