How can I compile the flux(ctl) Go package?

Help
1

Hello,

I’m a Go newbie which is trying to build the Flux package, whose
source code can be found at GitHub - fluxcd/flux: Successor: https://github.com/fluxcd/flux2
. There’s a brief document with instructions here:
https://github.com/weaveworks/flux/blob/4fa5fd4e487cc6b8670b44d47cd99e92924edb3f/site/building.md

Following this and the scarce documentation in the nixpkgs manual
I ended up with using dep2nix
to convert the dependencies to the Nix equivalent deps.nix and I’ve created the
following recipe (default.nix):

with import <nixpkgs> {};

buildGo110Package rec {
  name = "fluxctl";
  version = "1.11.1";

  goPackagePath = "github.com/weaveworks/flux";

  src = ./.;

  goDeps = ./deps.nix;

  meta = with stdenv.lib; {
    description = "The GitOps Kubernetes operator";
    license = licenses.apache;
    homepage = https://github.com/weaveworks/flux;
  };
}

but this is in part a failure (see https://termbin.com/t8ey for the
full log) which
shows lines where the compiler complains writing:

# github.com/weaveworks/flux/registry
go/src/github.com/weaveworks/flux/registry/client.go:115:42: cannot use &manifestDigest (type *"github.com/opencontainers/go-digest".Digest) as type *"github.com/docker/distribution/vendor/github.com/opencontainers/go-digest".Digest in argument to "github.com/docker/distribution/registry/client".ReturnContentDigest
go/src/github.com/weaveworks/flux/registry/client.go:116:56: cannot use "github.com/opencontainers/go-digest".Digest(ref) (type "github.com/opencontainers/go-digest".Digest) as type "github.com/docker/distribution/vendor/github.com/opencontainers/go-digest".Digest in argument to manifests.Get

but incredibly the compilation continues (so i thought it was just a
warning), but ends with an obscure error 28 which i failed to look up:

[...]
k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset
builder for '/nix/store/axzjwjs2wyk3yvm9537hz42mlak2q00g-fluxctl.drv' failed with exit code 28
error: build of '/nix/store/axzjwjs2wyk3yvm9537hz42mlak2q00g-fluxctl.drv' failed

No one (nor in the #nixos channel or on the Flux’s slack channel) had
a clue, but investigating that “vendor” thing mentioned in the logs
wondered if that dep ensure mentioned on the building document
is somewhat missing, so I ended up with one last try:

with import <nixpkgs> {};

buildGo110Package rec {
  name = "fluxctl";
  version = "1.11.1";

  buildInputs = [ go_1_10 dep ];

  goPackagePath = "github.com/weaveworks/flux";

  src = ./.;

  goDeps = ./deps.nix;

  postConfigure = ''
    cd go/src/$goPackagePath
    dep ensure
  '';

  meta = with stdenv.lib; {
    description = "The GitOps Kubernetes operator";
    license = licenses.apache;
    homepage = https://github.com/weaveworks/flux;
  };
}

but it ends with an error too ( for the full log see
https://termbin.com/i4n0 )

[...]
The following issues were found in Gopkg.toml:

  ✗ unable to deduce repository and source type for "k8s.io/api": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://k8s.io/api?go-get=1": Get http://k8s.io/api?go-get=1: dial tcp: lookup k8s.io on [::1]:53: read udp [::1]:40298->[::1]:53: read: connection refused
  ✗ unable to deduce repository and source type for "k8s.io/client-go": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://k8s.io/client-go?go-get=1": Get http://k8s.io/client-go?go-get=1: dial tcp: lookup k8s.io on [::1]:53: read udp [::1]:40298->[::1]:53: read: connection refused
  ✗ unable to deduce repository and source type for "k8s.io/code-generator": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://k8s.io/code-generator?go-get=1": Get http://k8s.io/code-generator?go-get=1: dial tcp: lookup k8s.io on [::1]:53: read udp [::1]:40298->[::1]:53: read: connection refused
  ✗ unable to deduce repository and source type for "k8s.io/apiextensions-apiserver": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://k8s.io/apiextensions-apiserver?go-get=1": Get http://k8s.io/apiextensions-apiserver?go-get=1: dial tcp: lookup k8s.io on [::1]:53: read udp [::1]:40298->[::1]:53: read: connection refused
  ✗ unable to deduce repository and source type for "k8s.io/apimachinery": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://k8s.io/apimachinery?go-get=1": Get http://k8s.io/apimachinery?go-get=1: dial tcp: lookup k8s.io on [::1]:53: read udp [::1]:40298->[::1]:53: read: connection refused
  ✗ unable to deduce repository and source type for "k8s.io/helm": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://k8s.io/helm?go-get=1": Get http://k8s.io/helm?go-get=1: dial tcp: lookup k8s.io on [::1]:53: read udp [::1]:40298->[::1]:53: read: connection refused

ProjectRoot name validation failed
builder for '/nix/store/rlfvaw0nspzmzds0gzirzyqwqsrvg0j2-fluxctl.drv' failed with exit code 1

Now, I’m all at sea, If anyone knows what to do he/she is very welcome
:wink:

Also, I’ve yet to figure out how to compile just the
$(GOPATH)/bin/fluxctl (as it’s mentioned in the Makefile) which is
what’s really interesting to me.

2

The nixpkgs manual mentions a subPackages attribute for buildGoPackage which I’m using like so in a similar derivation:

{ stdenv, buildGoPackage, fetchgit }:

buildGoPackage rec {
  name = "operator-sdk-${version}";
  version = "0.6.0";
  rev = "61e0c23e9d2e217f8d95ac104a8f2545c102b5c3";

  goPackagePath = "github.com/operator-framework/operator-sdk";
  subPackages = [ "commands/operator-sdk" ];

  # nix-prefetch-git https://github.com/operator-framework/operator-sdk.git --rev v0.6.0
  src = fetchgit {
    inherit rev;
    url = https://github.com/operator-framework/operator-sdk;
    sha256 = "0h8pjc9lxwyxr9jsqzaaky6ka1h3cwksz7rcx7nwpad4ddgnjkzf";
  };

  goDeps = ./deps.nix;

  meta = with stdenv.lib; {
    description = "SDK for building Kubernetes applications. Provides high level APIs, useful abstractions, and project scaffolding. ";
    license = licenses.asl20;
    homepage = https://coreos.com/operators;
  };
}

As someone who consumes a lot of Golang-built infrastructure tooling without writing much of it myself, I struggle a lot of the time when writing Nix derivations, and a minority of the time I cop out and package the project’s pre-built binaries instead. The introduction of Go modules made that even worse for me, for not better.

I haven’t had a chance to look into the recently announced “new Go/Nix infrastructure” that should help with this last piece, but it honestly won’t matter until more of my runtime tools switch from Glide/Dep/whatever to the module system, which still isn’t a given.

3

Just found my workflow notes to myself for packaging this particular utility, which hopefully might help. Notably I do any of the dep commands outside of the context of Nix and let it provide that content via the goDeps attribute courtesy of dep2nix.

cd $(mktemp -d)
export GOPATH=$PWD
mkdir -p src/github.com/operator-framework
cd src/github.com/operator-framework
git clone https://github.com/operator-framework/operator-sdk.git -b v0.6.0
# nix-shell -p dep dep2nix go ... (if needed)
go get -d -v ./...
dep ensure -v
dep2nix save
# grab deps.nix and write derivation
# clean up temp $GOPATH
4

In case it’s helpful, I got far enough to have a callable binary with this content based on similar techniques:

5

Thanks @shanesveller! I’ll try both… BTW, this thing of the having to reconstruct this $GOPATH/… with the sources seems a bit silly to me, but as I said, I’m a totally newbie about Go.

6

@shanesveller great, thanks very much!

But, can I ask you why you uses a package path like this?

… maybe you just copied the recipe for another package…

7

There are some shortcuts I could used, particularly around go get -d -v github.com/package/name but I don’t find their behavior transparent enough and if I already know I need something else (dep in this case) to grab the sibling packages, and I won’t be compiling it at the time, I don’t bother.

Nailed it. Sloppy copy-paste on my part, I will update the gist.

8

@shanesveller I’ve tried your recipe, but I’m still getting the error, i don’t know what’s happening, this is my log https://termbin.com/ihnb … I’m going crazy