git/buildGoModule + private repositories

Help
1

Hello,

I have a Nix package I need to deploy for one of my company’s servers, however our package (in Go, using buildGoModule) depends on some private Git packages. We can’t fetch them, as git would try and use https://gitlab.com, while it should’ve used SSH. If it’s worth mentioning, we’re using morph to deploy.

Question: Can I override git in buildGoModule to replace HTTPS to SSH, similar to this command: git config --global url.ssh://git@github.com/.insteadOf https://github.com/?

Current default.nix:

buildGoModule rec {
	name = "${pname}-${version}";
	version = "0.4.0";
	goPackagePath = "gitlab.com/${org}/${pname}";

	GOPRIVATE = "*";

	src = fetchGit {
		url = "git@gitlab.com:${org}/${pname}.git";
		ref = "v${version}";
	};

	modSha256 = lib.fakeSha256;
	subPackages = [ "." ];

	meta = {
		description = "Description";
		homepage = "https://${goPackagePath}";
	};
}

Build output:

go: gitlab.com/company/dependency@v0.0.0-20191204154344-27281d3459d0: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /build/go/pkg/mod/cache/vcs/bc548c19f0ecc33e875e3bb0d03833e7e80259b75ceb8b561992597e496ac6f: exit status 128:
        fatal: could not read Username for 'https://gitlab.com': terminal prompts disabled
builder for '/nix/store/5mgl69byqq4544wy1anwr7byy1adwddf-gopackage-0.4.0-go-modules.drv' failed with exit code 1
2 Likes
2

There probably isn’t an easy way to do this. Looking at the source for buildGoModule, it seems to be using a fixed-output derivation that actually invokes the go mod tool under-the-hood. You could probably override it (there’s a buildGoModule.overrideModAttrs) to create a local git configuration that uses a different host, but then you wouldn’t have your credentials from outside of the build available inside of the derivation. Putting the credentials (e.g. an SSH key) into the derivation would be possible but then they’re world-readable in your /nix/store.

Personally I use my own Nix build system for Go (buildGo.nix) where this would be possible because external libraries are just a normal derivation with a srcs attribute, see examples in my //third_party/gopkgs/ tree. This is a very different style of building Go packages though so it might not be what you’re looking for.

3

How would I “create a local git configuration that uses a different host” and put the credentials inside the derivation?

4

Something like this (untested, I don’t have a project using Go modules around):

buildGoModule {
  # ... other fields as prior, but add this:
  overrideModAttrs = _: {
    preBuild = ''
      git config --global url.ssh://git@github.com/.insteadOf https://github.com/?
    '';
  };
}
1 Like
5

Got this code:

	overrideModAttrs = (old: {
		buildInputs = [ openssh ];

		preBuild = ''
			export HOME="$PWD"
			export GIT_SSH_COMMAND="ssh -i ${key} -o StrictHostKeyChecking=no"

			git config --global \
				url.ssh://git@gitlab.com/.insteadOf \
				https://gitlab.com/
			git config --global \
				url.ssh://git@github.com/.insteadOf \
				https://github.com/
		'';
	});

	GOPRIVATE = "*";

Seems fine until this error:

unpacking sources
unpacking source archive /nix/store/gsf9vdqaa5az4gi5fsxdfnzz1vymcbhc-source
source root is source
patching sources
configuring
building
Building subPackage ./.
unpacking sources
unpacking source archive /nix/store/kpj7s3lckd8n2fg5fy62v4qpljk294ab-source
building '/nix/store/gv9j36bmm2blv11fgd557l46aygc4h2i-activate-root.drv'...
go: github.com/bwmarrin/snowflake@v0.3.1-0.20190412223032-c09e69ae5993: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /build/go/pkg/mod/cache/vcs/86d47e47729325213189950c1ce227fe9058a8dff4bfc58f763116e7446b9213: exit status 128:
        fatal: unable to access 'https://github.com/bwmarrin/snowflake/': Could not resolve host: github.com
source root is source
builder for '/nix/store/z6sym6db8bzqkg4zqi6j7jvy22m8a9sv-company-0.0.2.drv' failed with exit code 1
6

For anyone else trying to solve private repos with modules:

go mod vendor to set up a private repo followed by a GOFLAGS="-mod=vendor" in the derivation works well. This only works if you have control over the repo in question.

{ buildGoModule , nix-gitignore }:

buildGoModule {
  pname = "some-program";
  version = "0.0.1";
  src = nix-gitignore.gitignoreSource [] ./.;
  goPackagePath = "gitlab.com/some/repo";
  goDeps = ./deps.nix;
  modSha256 = "sha256-pQpattmS2VmO3ZLQUFn66bz1GSmB4IvYhTTCFn6SUmo=";
  subPackages = ["."];
  GOFLAGS="-mod=vendor";

  # Skip the "go mod download" step
  overrideModAttrs = (_: {
        buildPhase = ":";
  });
}
7

A hack, but works to bring in SSH access and configuring go/git to fetch private dependencies. Is there a better way?

package = with pkgs; buildGoModule {
    pname = "project";
    version = "local";
    src = nix-gitignore.gitignoreSource [] ./.;
    goPackagePath = "path/to/project";
    subPackages = [ "cmd/thing" ];
~   vendorSha256 = "sha256-EtmOMdxN1/bI1ySaQNsy2SBFIu5htDxzWHPjzGb5ynQ=";
    overrideModAttrs = old: {
      preBuild = ''
        export HOME=$(pwd)
        cat <<EOF > $HOME/.gitconfig
~       [url "git@gitlab.company.com:"]
~           insteadOf = "https://gitlab.company.com/"
        EOF
~       export GOPRIVATE="company.com"
        export GIT_SSH_COMMAND="${pkgs.openssh}/bin/ssh -o StrictHostKeyChecking=no"
        '';
      impureEnvVars = lib.fetchers.proxyImpureEnvVars ++ [
        "GIT_PROXY_COMMAND" "SOCKS_SERVER" "SSH_AUTH_SOCK"
      ];
    };
    buildInputs = dependencies;
    nativeBuildInputs = [
      pkgconfig
    ];
  };
8

I worked around this in GitHub by doing:

    - run: |
        eval $(ssh-agent -s)
        ssh-add - <<< "${MACHINE_USER_KEY}"
        nix build -L .#image
      env:
        MACHINE_USER_KEY: ${{secrets.MACHINE_USER_PAT }}

This allows the cloning of private git repos that use the git transport.

9

Asummetric’s soltuion seems to assume using a flake (nix build ...). The ssh-agent socket file isn’t present in the build sandbox so it isn’t clear how tomberek got things going.

10

I’m also facing this issue. I can pull the original tarball down fine but can’t pull the go modules it depends on. Has anyone gotten this to work?