Rethink GOPROXY

Development
1

after seeing a few posts detailing the privacy and correctness issues with GOPROXY, specifically proxy.golang.org, I thought I would start a discussion here if there was appetite in making a change to hardcode GOPROXY=direct within nixpkgs, nixos, or some other subset of our codebases?

2

The go proxy is off by default for package builds https://github.com/NixOS/nixpkgs/blob/dfc8fe37709f969e930351f3a4b779ed840ac86c/pkgs/build-support/go/module.nix#L176

3

while GOPROXY=off may be set during the literal go build phase, short of cases where we use the upstream vendor directory, there are a few places where nix(os) uses proxy.golang.org:

  • nix fetches modules during buildGoModule via proxy.golang.org
  • nix builds a go binary that does not set GOPROXY, thus falling back to GOPROXY=proxy.golang.org
  • nixos does not set it as a session default (not that it necessarily should)
  • home-manager’s go program settings leaves GOPROXY unset by default

I am happy to consider all of these cases breaking and approach them as such, and this would be a divergence from upstream go, but I still think it is worth pursuing.

the easiest option to resolve this seems to be to patch the default value of GOPROXY within the go toolchain directly, so we do not have to play wack-a-mole with all the environments. it also seems reasonable to do this with a specific go release on the unstable branch. we can also leave an mirror for those wanting the upstream behaviour: pkgs.go.withProxy.

4

As far as I know only when proxyVendor = true; is set.

I am not sure if we should mess with that. Go vendoring is already brittle enough and messy and setting this by default is unexpected.

if we decide to do such things then we should do it the other way around and add an extra package that removes it.

5

both go mod vendor, proxyVendor = false;, and go mod download, proxyVendor = true; should reach out to proxy.golang.org. let me know what evidence you would be looking for here.

I agree, this wording was supposed to reflect what is currently happening, not what we should do. currently pkgs.go has a default of GOPROXY=proxy.golang.org, as stated below in that post, I think we should patch the go toolchain to change the value of the default, not play around with environment variables.

yeah, that would be a good way to start. I am still keen to try and solve the privacy/correctness issue in the long run, and make it default, but that may be unpopular or infeasible. that said, I think there is a strong reason (see correctness) to do this in buildGoModule as the default as soon as feasible.

6

No because turning off the go proxy has real world impact and can lead to errors presented very poorly by go. Also I think git cloning is way slower.

7

if the go toolchain is presenting errors poorly that is a bug that should be filed upstream. git fetch is still a core feature to the toolchain that is actively supported. though I will concede that the ux is important, and may push us to delay any efforts to make this default.

there is plenty of stuff about nix that falls into the “way slower camp”, so this seems moot, but in that case we can provide an escape hatch. though this seems like a great time to just cache modules and intermediate artefacts, which will be faster than repeatedly hitting the mod proxy.

8

Trust me, there are plenty of them and they don’t care and things didn’t improve over the last years.

Also this would require us to use extra FODs at which point it is just gomod2nix.

9

yeah, maybe I am missing some context, but considering the amount of module reuse in the go ecosystem, it seems better to have one fod per module, as opposed to per package, as the current implementation shares nothing?

10

I just stumbled on this. I have a self-managed gitlab with private go modules that I use additionally with replace in go.mod. This also relies on a private GOPROXY because go doesn’t handle ssh://git@... but only straight https.

Maybe for the above topic (regardless of the aguably valid reasons stated above), one could start to actually offer a way to define GO* env vars in the build env that is run by e.g. buildGoModule?

What would be the most direct way to do this?

EDIT:
I played a bit with it (and read some of this issue as well),
and managed to make it work like this:

buildGoModule rec {
...
  src = fetchGit {
    url = "ssh://git@my.gitdomain.com:/mygoprog.git";
    ref = "main";
  };

  vendorSha256 = "sha256-eemA0VQKgsrr0r5Y3NjM+8eDvLrpVUNNFHszhtjG4qw=";
  # proxyVendor = true;
  # vendorHash = null;

  buildFlags = "-mod=mod";

  preBuild = ''
    GONOSUMDB="my.gitdomain.com/*,replaceddomain.com/*"
    GOPROXY="http://192.168.1.114:3000,direct"
    GOSUMDB="sum.golang.org"
    export GONOSUMDB GOPROXY GOSUMDB
  '';
...
}

This additionally requires buix-build --options sandbox false because the build isn’t allowed access to the go proxy otherwise. So hardly an acceptable idiomatic UX. But this must be solvable because the “vendor fetch” step also connects, so the options must probably rather be set there?

11

You want something like:

  overrideModAttrs = (oldAttrs: {
    nativeBuildInputs = oldAttrs.nativeBuildInputs or [ ] ++ [
      openssh
    ];

    impureEnvVars = oldAttrs.impureEnvVars or [ ] ++ [
      "NIX_GITHUB_PRIVATE_USERNAME" "NIX_GITHUB_PRIVATE_PASSWORD"
    ];

    preBuild = /* bash */ ''
      export HOME=$(mktemp -d)
      export NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

      mkdir -p ~/.ssh/
      # ssh-keyscan github.corp
      cat > ~/.ssh/known_hosts <<EOF
      github.corp ssh-rsa ...
      github.corp ecdsa-sha2-nistp256 ...
      EOF

      cat > ~/.netrc <<EOF
      machine github.corp
        login $NIX_GITHUB_PRIVATE_USERNAME
        password $NIX_GITHUB_PRIVATE_PASSWORD
      EOF
    '';

    preferLocalBuild = true;
  });

  GOPRIVATE = "github.corp";

and add the envs to nix-daemons environment. As a password you use a token.

2 Likes