Help packaging Go binaries

Help
1

I’m a new user of Nix and have been trying to package a small Go CLI app:

{
pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/f151bae7ab9bfdf6f125a2a03aad0ba6bf0589dc.tar.gz") {}
}:

pkgs.buildGoModule rec {
    pname = "boots";
    version = "0.1.1";

    vendorSha256 = null;

    src = ./cli/.;
    subPackages = [ "./cmd/boots" ];
}

The above works, however, before executing nix-build I have to manually run go mod vendor in the source directory. This is pretty jarring to me since I don’t vendor anything in the repository (nor do I desire to) and, as far as I can tell, I have to figure out how to ensure this is run before attempting to create a shell with my tool included, otherwise, the package won’t build.

Has anyone done packaging with Go and figured out a way around not having to run go mod vendor to get the package to build? Is there a way to automate this so that I don’t have to think about it?

2

I’m not super familiar with Go binaries, but it looks like this is intended behavior: https://github.com/NixOS/nixpkgs/blob/78e0de4c9b056d2a6168bac2ef974863e7ac5feb/doc/languages-frameworks/go.section.md

I think it has something to do with how vendorSha256 is used to checksum the dependencies. If you want your dependencies to update dynamically (not always a good idea, but sometimes is), you could probably have go mod vendor automatically run in a function and return a modified src.

3

Provide a vendorSha256, then nix will fetch the deps for you as described in the go.sum, and check if their hash matches the expectation.

Start with lib.fakeSha256 and then replace it with the one from the error.

When you update your go.mod or go.sum you need to redo this procedure and update the hash as nix will happily reuse the old download it did as it only uses the vendorSha256 to identify the prebuilt vendor folder.

Not updating the hash properly will result in weird errors during nix build.

2 Likes
4

Thank you, that helps clear things up a bit. Is there no native way to automate this in Nix? I’m fine developing a git hook to perform this check automatically - but it just seemed a bit excessive at first. My problem is that this repository is collaborated on and most people don’t know Nix and so I was trying to make the solution as unobtrusive as possible on the development experience.

5

Best way was to build in a pre-commit (or pre-push).

You can assume that not fixing the hash will always result in a build error about missing packages. You can then train your co workers to fix the hash and try again.

If you do not want them to edit the nix files, you can use fromJson or fromToml to read the hash from auxiliary files.

6

The reason it’s not automated is that all languages “lock” their dependencies in different ways, so a generic implementation falls short and downstream implementations for each language quickly become unmaintainable, as downstream plays catch-up with upstream language changes while having much fewer developers.

It’d be nice if there were cross-ecosystem standards for software, but that’s not the world we live in.

You might be interested in these issues, generally when looking for a solution to resolving checksums automatically before build time you’re looking for a “prefetch” option:

GitHub - berberman/nvfetcher: Generate nix sources expr for the latest version of packages also exists, but it won’t seamlessly integrate with go.

Generally though, nix assumes that dependency changes are uncommon, and that updating a checksum whenever you do that should be the minimum amount of effort spent checking that you’re not throwing malware or broken code at users. Most Linux distros try to follow that ideal, and it sadly clashes a bit with the dependency model that npm popularized.

1 Like