How to build a go project and include it into a nixos install

[uskeutia]

Hi,

I’m totally new to NixOS and still figuring out if I should switch to it or not.

So, I startet to build a configuration which at least works and installs most of the stuff I need: GitHub - TLINDEN/nixos-config: My personal nixos config. This just installs packages, I’m not interested to also install configs at this point. I’ll just checkout my dotfiles repo to have them.

However, I am using a couple of go binaries which are not available as a nix package (or are outdated). So, I want to build by own packages for those tools and integrate them into my nixos configuration.

I started to create a new package with nix flake init -t templates#go-hello. I renamed the package, added a github source, fixed the sha256:

{
  description = "A simple Go package";
  inputs.nixpkgs.url = "nixpkgs/nixos-21.11";

  outputs = { self, nixpkgs }:
    let
      lastModifiedDate = self.lastModifiedDate or self.lastModified or "19700101";
      version = builtins.substring 0 8 lastModifiedDate;
      supportedSystems = [ "x86_64-linux" ];
      forAllSystems = nixpkgs.lib.genAttrs supportedSystems;
      nixpkgsFor = forAllSystems (system: import nixpkgs { inherit system; });

    in
    {
      packages = forAllSystems (system:
        let
          pkgs = nixpkgsFor.${system};
        in
        {
          rpnc = pkgs.buildGoModule {
            pname = "rpnc";
            inherit version;
            src = pkgs.fetchFromGitHub {
              owner = "tlinden";
              repo = "rpnc";
              rev = "v2.1.0";
              sha256 = "sha256-CojP1a19b2zKfUMp+wN7FFs+SzSoc8sYqKvXTg4RnOA=";
            };
            vendorSha256 = "sha256-pQpattmS9VmO3ZIQUFn66az8GSmB4IvYhTTCFn6SUmo=";
          };
        });

      devShells = forAllSystems (system:
        let
          pkgs = nixpkgsFor.${system};
        in
        {
          default = pkgs.mkShell {
            buildInputs = with pkgs; [ go gopls gotools go-tools ];
          };
        });

      defaultPackage = forAllSystems (system: self.packages.${system}.rpnc);
    };
}

but it’s not compiling:

% nix flake check 
warning: flake output attribute 'defaultPackage' is deprecated; use 'packages.<system>.default' instead
warning: The check omitted these incompatible systems: aarch64-darwin, aarch64-linux, x86_64-darwin
Use '--all-systems' to check all.

% nix build 
error: builder for '/nix/store/025sdqn58pxj531hgmqphr1klzfyv1g2-rpnc-20240514-go-modules.drv' failed with exit code 1;
       last 10 log lines:
       > patching sources
       > configuring
       > building
       > go: downloading github.com/chzyer/readline v1.5.1
       > go: downloading github.com/spf13/pflag v1.0.5
       > go: downloading github.com/yuin/gopher-lua v1.1.0
       > go: downloading github.com/rogpeppe/go-internal v1.11.0
       > go: downloading golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f
       > go: downloading golang.org/x/tools v0.1.12
       > internal error: failed to find embedded files of github.com/rogpeppe/go-internal/internal/os/execpath: //go:build comment without // +build comment
       For full logs, run 'nix-store -l /nix/store/025sdqn58pxj531hgmqphr1klzfyv1g2-rpnc-20240514-go-modules.drv'.
error: 1 dependencies of derivation '/nix/store/qilj13v4cv63x4hpaiw6fjzj75p74a4r-rpnc-20240514.drv' failed to build

This error message doesn’t make any sense. It occurs if you want to compile newer go code with go 1.16 or less. However, I have 1.22.2 installed on my test nix system and the module in question requires 1.20.

How can I find out, which go version the builder is using? How can I change the version? And why isn’t it just using the one specified in the modules go.mod?

And I’ve got even more questions: let’s assume I get the package build to work, what next: how can I put it into the nixos configuration linked above?

[uskeutia]

Ok, I started nix development and the go version inside is: go1.16.13. Way too old.

[kotatsuyaki]

@uskeutia

inputs.nixpkgs.url = "nixpkgs/nixos-21.11";

You are using Nixpkgs from its nixos-21.11 branch, which was released in 2021. You probably want to use the latest release branch (nixos-23.11 at the moment) or the unstable branch nixos-unstable.

[uskeutia]

Ok, now the build complains about vendoring:

nix build 
error: builder for '/nix/store/82hvy7pli62v19nw9ja9vz446b3iv9rp-rpnc-20240514.drv' failed with exit code 1;
       last 10 log lines:
       >        github.com/chzyer/readline@v1.5.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
       >        github.com/rogpeppe/go-internal@v1.11.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
       >        github.com/spf13/pflag@v1.0.5: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
       >        github.com/yuin/gopher-lua@v1.1.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
       >        golang.org/x/sys@v0.0.0-20220722155257-8c9f86f7a55f: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
       >        golang.org/x/tools@v0.1.12: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
       >
       >        To ignore the vendor directory, use -mod=readonly or -mod=mod.
       >        To sync the vendor directory, run:
       >                go mod vendor

The source has no vendor/ dir. Can I pass this flag -mod=mod somehow?

[uskeutia]

Never mind, I had to put a proper vendorHash into it, then it compiled. But now it fails during the go test phase because less is not available

[uskeutia]

I tried adding less like so (after the src definition):

buildInputs = with pkgs; [ go gopls gotools go-tools less ];

but this doesn’t help, less stays missing.

PS: I also tried nativebuildInputs with the same result.

[eljamm]

In this case, the easiest way to add the package in my opinion is by using overlays:

# /etc/nixos/configuration.nix

  nixpkgs.overlays = [
    (final: prev: {
      rpnc = pkgs.buildGoModule rec {
        pname = "rpnc";
        version = "2.1.0";
        src = pkgs.fetchFromGitHub {
          owner = "tlinden";
          repo = "rpnc";
          rev = "v${version}";
          sha256 = "sha256-CojP1a19b2zKfUMp+wN7FFs+SzSoc8sYqKvXTg4RnOA=";
        };
        vendorHash = "sha256-KO8cbkqdAkGkNrqBh3wIWaQyvf9hqrKjXWbElpQzMpg=";
        nativeCheckInputs = with pkgs; [ less ];
      };

      another-package = pkgs.buildGoModule rec {
        ...
      };
    })
  ];

Then install the package normally:

  environment.systemPackages = with pkgs; [
    rpnc
  ];

Edit: This is a cleaner way to do this:

How to build a go project and include it into a nixos install

I’d argue it’s easier (and maybe clearer) without assigning rpnc in an overlay: { config, pkgs, ... }: let rpnc = pkgs.buildGoModule ....; in { .... environment.systemPackages = with pkgs; [ rpnc ]; .... }

It’s nice to add the missing packages to nixpkgs, though. For that, you should probably begin with reading:

• Nixpkgs/Contributing - NixOS Wiki
• nixpkgs/CONTRIBUTING.md at 609d8d60c183f03dd7f7cd47c7c9409b84b1ab2e · NixOS/nixpkgs · GitHub

[eljamm]

As the Nixpkgs Reference Manual describes, you should use nativeCheckInputs, since the test tries to access less from the path:

Dependencies needed only to run tests are similarly classified between native (executed during build) and non-native (executed at runtime):

• nativeCheckInputs for test tools needed on $PATH (such as ctest) and setup hooks (for example pytestCheckHook)
• checkInputs for libraries linked into test executables (for example the qcheck OCaml package)

[uskeutia]

Yeah! It worked with nativeCheckInputs Thank you very much @eljamm!

Now where do have to put these overlays? Directly into the flake.nix file or can I put them somewhere else and include them?

[eljamm]

You can put it in your /etc/nixos/configuration.nix or another module in your configs.

[rgoulter]

I’d argue it’s easier (and maybe clearer) without assigning rpnc in an overlay:

{ config, pkgs, ... }:

let
  rpnc = pkgs.buildGoModule ....;
in
{
  ....
  environment.systemPackages = with pkgs; [
    rpnc
  ];
  ....
}

[uskeutia]

Ha!

[14.Mai 13:21:58] --- [~/.nix] ---
scip@c9: % make build 
sudo nixos-rebuild switch --flake .#system
[..]

[14.Mai 13:23:20] --- [~/.nix] ---
scip@c9: % rpn -v
This is rpn version 2.1.0

Very very nice. I suspect, I’ll install NixOS

[eljamm]

Thanks. This probably makes more sense since there isn’t anything to overlay in the first place. I’m just too used to writing overlays

[uskeutia]

Yes, works w/o overlay as well, thanks!

[uskeutia]

One more thing: is there a way to pre-compute the two sha256 hashes somehow? Can’t be the official way to wait for the error to occur and copy/paste the correct hash can it?

[uskeutia]

@eljamm

It’s nice to add the missing packages to nixpkgs, though.

I might do that, but one question: is it allowed for me as the upstream maintainer to submit a package for a tool? Some distros don’t like this.

[eljamm]

@uskeutia

is there a way to pre-compute the two sha256 hashes somehow?

I know that you can use nix-prefetch-url, but aside from that I don’t really know.

$ nix-prefetch-url --print-path https://github.com/TLINDEN/rpnc/archive/refs/tags/v2.1.0.tar.gz
06alasf9875dc8m4fvjn7wnpvwc0j6141a31bnqwi5dj2iqi4d37
/nix/store/r2d37afbxrncjbkj8yn6avcnaxj9jhnk-v2.1.0.tar.gz

@uskeutia

is it allowed for me as the upstream maintainer to submit a package for a tool?

Unless I’m wrong, I don’t think there is anything that prohibits you from doing that.