Help packaging go project

Help
1

Hello, I’m trying to package a go project, and I know next-to-nothing about go. I’m not sure if this is a go package or a go module. I’ve made a start on it and have included the files I’m using below. This project is for downloading models from hugging face and is on github here: GitHub - bodaay/HuggingFaceModelDownloader: Simple go utility to download HuggingFace Models and Datasets

The error I get when doing nix build . is as follows. It seems to be unable to find a package ‘hfdownloader’, which I believe is part of the same repo. I’ve tried messing with the goPackagePath, but I got stuck. Any tips appreciated!

error: builder for '/nix/store/4vbx8b88j9vm524az678240yzl0khp1c-HuggingFaceModelDownloader.drv' failed with exit code 1;
       last 10 log lines:
       > unpacking source archive /nix/store/cw1r1n8jb6wqqw975xrm2x958drh1f97-source
       > unpacking source archive /nix/store/cw1r1n8jb6wqqw975xrm2x958drh1f97-source
       > unpacking source archive /nix/store/cw1r1n8jb6wqqw975xrm2x958drh1f97-source
       > unpacking source archive /nix/store/cw1r1n8jb6wqqw975xrm2x958drh1f97-source
       > unpacking source archive /nix/store/7w0j26wkzxhbiigqf6a0yzf8ypd3svz4-source
       > building
       > Building subPackage github.com/booday/HuggingFaceModelDownloader/hfdownloader
       > go/src/github.com/booday/HuggingFaceModelDownloader/hfdownloader/main.go:15:2: cannot find package "github.com/bodaay/HuggingFaceModelDownloader/hfdownloader" in any of:
       >        /nix/store/y7abhs9glxfcg7lgcdc8i4ml5wg5ly92-go-1.21.4/share/go/src/github.com/bodaay/HuggingFaceModelDownloader/hfdownloader (from $GOROOT)
       >        /build/go/src/github.com/bodaay/HuggingFaceModelDownloader/hfdownloader (from $GOPATH)
       For full logs, run 'nix log /nix/store/4vbx8b88j9vm524az678240yzl0khp1c-HuggingFaceModelDownloader.drv'.

– flake.nix

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/23.11";
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, flake-utils }:
    let
      system = flake-utils.lib.system.x86_64-linux;
      pkgs = import nixpkgs { inherit system; };
      hfdownloader = pkgs.callPackage ./hfdownloader.nix { };
    in { packages.${system}.default = hfdownloader; };
}

– hfdownloader.nix

{ fetchFromGitHub, buildGoPackage }:
buildGoPackage {
  name = "HuggingFaceModelDownloader";
  src = fetchFromGitHub {
    owner = "bodaay";
    repo = "HuggingFaceModelDownloader";
    sha256 = "sha256-cDrFQTo9E112UFtam40kQ+lxPStSg4Edx1fjK2WyWHw=";
    rev = "2f38356a6d6ce9c7731eb48bee35d795a4f1f67c";
  };
  goPackagePath = "github.com/booday/HuggingFaceModelDownloader/hfdownloader";
  goDeps = ./deps.nix;
}

– deps.nix

[
  {
    goPackagePath = "github.com/spf13/cobra";
    fetch = {
      type = "FromGitHub";
      owner = "spf13";
      repo = "cobra";
      rev = "v1.7.0";
      sha256 = "sha256-oAE+fEaRfZPE541IPWE0GMeBBYgH2DMhtZNxzp7DFlY=";
    };
  }
  {
    goPackagePath = "github.com/joho/godotenv";
    fetch = {
      type = "FromGitHub";
      owner = "joho";
      repo = "godotenv";
      rev = "v1.5.1";
      sha256 = "sha256-kA0osKfsc6Kp+nuGTRJyXZZlJt1D/kuEazKMWYCWcQ8=";
    };
  }
  {
    goPackagePath = "github.com/fatih/color";
    fetch = {
      type = "FromGitHub";
      owner = "fatih";
      repo = "color";
      rev = "v1.16.0";
      sha256 = "sha256-6sX3ZvuVi8/3DAU1+8zN9IUpUdtT2JqwxSGldXmywzw=";
    };
  }
  {
    goPackagePath = "github.com/inconshreveable/mousetrap";
    fetch = {
      type = "FromGitHub";
      owner = "inconshreveable";
      repo = "mousetrap";
      rev = "v1.1.0";
      sha256 = "sha256-6sX3ZvuVi8/3DAU1+8zN9IUpUdtT2JqwxSGldXmywzw=";
    };
  }
  {
    goPackagePath = "github.com/mattn/go-colorable";
    fetch = {
      type = "FromGitHub";
      owner = "mattn";
      repo = "go-colorable";
      rev = "v0.1.13";
      sha256 = "sha256-6sX3ZvuVi8/3DAU1+8zN9IUpUdtT2JqwxSGldXmywzw=";
    };
  }
  {
    goPackagePath = "github.com/mattn/go-isatty";
    fetch = {
      type = "FromGitHub";
      owner = "mattn";
      repo = "go-isatty";
      rev = "v0.0.20";
      sha256 = "sha256-6sX3ZvuVi8/3DAU1+8zN9IUpUdtT2JqwxSGldXmywzw=";
    };
  }
  {
    goPackagePath = "github.com/spf13/pflag";
    fetch = {
      type = "FromGitHub";
      owner = "spf13";
      repo = "pflag";
      rev = "v1.0.5";
      sha256 = "sha256-YTRLqLRZJHBh2m1dA99/EepY3DAi/rks1feB9ixT9T4=";
    };
  }
]
2

I don’t really know much about packaging GO projects with nix, but I got curious and I wanted to try it out. The tool I used to make this work is called gomod2nix.

First, I followed the quickstart guide and I generated the necessary template:

$ nix flake init -t github:nix-community/gomod2nix#app

Then, I entered the shell and I converted the modules file:

$ nix-shell
$ gomod2nix generate

Finally, I replaced the template in default.nix with the package details and I built the project:

{ pkgs ? (
    let
      inherit (builtins) fetchTree fromJSON readFile;
      inherit ((fromJSON (readFile ./flake.lock)).nodes) nixpkgs gomod2nix;
    in
    import (fetchTree nixpkgs.locked) {
      overlays = [
        (import "${fetchTree gomod2nix.locked}/overlay.nix")
      ];
    }
  )
, buildGoApplication ? pkgs.buildGoApplication
, fetchFromGitHub ? pkgs.fetchFromGitHub
}:

buildGoApplication {
  pname = "HuggingFaceModelDownloader ";
  version = "1.3.4";
  pwd = ./.;
  src = fetchFromGitHub {
    owner = "bodaay";
    repo = "HuggingFaceModelDownloader";
    sha256 = "sha256-cDrFQTo9E112UFtam40kQ+lxPStSg4Edx1fjK2WyWHw=";
    rev = "2f38356a6d6ce9c7731eb48bee35d795a4f1f67c";
  };
  modules = ./gomod2nix.toml;
}

$ nix build .
$ ./result/bin/HuggingFaceModelDownloader
A Simple HuggingFace Models Downloader Utility
Version: 1.3.4
...

There might be a better way to do this, but I think this is a good start.

3

So, with a little bit of help from the Nixpkgs Manual, I got this working as a nixpkgs package.

First, we clone the nixpkgs repo (or your fork, if you have one):

$ git clone https://github.com/NixOS/nixpkgs --depth=1
$ cd nixpkgs

Next, we put the package under a category and enter its details in pkgs/tools/misc/huggingfacemodeldownloader/default.nix:

{ buildGoModule, fetchFromGitHub, lib }:

buildGoModule rec {
  pname = "huggingfacemodeldownloader";
  version = "1.3.4";

  src = fetchFromGitHub {
    owner = "bodaay";
    repo = "HuggingFaceModelDownloader";
    rev = "2f38356a6d6ce9c7731eb48bee35d795a4f1f67c";
    sha256 = "sha256-cDrFQTo9E112UFtam40kQ+lxPStSg4Edx1fjK2WyWHw=";
  };

  vendorHash = "sha256-0tAJEPJQJTUYoV0IU2YYmSV60189rDRdwoxQsewkMEU=";

  meta = with lib; {
    description = "Simple go utility to download HuggingFace Models and Datasets ";
    homepage = "https://github.com/bodaay/HuggingFaceModelDownloader";
    license = licenses.asl20;
    maintainers = with maintainers; [ ];
  };
}

Inside pkgs/top-level/all-packages.nix, we add:

  huggingfacemodeldownloader = callPackage ../tools/misc/huggingfacemodeldownloader { };

Then, from inside the nixpkgs directory, we execute:

$ nix-build . -A huggingfacemodeldownloader
$ ./result/bin/HuggingFaceModelDownloader 
A Simple HuggingFace Models Downloader Utility
Version: 1.3.4
...

Still inside the nixpkgs directory, you can also install/uninstall it to your system like this:

$ nix-env -f . -iA huggingfacemodeldownloader
$ nix-env --uninstall huggingfacemodeldownloader

Finally, if you want this to be integrated into nixpkgs, you can open a pull request here.

Hope this helped :snowflake:

4

Thank you! I followed your lead from the second post and incorporated it into a flake like so. I believe the main difference from what I originally posted and your working solution is the use of buildGoPackage in the former and buildGoModule in the latter, though I could have sworn that I tried buildGoModule as well.

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/23.11";
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
        hfdownloader = pkgs.callPackage ./. { };
      in { packages.default = hfdownloader; });
}

# default.nix
{ buildGoModule, fetchFromGitHub, lib }:

buildGoModule {
  pname = "huggingfacemodeldownloader";
  version = "1.3.4";

  src = fetchFromGitHub {
    owner = "bodaay";
    repo = "HuggingFaceModelDownloader";
    rev = "2f38356a6d6ce9c7731eb48bee35d795a4f1f67c";
    sha256 = "sha256-cDrFQTo9E112UFtam40kQ+lxPStSg4Edx1fjK2WyWHw=";
  };

  vendorHash = "sha256-0tAJEPJQJTUYoV0IU2YYmSV60189rDRdwoxQsewkMEU=";

  meta = with lib; {
    description =
      "Simple go utility to download HuggingFace Models and Datasets ";
    homepage = "https://github.com/bodaay/HuggingFaceModelDownloader";
    license = licenses.asl20;
  };
}
1 Like
5

Glad this helped. For future reference, buildGoModule should be used because it automatically hashes dependencies, so we don’t have to insert them seperately. Also, buildGoPackage is now marked as legacy.