Trouble packaging sqlc

eike · March 19, 2022, 7:20pm

I’m still quite new to NixOS but enjoying it. Packaging my first go applications has worked fine so far but with sqlc I’m a bit stuck. I found out some build dependencies (e.g. libpq_queryand xxHash) and added them. This resolved previous errors but now I’m stuck with the following error:

# github.com/pganalyze/pg_query_go/v2/parser
In file included from pg_query.c:2:
pg_query_internal.h:4:10: fatal error: postgres.h: No such file or directory
    4 | #include "postgres.h"
      |          ^~~~~~~~~~~~
compilation terminated.

I thought adding postgresql to the buildInputs should fix this but it does not. What am I missing?

{ lib, buildGoModule, fetchgit, makeWrapper, postgresql, libpg_query, xxHash}:

buildGoModule rec {
  pname = "";
  version = "v1.12.0";

  nativeBuildInputs = [ makeWrapper ];

  buildInputs = [libpg_query xxHash postgresql];

 src = fetchgit {
    url    = "https://github.com/kyleconroy/sqlc";
    rev    = "refs/tags/${version}";
    sha256 = "sha256-YlOkjqkhN+4hL1+KJ0TuqcbQXJad/bHZclgpgFPr4to=";
  };

  vendorSha256 = "sha256-U2tORgGiFcODCijfq/ZQ1kWk2UpCz0oBn4oVGFOs8Lk=";

  doCheck = false;

  subPackages = [ "cmd/sqlc" ];

  meta = with lib; {
    description = "sqlc generates type-safe code from SQL.";
    homepage    = "https://sqlc.dev/";
    license     = licenses.mit;
    platforms   = platforms.unix;
  };
}

TLATER · March 19, 2022, 7:57pm

I don’t have an answer, but that "postgres.h" refers to a file in the package you’re building: pg_query_go/postgres.h at 45ef16dd073bb385e54d1c98ae9cef265eb53f2f · pganalyze/pg_query_go · GitHub

<postgres.h> in turn would refer to an external one, but admittedly that depends on how broken the C project is…

Edit: Ah, it looks like the -Iinclude from the cgo pragma in parser.go is being ignored, or the include directory simply isn’t fetched with the rest of the repo. I don’t know enough about how nix does go builds to help debug that.

eike · March 19, 2022, 8:07pm

Thanks that already that brings me closer. According to GitHub - pganalyze/pg_query_go: Go library to parse and normalize SQL queries using the PostgreSQL query parser

When integrating this library using Go modules, and using a vendor/ directory, you will need to explicitly copy over some of the C build files, since Go does not copy files in subfolders without .go files whilst vendoring.

The best way to do so is to use modvendor, and vendor your modules like this:

But I have no clue how to incorporate into the build process.

Also found this: buildGoModule: vendoring doesn't copy C sources · Issue #89128 · NixOS/nixpkgs · GitHub but I don’t really get if it has been fixed by now.

TLATER · March 20, 2022, 2:10am

Looks to me like it was closed because go packages are broken as intended (by the go devs).

The suggested workaround is to create the go vendor directory manually (well, in a bespoke script that just downloads all the dependencies)… Ugly, but admittedly upstream’s problem. This module apparently shouldn’t be written the way it is.

eike · March 20, 2022, 11:16am

Thanks for looking into it. That what I understood as well. That would mean keeping a local copy of the whole repo and a vendor folder as I cannot download additional resources during the build process, right? Seems a bit overkill to me.

I also found the proxyVendor argument for the buildGoModule-function: nixpkgs/default.nix at 292daa1285c888e110af4d03ce544b9d2963b9aa · NixOS/nixpkgs · GitHub which seems applicable:

This is useful if your code depends on c code and go mod tidy does not include the needed sources to build

But then the build fails early as no dependencies seem to be downloaded at all.

TLATER · March 20, 2022, 12:35pm

You can probably do something like this:

{ lib, buildGoModule, fetchgit, fetchFromGithub, makeWrapper, postgresql, libpg_query, unzip, xxHash}:

let
  pg_query_go = fetchFromGithub {
    owner = "pganalyze";
    repo = "pg_query_go";
    rev = "v2.1.0";
    sha256 = ""; # Nix will tell you the correct sha
  };

in buildGoModule rec {
  pname = "";
  version = "v1.12.0";

  nativeBuildInputs = [ makeWrapper unzip ];

  buildInputs = [libpg_query xxHash postgresql];

 src = fetchgit {
    url    = "https://github.com/kyleconroy/sqlc";
    rev    = "refs/tags/${version}";
    sha256 = "sha256-YlOkjqkhN+4hL1+KJ0TuqcbQXJad/bHZclgpgFPr4to=";
  };

  postPatch = ''
    unzip '${pg_query_go}' 'pg_query_go/parser/*.c' 'pg_query_go/parser/*.h' -d vendor/github.com/pganalyze/pg_query_go/parser
  '';

  vendorSha256 = "sha256-U2tORgGiFcODCijfq/ZQ1kWk2UpCz0oBn4oVGFOs8Lk="; # This may need updating

  doCheck = false;

  subPackages = [ "cmd/sqlc" ];

  meta = with lib; {
    description = "sqlc generates type-safe code from SQL.";
    homepage    = "https://sqlc.dev/";
    license     = licenses.mit;
    platforms   = platforms.unix;
  };
}

TLATER · March 20, 2022, 12:42pm

Or apparently you could use proxyVendor or runVend on stable. Also documented in the nixpkgs manual: NixOS - Nixpkgs 21.11 manual

eike · March 20, 2022, 1:04pm

I tried both. For proxyVendori get:

go: github.com/antlr/antlr4/runtime/Go/antlr@v0.0.0-20211208212222-82c441726976: reading file:///nix/store/jqr2rqb2mwk0dln2b5ii10jplcy7m5fc–v1.12.0-go-modules/github.com/antlr/antlr4/runtime/%21go/antlr/@v/v0.0.0-20211208212222-82c441726976.mod: open /nix/store/jqr2rqb2mwk0dln2b5ii10jplcy7m5fc–v1.12.0-go-modules/github.com/antlr/antlr4/runtime/!go/antlr/@v/v0.0.0-20211208212222-82c441726976.mod: no such file or directory

With runVend the header file is still missing.

TLATER · March 20, 2022, 1:07pm

You can check what’s going wrong by manually ls-ing around the store. My other approach should work regardless, plus or minus some hacking.

eike · March 20, 2022, 3:51pm

That is what I’m currently doing (both trying to your great idea to work an ls-ing through the store. It seems that the downloaded source is not a zip after all. Right now I’m having trouble patching the vendor folder as it seems that go stores the source in a zip file and also hashes it. I will try some more though.

eike · March 21, 2022, 7:20pm

@TLATER, thanks again for your help and your patience. In the end it took my a bit more time than anticipated but the following is working.

{ 
  lib, 
  buildGoModule, 
  fetchFromGitHub, 
  postgresql, 
  libpg_query, 
  xxHash, 
}:

let
  pg_query_go = fetchFromGitHub {
    owner = "pganalyze";
    repo = "pg_query_go";
    rev = "v2.1.0";
    sha256 = "sha256-nT1QHT5th1pwTpLakSyfPQQbE+RnOvZyvy1Mqk57RrQ="; # Nix will tell you the correct sha
  };

in buildGoModule rec {
  pname = "sqlc";
  version = "v1.12.0";

  buildInputs = [libpg_query xxHash postgresql];

 src = fetchFromGitHub {
    owner  = "kyleconroy";
    repo   = "sqlc";
    rev    = "${version}";
    sha256 = "sha256-YlOkjqkhN+4hL1+KJ0TuqcbQXJad/bHZclgpgFPr4to=";
  };

  postConfigure = ''
    go mod edit -replace github.com/pganalyze/pg_query_go/v2=${pg_query_go}
  '';

  proxyVendor = true;

  vendorSha256 = "sha256-oImfSQZNBgq020V6QN222a4sqJ0OFIfWkLhyBqu3KOM=";

  doCheck = false;

  subPackages = [ "cmd/sqlc" ];

  meta = with lib; {
    description = "sqlc generates type-safe code from SQL.";
    homepage    = "https://sqlc.dev/";
    license     = licenses.mit;
    platforms   = platforms.unix;
  };
}

TLATER · March 22, 2022, 1:16am

No problem whatsoever - I learned a lot about go in nix too, hopefully I can help the next person before they get too frustrated

You should consider upstreaming that package, it looks very nice!

nanzhong · May 12, 2022, 2:17pm

I stumbled here with the exact same issue (and it seems like trying to find examples of the use case for proxyVendor also ends up here), and wanted to share a pitfall that left me scratching my head.

proxyVendor should eliminate the need to go mode -replace the dependency to point to a local path. It does exactly what it’s documented to do (runs go mod download which populates the module cache with all the files needed for go proxy to work).

The gotcha that I kept running into was the same error that @eike saw around the zip files not being found. This non-obvious thing here is that vendorSha256 changes when you switch from proxyVendor = false (default) to proxyVendor = true (because of the additional files that make up the module cache), which means that if you don’t update the vendorSha256 it will be referencing the go module dependencies retrieved via go mod vendor instead of the go mod download, which do not conform to what go proxy expects.

The result is that the following snippet is all that needed:

          sqlc = pkgs.buildGoModule {
            name = "sqlc";
            src = sqlc; # replacing this which however you are getting the source e.g. fetchFromGitHub, via flake inputs, etc.
            subPackages = [ "cmd/sqlc" ];
            doCheck = false;
            vendorSha256 = "sha256-LX8C7098P940wdg8yig6h6azAsxdai+vEVhjdILIoMQ=";
            proxyVendor = true;
            buildInputs = [
              pkgs.xxHash
              pkgs.libpg_query
              pkgs.postgresql_14
            ];
          };

eike · May 12, 2022, 7:07pm

@nanzhong you are absolutely right! I had that feeling as well when I saw the solution posted here: buildGoModule: vendoring doesn't copy C sources · Issue #89128 · NixOS/nixpkgs · GitHub. I never checked though as I had a working configuration. It really is a “beginner mistake” that is really hard to catch unless you know a bit how nix (and it’s caching) works.

adisbladis · May 13, 2022, 8:02am

I recently started using sqlc for a personal project of mine and decided to upstream my package: sqlc: init at 1.13.0 by adisbladis · Pull Request #172816 · NixOS/nixpkgs · GitHub.