Go(lang) and Nix(OS): Go programs must be patched?

Help
1

Disclaimer: I know nothing about Go and very little about Nix

I started to use gh (installed as a Nix package) and some of its extensions are built with Go.

Some of those extensionse work on NixOS, some do not (apparently failing as a non patched program). If I compile locally an extension, it works.

I assume that this behavior depends on how the extension/program is built and, perhaps, patched.

How can I reliably figure out if any program is or not (and need to be) patched?

So far I basically fixed the issue locally compiling on install all the extensions, including the ones that I know that would work anyway, but it is not an optimal solution.

The point is that would probably be overwhelming to (find and) package all those extensions for Nix.

I did a research to figure out if all Go programs compiled on Nix(OS) must be patched, without finding anything really. My curiosities:

  • are all Go programs compiled on Nix(OS) patched (independently of needing or not a patch)?
    • that seems not to be the case, as I can run some Go programs that are not built on Nix(OS)
  • so far my guess is that some programs need to be actually patched, other do not
    • if that is true, how can I find out when it is necessary?
      • on what generally depends the need for a patch?
        • external dependencies (C modules, external libs, etc), Go libs, ā€¦?

I need some parameter to figure this outā€¦

2

The general rule of thumb is that programs will need to be patched if they depend on libraries being in /lib instead of /nix/store. Go binaries built on NixOS donā€™t need to be patched because theyā€™ll automatically be depending on libraries in /nix/store. Additionally, Go programs that donā€™t depend on libraries written in other languages wonā€™t have this issue (to my knowledge, Go is one of the only languages with this property).

1 Like
3

Thank you very much.

This helps a lot clarifying the topic. It is more or less as I imagined, but I was also absolutely not sure about.

The simpler solution (as to check if a program and its eventual dependencies** depend or not on /lib etc would introduce a crazy complexity, I suppose) would be then to always simply build the extensions on install phase, right?

** extensions are numerous, installed by the primary tool, written in several different languages, ā€¦

4

In the Go world, if you can build with CGO_ENABLED=0 and netgo, then you can likely not patch it. Basically if it is a static build, otherwise it will need patching.

ref: On Golang, ā€œstaticā€ binaries, cross-compiling and plugins | by Diogok | Medium

1 Like
5

Thank you, thatā€™s usefulā€¦

Could be a best practice proposal , unfortunately would be complicated to assure this, as extensions are an undefined number of present and future tools built by unrelated people.

6

Making Go builds static isnā€™t always possible. A common example that comes up is anything that depends on GitHub - mattn/go-sqlite3: sqlite3 driver for go using database/sql, but there are tons of cgo dependencies that makes this hard to have a general approach.

2 Likes
7

That arises the issue. There is no way that I am aware of right now to check (unless doing it manually one by one) what exactly an installed program (extension, in this case) would depends on (also, what language is built with, etc).

At this point, talking exclusively about Go programs/extensions, I guess that does not really matter if the original program is or not static: all the extensions should be locally compiled (on NixOS) at install time.

8

If you use Nix to perform the build, then all runtime dependencies are tracked and copied around with the closure or substituted as required. Eg: nix copy.

Or to take another tack at the question - when you say ā€œinstall timeā€, how are you doing installs of Go programs?

9

It is about gh extensions.

There are discussions on how to install those. Example: Support binary extensions Ā· Issue #4194 Ā· cli/cli Ā· GitHub

So far I found two issues:

  • #!/bin/bash is often the used shebang for scripts (already trying to spread the word among authors)
  • some Go extensions need to be patched (as per replies to this topic): gh some-extension simply fails if a pre-compiled (and not patched) binary is downloaded and used

Generally extensions have a (non standard, there is a more commonly used one but any author generally has his own) script that manages the installation.

Currently, the logic ends up always downloading a non patched binary on NixOS.

With this topic I wanted to have some grasp on how Go things work on NixOS.

So far seems to me that the straightforward solution for NixOS would be to patch the installation script to build locally and spread the word among authors (and there is also the upgrade phase).

Any suggestion is always welcome

10

They are going to have difficulties packaging dynamic-linking binaries for gh that work in all distros. It is probably for that reason that these releases for example (Releases Ā· heaths/gh-label Ā· GitHub) are static. That would be a fine way to proceed: ensure gh extensions are static. Note: this is not specifically a NixOS problem, there is no Linux-wide standard distribution mechanism for dynamic-linking binaries.

If they are not static, then it seems the discussion is going down the path of hoping the user has a Go compiler and all other dependencies for the extensions they are fetching, this will be fragile, but will work sometimes. In fact, this is where Nix could shine, because their install scripts can detect Nix, and then get a very reliable/reproducible install of their extensions, most of time bypassing compilation and using a binary substitution.

Yes, the /bin/bash problem is annoying, but people are usually okay with fixing this for portability sake.

11

Thank you for taking the time to look into this.

Can you expand on this, please? How would be done?
They could build a release for NixOs via an action and then eventually download it, is it what you mean?

This .goreleaser.yml looks interesting.

I ā€˜guessā€™ that dependencies are fetched when neededā€¦ not sure if it is a general behavior.

/bin/bash here is an issue because a few authors that I contacted were unaware of it being a possible culprit.

In the meantime, I narrowed down to a simple check for the existence of /bin/bash: if it fails, it assumes that is on NixOS and compiles locally the downloaded extension source. Probably it is too fragile to be useful on other platforms (would that possibly work on other non FHS compliant systems?), but I am trying it as cheap alternative, as I search for a temporary, simple and possibly not only NixOS oriented solution.

12

Not really, as the preference is for not using it