Making packages/projects available to nixos

Help
1

I still trying to wrap my head around how to make personal (open source) projects available to nixos (via flakes). I will focus on go for the moment (but rust, zig, java and node would also be interesting).

I have a project that uses goreleaser to build binaries and deb and apk for various architectures. It makes the binaries available to homebrew via my personal tap repo.

Now I am aware that things are very different in nix land. But I am still struggling to make the transition and supporting nix.

Since nix (usually) builds from source it would be so nice if I could just point my flake to my github repo, and (after adding some nix files) I could just build (install when using cachix.org) it via nix.

But is it that easy? Here is what I looked at:

1) goreleaser

I found that goreleaser now has some support for nix but seems to require for me to have my own NUR. Which seems similar to having to maintain the homebrew tap. But IIUC the goreleaser integration is not source based. So that’s not really “the nix way” I guess?

2) nixpkgs

As long as the projects are tiny including them into the mighty nixpkgs is probably not the way to go. Not sure.

But just to see how things work I thought to look into the nixpkgs repo. I picked to look at envsubst as an example. But frankly speaking I don’t see how it works yet. There is nothing about nix in the envsubst repo and in nixpkgs it just has (summary, see the link for full)

buildGoModule rec {
  pname = "envsubst";
  version = "1.4.2";
  src = fetchFromGitHub {
    rev = "v${version}";
    sha256 = "sha256-gfzqf/CXSwGXBK5VHJnepFZ1wB3WElpEp6ra9JI4WtY=";
  };
  vendorHash = "sha256-L0MbABgUniuI5NXc4ffBUsQRI716W/FiH38bGthpXzI=";

…and I am not sure how I could/should maintain these hashes myself (if this was my project).
Manually?

I found these docs but that didn’t really help. The docs in the wiki are also unclear on that.

After reading a particular part in the manual, a few links later I found the contributor guide to nixpkgs and IIUC I would need to manually run something like this?

Find the value to put as hash by running nix-prefetch-github --rev 1f795f9f44607cc5bec70d1300150bfefcef2aae NixOS nix

And I am still not sure I follow.

And then…

3) xs as an example

After some more research I found this project.

  • It seems to be it’s own flake referencing the local package
  • It uses buildGoModule
  • It has code to update the hashes for xc.nix (via script from a xc.nix.tmpl)
  • It has goreleaser as a separate build for the binary assets that just updates the flake on releases

Is this a good example to follow? Are there better?
Are there docs I have missed?

Thanks for the help.
cheers,
Torsten

1 Like
2

Personal piece / opinion (thank you for writing those structured posts, I really enjoy reading them and it gives me the opportunity to express those opinions)

I think there are two models of consumption: upstream (project’s binaries, etc.) and downstream (distribution’s binaries, etc.).

In almost all the cases, you want the latter for users and very rarely the former. Because of that, optimizing for the former is a waste of time and should focus on what is the audience consuming that, e.g. developers/beta-testers/etc. ?

With that in mind, GoReleaser is an anti-pattern and overengineering the thing is not really important. xc seems to be self-packaging itself where it could just package itself in nixpkgs IMHO.

I personally will rarely [1] or never consume package which are not in nixpkgs, it’s a quality and standards problem.

Therefore, xc for me is not a good example neither.

I think the good example is the one where you build a default.nix or a flake.nix that works for you as a developer, you make it grow as you get feedback from other developers and testers and when you can, you just send it to nixpkgs.

Normal users use nixpkgs, advanced/developers/testers users can consume your upstream if needed in some scenarios.

[1]: Only special cases.

1 Like
3

There’s nix-init which seems to be doing some (or most) of the work involved when making a new package. It’s maintained by @figsoda.

This is the relevant documentation. There’s also nurl for a bit more automation around Nixpkgs fetcher calls:

Unfortunately, Nixpkgs docs concerning how to actually make packages generally suck. If you really want to understand it or need a custom solution, at the moment you have to read the code, a lot of code, and figure out what could work for you.

The documentation team is of course, in a sense, responsible to improve the situation, but the sheer scale of it makes things take forever. @danielsidhion is currently doing the heroic work of a complete review pass of the Nixpkgs manual. Just to give you a sense of the API surface area, here’s a survey of the Python infrastructure @alejandrosame diligently compiled last year.

For a general introduction to packaging in Nixpkgs, @proofconstruction wrote a tutorial for nix.dev:

You’d then expose your Nixpkgs-style packages in flake.nix with callPackage, as nicely outlined by @jade:

If you need more depth, @NobbZ wrote an overview of how to use callPackage, and if that’s still confusing, @roberth and @infinisil made a really good introduction to fixed-point computation on attribute sets.

Everything is kind of there, but both in terms of tooling and documentation we’re still lacking well-rounded packaging. For overview-style documentation, the place for that is nix.dev – contributions highly appreciated!

2 Likes
4

First of all - thanks for all the feedback and pointers. :heart:
I got distracted for a while but are now back on this.

Maybe to step back and formulate the outcome of this exercise:

  1. I would like to be able to install my project from github as nixOS package during the development phase
  2. I then would like take the package definition of a release and bring it over to nixpkgs for wider consumption

My hope was that I could point to a project with a lock file which then could be turned into a nixOS package. But it seems like the hashes needed on the nixOS side might not actually be the same as the hashes in the lock files or commit SHAs.

And it seems like that’s what nix-init is trying to help with (via nurl) - finding the hashes. That’s similar to what this shell script is doing - which is also outlined in this tutorial.

The tutorial does not cover integration with existing language level dependency managers though. I am curious to see how that changes things.

I don’t quite understand why nix-init failed for me in the beginning, but I do have the default.nix now.

The next steps are

  1. refer to the default.nix to install it on nixOS
  2. add some dependencies to the go build and see how that changes the process
  3. find a way to maintain the default.nix from github actions
  4. find a way to install from the main branch instead of just a release
5

As for the install it seems I also need a flake.nix to include it in my flake config.

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    release-go.url = "github:tcurdt/release-go";
  };

I was looking at deploy-rs as an example.

But that setup looks quite different from the default.nix generated by nix-init.