A somewhat helpful update script which also changes how we pull in the *buf.buil…d* dependencies, hopefully causing less breakage whenever *buf* receives an update (see #388353).
Note: the script can be used to pull the package from 2.71.1 to 2.71.3 and it seems to work?
~~Also checking the GitHub API should be done better considering zitadel supports multiple versions as far as I can tell?~~ *Solved in the current version.*
In any case, the code quality and especially readability is terrible, this should be a starting point for improvement.
Below the usual checklist is a write-up on the current state of covering *buf* in nixpkgs, in relation to the zitadel package which was intended for a comment in a PR originally, however since this commit implements one of the approaches we may as well discuss it as part of the PR.
Please note that I am not an expert in zitadel, protobuf, golang, Nix, and especially not buf.build or buf in general, so many grains of salt adviced.
# Things done
- Built on platform(s)
- [x] x86_64-linux
- [ ] aarch64-linux
- [ ] x86_64-darwin
- [ ] aarch64-darwin
- For non-Linux: Is sandboxing enabled in `nix.conf`? (See [Nix manual](https://nixos.org/manual/nix/stable/command-ref/conf-file.html))
- [ ] `sandbox = relaxed`
- [ ] `sandbox = true`
- [ ] Tested, as applicable:
- [NixOS test(s)](https://nixos.org/manual/nixos/unstable/index.html#sec-nixos-tests) (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests))
- and/or [package tests](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#package-tests)
- or, for functions and "core" functionality, tests in [lib/tests](https://github.com/NixOS/nixpkgs/blob/master/lib/tests) or [pkgs/test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/test)
- made sure NixOS tests are [linked](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#linking-nixos-module-tests-to-a-package) to the relevant packages
- [ ] Tested compilation of all packages that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"`. Note: all changes have to be committed, also see [nixpkgs-review usage](https://github.com/Mic92/nixpkgs-review#usage)
- [ ] Tested basic functionality of all binary files (usually in `./result/bin/`)
- [25.05 Release Notes](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2505.section.md) (or backporting [24.11](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2411.section.md) and [25.05](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2505.section.md) Release notes)
- [ ] (Package updates) Added a release notes entry if the change is major or breaking
- [ ] (Module updates) Added a release notes entry if the change is significant
- [ ] (Module addition) Added a release notes entry if adding a new NixOS module
- [x] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md).
---
Add a :+1: [reaction] to [pull requests you find important].
[reaction]: https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/
[pull requests you find important]: https://github.com/NixOS/nixpkgs/pulls?q=is%3Aopen+sort%3Areactions-%2B1-desc
---
This solution has opted for *pointing to buf export data*, which requires that a bunch of hashes from FODs get precomputed.
Because the FOD is a derivation (three of them actually) with a command that actually runs, this is not trivial.
However the script also covers the repo itself by checking GitHub for updates, yarn, the protobuf dependencies, and since it is the only FOD hash left also the go modules.
The repo and yarn work with the usual pre-fetchers.
Protobuf by running the buf commands with modified cache dir (so as to not bother anyone with extra files lying around).
And the go modules, by virtue of being the only thing left, can be resolved by trying to build the package with lib.fakeHash and getting the FOD error and parsing that.
# buf building
Let's start by describing the problem, our prior approach, and its issues.
The problem is relatively simple, we need to build the protobuf specs and get bindings for them, however those specs depend on other specs, and the only reference we have is to an online service which gives us artifacts.
So since we don't have the source repositories for any of this we have to make do by fetching the artifacts in an FOD (to allow us network access for fetching it) using upstream tooling.
The approach has the issue of the FOD being dependent on the upstream tooling which on version change occasionally changes its output and therefore the FOD hash.
Based on this a change can mean several possible things, which can roughly be classified by how the build is changed.
Since many of those approaches have obstacles that are the same for each subset there is a context section further down so I don't have to explain the same thing multiple times.
Terms described in the context have *emphasis*.
## same build, extending the fetch
### fetchtarball
Since the dependency is based on the *BSR commit* we could include that commit in a fetcher much like we include git revs.
If we had a deterministic fetcher which could fetch the *BSR commit* (or rather its output), then we would not depend on *buf*, and therefore not be affected by its version changes as part of the FOD.
The build process would then be separate but offline, thus not an FOD, so a version change in *buf* would also cause a rebuild of the bindings and package.
However *buf.build* has apparently [disabled (or broken) their tar export with the examples on their docs not working](https://buf.build/docs/bsr/module/export/#export-with-curl) as far as I can tell.
This means we cannot use a tarball fetcher.
### BUF_CACHE_DIR
By keeping the build exactly the same but running part of it (`buf dep graph`) beforehand to fetch all dependencies, FODing the cache, and reinjecting it into the build, we can remove the FOD from the actual build.
However testing *nixos-24.11* vs. *master* nixpkgs *buf*, the cache dir shows differences (newer version creates an empty folder for plugins), which means we're back to the point where we depend on *buf* versions for the output of an *FOD*, which on *buf* updates can cause hash mismatches.
The only difference is that now we're hashing internals instead of output-related data, which might prove more stable, but can also change without notice (e.g. in a patch version) so even small updates can cause this.
Prebuilding the cache dir ourselves doesn't seem quite feasible since it includes a plethora of extra files including YAML files generated from build metadata.
### naive buf export
*buf* has an export subcommand which fetches the build artifacts and outputs them in an output directory.
Since this includes only the built output of what is part of the secure hash of the *BSR commit*, it will most likely not change in between *buf* versions.
The problem here lies with integrating it into the build since we cannot point *buf* at the directory and say "go" without modifying the build files significantly.
## changing the build
### pointing to buf export data
If we used the naive *buf* export approach from above, we'd have to somehow point *buf* at the new data.
Unfortunately the *buf.yaml* is structured in a way that we cannot easily replace those paths since zitadel uses the v1 version and the "deps" key which only accepts a *BSR* reference and nothing else (no path, no git repo, no nothing).
Moving the import of that data elsewhere would require changes to the yaml files which is quite hacky tbh.
Especially since we're now pulling in already built protobuf files vs. the original source repositories.
Luckily protobuf itself has an import system which accomodates files lying around in directories, that's how it usually works without *buf*.
By simply copying the prefetched directories over and erasing the deps from the *buf.yaml* as well as *buf.lock* and instead adding them to the buf.work.yaml which lists the project-wide directories to be built we can shorten out every *buf* logic pertaining to those files and treat them as natural protobuf files.
It still involves wild hacking around in *buf* configuration, however the code pulls from the repository itself.
As long as upstream doesn't introduce any indirect dependencies this should continue to work indefinitely.
## source code all the way down
Instead of fetching the build artifacts, which are the thing fetched online, we could try and fetch the sources of the dependencies.
This has multiple issues explored in the next few sections.
### getting the source repository
First of all we would have to get the source repositories, in case upstream ever changes these.
These are technically easily guessable (replace *buf.build* with github.com works for most of them, but I wouldn't feel comfortable with that.
Since we're already in "update script" territory we could also ask the upstream API about what the *BSR commit* was built from.
Now the API there seems to be protobuf over gRPC only, which makes it a bit unwieldy in a shell script (compared to god ol' curl&jq), however *buf* comes with a curl subcommand which allows us to get JSON from that API relatively cleanly.
The only part of the API which is public yet is the reflection API which is entirely irrelevant to us.
However since this API is used by *buf*, we can just use `--debug` and have a look.
Easy enough we get access to `buf.build/buf.registry.module.v1.CommitService/GetCommits` and can fetch [*BSR commit* information which includes [a source control reference](https://buf.build/bufbuild/registry/docs/main:buf.registry.module.v1#buf.registry.module.v1.Commit).
Well, an optional one, which isn't present for the one dependency I tested with, so that's a dead end.
### fetching the source
So if we went that way, a mix of hardcoding and guessing URLs it is?
So if we assume we get all the sources in a JSON file in the nixpkgs repo that can be updated with an update script we're set to pull those dependencies and build them.
The easy YAML path would be to include the git repo in the sources from where it pulls, but that leads us back to *buf* accessing the network.
Fetching them outside and passing them into the build as paths it is.
This is probably possible with some extensive yq-go hackery.
### building the source
Assuming we have a fully local *buf.yaml*, we can start building.
Or can we?
Well, given that zitadel only has three deps which AFAIK are all standalone with no further deps, that'd work out, but as soon as that changes we're met with recursive dependency resolution.
So let's assume we're fine, either because we implemented the recursive resolution in the update script or because we assert no further dependencies.
Now we also have to assume that *buf* can magically handle those repos.
Without the sources we've been fetching the already built protobuf files which have a very streamlined format since they're built for consumption by *buf*.
Now we might be building with different build systems, at which point this becomes rather convoluted and complex, but potentially possible.
Now this section alone has significant assumptions and required effort so it's unlikely we'll find maintainers willing to build and maintain that, as we'd basically vendor protobuf outside the source repo.
## skipping the build
### zitadel on buf.build
There's another option, technically.
We could fetch the entire zitadel protobuf build from *buf.build*; it's on there.
If we somehow managed to fetch the entirety of it including the dependencies, or maybe even go all-in and download the code-gen'd bindings from there (which seems to be a thing?) we'd only have to fetch that and FOD it.
I don't particularly like the idea, but if it could be as simple as fetching a tar and extracting it over the source repo in preBuild then I guess it works.
With the *BSR commit* being immutable and pinned this should work fine, so we'd just have to update that with respects to the corresponding zitadel version, which may or may not cause headaches.
Since the output should be tied to the commit we should also be able to fetch that using *buf* if need be, and not worry about version changes.
However I have not looked into that.… basically at all.
# Context
## BSR commit
A repository on a *BSR* has a "commit" structure for the artifacts like git has for repository contents.
Commits can be pinned in a *buf.lock* file.
They look like your usual sha256 and are a secure hash over the entire commit contents, so they are verifiable.
Since we're talking about the hash of the output, this has no 1:1 relation with the source code repository; source changes which do not touch the proto files or do not cause a semantic change do not generate a new commit.
## buf.build
*buf.build* is the flagship instance of a *BSR*.
## buf.yaml
The configuration file for a *buf* build.
Notably there are two versions, v1 and v2, with a different structure and slightly different feature set.
The v2 is also much closer to the buf workspace configuration for v1, but differences exist.
## buf.lock
Similar to Cargo.lock or whatever else system you know, *buf* allows pinning a dependency to a specific *BSR commit*, thereby allowing you to reproduce a build (if you have connectivity of course).
## buf
*buf* is a CLI tool for building protobuf specs and using plugins to generate bindings.
## BSR
A Buf Schema Registry or *BSR* is sort of a combined CI service which gets protobuf input using *buf*, and builds it, offers artifacts, documentation, and some extra features.
This is how *buf* builds plugins since plugins are basically containers and [buf does not want to depend on a container runtime](https://github.com/bufbuild/buf/issues/2543#issuecomment-1841804678).
It also offers a *BSR API*.
The *BSR API* is a gRPC and protobuf based API offered by *BSR* instances for querying information.
As far as I can tell there is no REST API available?
Signed-off-by: benaryorg <binary@benary.org>