Patch nixpkgs using cherry-picks without local clone

[perchun]

Did you know it is possible to cherry-pick a commit only using the GitHub API? You don’t even need to clone 5 GB nixpkgs to do that.

This is actually a better version of nix-patcher, but instead of using raw patch files (that need constant rebasing), the tool uses cherry-picks, which are very conflict-tolerant (=git is smarter with commits)

How it works

gh-cherry-pick interacts directly with the GitHub API to cherry-pick or merge branches. No local clone required!

# This tool requires a classic token with `repo` and `workflow` permissions
# or a fine-grained token with `contents` and `workflows` permissions
GITHUB_TOKEN=ghp_... gh-cherry-pick \
  --target MyOrg/nixpkgs@patched \
  `: # cherry-pick these commits` \
  NixOS/nixpkgs/3f5ba52cc4701bf341457dfe5f6cb58e0cbb7f83 \
  NixOS/nixpkgs/49ba75edefc8dc4fee45482f77a280ddd7121797 \
  `: # or merge the entire branch!` \
  Someone/nixpkgs@pr-branch

Use-case for merging a branch instead of cherry-picking specific commits is that you can track your PR without needing to update commit hashes constantly.

You can also pass links like https://github.com/NixOS/nixpkgs/commit/4e92bbc or https://github.com/MyOrg/nixpkgs/tree/patched instead of NixOS/nixpkgs/49ba75e / MyOrg/nixpkgs@patched.

Automating with Flakes

For nixpkgs specifically, you could use this script

nixpkgs_rev=$(
  nix flake metadata --json \
  | jq -er '.locks.nodes."nixpkgs-upstream".locked.rev'
)

nix run github:NixOS/nixpkgs/nixpkgs-unstable#gh-cherry-pick -- \
  `: # don't forget to replace my name with yours!` \
  --target PerchunPak/nixpkgs@patched \
  --first-hard-reset-to "NixOS/nixpkgs/$nixpkgs_rev" \
  PerchunPak/nixpkgs/41b5eba52596219fd7748228dd895a2da40d9695 \
  PerchunPak/nixpkgs/a4a2d20aa1ab1721d9f2933a935787b8e74e2309 \
  PerchunPak/nixpkgs@zellij-plugins \
  PerchunPak/nixpkgs@wayle

nix flake update nixpkgs

This will cherry-pick commits 41b5eba, a4a2d20 and then merge branches zellij-plugins, wayle into patched branch (all on my own nixpkgs fork). You just have to make a few more changes to your flake.nix:

inputs = {
  nixpkgs.url = "github:PerchunPak/nixpkgs/patched";
  # Track the official upstream here to use as a base for the script above
  nixpkgs-upstream.url = "github:NixOS/nixpkgs/nixos-unstable";
};

[perchun]

For anyone interested, would appreciate any help with merging Thanks drupol!

gh-cherry-pick: init at 1.2.0

[toastal]

Nixtamal supports patches generically patching any input with a diff/patch file (local or remote reference supported) while also not requiring flakes or giving proprietary Microsoft GitHub any special privilege tool or syntax (like Someone/nixpkgs@pr-branch).

version "1.0.0"
patches {
	// Unique name for referencing in manifest inputs
	chroma-0.22.0 "https://patch-diff.githubusercontent.com/raw/NixOS/nixpkgs/pull/478519.patch" {
		// Optionally override the project default hash algorithm for the patch
		hash algorithm=SHA-512 expected="1mdsfx204bgia572fydnmjy78dkybbcnjx20qn9l4q65r29ry28c"
	}
}
// Define inputs
inputs {
	// Unique name for referencing in Nix
	nixpkgs {
		// Fetch an archive with string templating support
		archive {
			url "https://github.com/NixOS/nixpkgs/archive/{{fresh_value}}.tar.gz"
		}
		hash algorithm=SHA-256
		// Apply patches to the source now while awaiting review
		patches chroma-0.22.0
		fresh-cmd {
			$ git ls-remote --branches "https://github.com/NixOS/nixpkgs.git" --refs nixpkgs-unstable
			| cut -f1
		}
	}
}

Which generates a lockfile + shim that allows users to.

let
   inputs = import nix/tamal { };
   pkgs = import inputs.nixpkgs { };
in
pkgs.chroma

Since the patch URL points to a pull request, next time you run nixtamal refresh you can get the new patches as it goes thru review. But just as easily you could point to each commit individually if preferred. Patches are just pkgs.applyPatches under the hood.

[perchun]

New syntax for flake inputs seems interesting, but that is overkill sometimes and has a learning curve. Anyway, the project seems very cool. Also, I had to constantly rebase my patch files when using nix-patcher. Maybe nixtamal does additional magic, since nix-patcher uses patch2pr instead of git apply

[rina]

Really cool project!

The API cherry pick approach is really clever. I wonder if it’s something patch2pr would be interested in because they could integrate this into thwir library. But maybe their main focus is on PRs and GitHub would handle merging itself.

@toastal

[not] giving proprietary Microsoft GitHub any special privilege tool

For me with nix-patcher, the constraint of not checking out the 5GB+ git history was a hard constraint, but a worthwhile one I think (correct me if tamal can do that too). I’d love to get off GitHub too, but using the GH API was the easiest way to do that.

Maybe it’s possible to do something similar for generic git servers using some blob processing. Github must be doing something behind the scenes of these API calls. I don’t know enough about git internals to know if this could work and couldn’t find any resources.

I also have one feedback for @perchun Since we’re stuck on GitHub for now, it would be nice to support commit links like https://github.com/UQ-PAC/BASIL/commit/fd206265a2b714377d118a91fe471e601930f0dc as a command line input, especially if this is a generic tool and people might be unfamiliar with nix flake syntax. It would also make them clickable to go to the commit page

Anyway, really great tool!

[perchun]

@rina

I don’t know enough about git internals to know if this could work and couldn’t find any resources.

GitHub exposes an API for operating on raw Git blobs. You can actually read an instruction on how to implement cherry-picks here. Although it is in pseudo-code, so I find it confusing; you might want to read the implementation in Python instead.

@rina

it would be nice to support commit links like https://github.com/UQ-PAC/BASIL/commit/fd206265a2b714377d118a91fe471e601930f0dc as a command line input

Love the idea! Added it in 1.3.0 (nixpkgs PR)

---

BTW, the tool is already in nixpkgs-unstable

[toastal]

If you add .patch to any {Forgejo,Codeberg,GitLab,MS GitHub} pull request’s URL you get the plaintext patch set, the kind you would get from git send-email (mboxes on SourceHut), which you can apply with pkgs.fetchpatch2 + pkgs.applyPatches to any input. This can be a low-tech, serviceable option that usually doesn’t require the same throttling, tokens, & maintenance of one specific HTTP API.

[rina]

Yeah I know the idea. It has been a while, but last time I tried to use applyPatches to patch a flake input, it caused trouble with cross compiling because of import from derivation.

I guess you’re suggesting to replace flakes with nixtamal. It’s just not clear from your website whether nixtamal uses import from derivation to apply patches. Or, maybe you can have IFD + cross compiling with some more careful pkgs splicing, not sure.

[doronbehar]

Interesting workflow for patching Nixpkgs! I too created such a workflow and shared it here:

GitHub actions to update your Nixpkgs fork with PRs' patches

Motivation After a few small dissatisfactions from nixpkgs-patcher by @gepbird, and after encountering substantial issues with nix-patcher, I created a setup that avoids many of the drawbacks of the above two solutions. This solution is based upon a periodic GH action that runs on your fork of Nixpkgs. It updates a branch on your fork called channel/personal-stable (name can be changed of course) with commits from PRs listed in the GH action’s workflow yml file. With this branch, you can use th…

[rina]

With nix-patcher I encountered substantial issues I didn’t bother reporting as it seems the project as abandoned.

I really wish people would submit bugs or experiences. Even if not for the maintainers, then for future people evaluating the project.

Anyway, yeah, a task that runs routinely is good way to make sure it keeps working. I’ve also come around to using Git commands is probably the most reliable way forwards.