Proper way of applying patch to system managed via flake

anpin · August 18, 2022, 2:02pm

Hello nix community, I’m trying to apply a patch for the first time in my flake config as shown below, but it fails with can't find file to patch at input line 3 what am I doing wrong? Thank you in advance for support!
Error:

applying patch /nix/store/33bfi40rxq2wiv4g0ngsgygk3wyx2pzc-b693f928f0c924736c6e958064a170d3e130e19c.patch
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/pkgs/development/libraries/raft-canonical/default.nix
|+++ b/pkgs/development/libraries/raft-canonical/default.nix
--------------------------

Flake.nix

.......
laptop = nixpkgs.lib.nixosSystem {
        modules = [
          ({ nixpkgs.overlays =
          [
            (final: prev:
            {
              raft-canonical = prev.raft-canonical.overrideAttrs (old: {
                patches = (old.patches or []) ++ [
                  (prev.fetchpatch {
                    url = "https://github.com/NixOS/nixpkgs/commit/b693f928f0c924736c6e958064a170d3e130e19c.patch";
                    sha256 = "sha256-4sak/cp/I9USx9RLu4ZUWRcIFSqRgOor0CAHoDH77ls=";
                  })
                ];
              });
            })
            ......
          ] ; 
          })
          home-manager.nixosModules.home-manager
          {
              .......

Edit: I have also tried to use PR link as a patch insted of commit and gives me the same result

NobbZ · August 18, 2022, 5:34pm

You are trying to apply a patch meant for nixpkgs on a source of something else.

That can not work.

What are you actually trying to do?

anpin · August 18, 2022, 6:29pm

I’m trying to enable lxd virtualization and it has a broken test in dependency raft-canonical which was fixed here (or PR) but haven’t made it to nixos-unstable. Path to the file in error seems legit and matches the patch, so I don’t really understand what am I doing wrong here.

NobbZ · August 18, 2022, 6:44pm

Your overlay tries to apply this patch:

https://github.com/NixOS/nixpkgs/pull/184524/files

To this codebase:

That codebase does not hold a folder pkgs, nor any nix file at all.

You need to use overrideAttrs to do the changes manually:

prev.raft-canonical.overrideAttrs (_: {
  postPatch = ''
    substituteInPlace test/integration/test_uv_tcp_connect.c --replace \
      "TEST(tcp_connect, closeDuringHandshake, setUp, tearDownDeps, 0, NULL)" \
      "TEST(tcp_connect, closeDuringHandshake, setUp, tearDownDeps, MUNIT_TEST_OPTION_TODO, NULL)"
  '';
}

PS: The fix should be in unstable already, so you actually do not need to do anything but updating: https://nixpk.gs/pr-tracker.html?pr=184524

TLATER · August 19, 2022, 3:20am

If the commit had really not made it to unstable yet, the way to change the version of nixpkgs you’re using to a specific commit would be:

inputs.nixpkgs.url = "github:nixos/nixpkgs?rev=b693f928f0c924736c6e958064a170d3e130e19c";

Patching nixpkgs is generally the wrong approach for flakes (and hard, but maybe not impossible? At worst you can clone nixpkgs and use a local path:). Just use the commit as your input directly.

What you’re doing would work fine for patches made against specific projects, not patches to nix derivations for those projects.

anpin · August 19, 2022, 4:46am

Thank you both for your responses. The solution @NobbZ provided works and everything builds up and I finally see the following new units were started: lxd.service. However if the patch already made it to unstable as the https://nixpk.gs/pr-tracker.html?pr=184524 claims the issue seems to persist as with my patch commented out it returns the same result:

# nix-channel --list
nixos https://nixos.org/channels/nixos-unstable
# nix-channel --update
unpacking channels...
# nixos-rebuild switch --flake .#$HOST
error: builder for '/nix/store/zmrxirkwscp1nwgqs3sg22wdfxm6yy6s-raft-canonical-0.11.2.drv' failed with exit code 2;
       last 10 log lines:
       > # XPASS: 0
       > # ERROR: 0
       > ============================================================================
       > See ./test-suite.log
       > ============================================================================
       > make[2]: *** [Makefile:4070: test-suite.log] Error 1
       > make[2]: Leaving directory '/build/source'
       > make[1]: *** [Makefile:4178: check-TESTS] Error 2
       > make[1]: Leaving directory '/build/source'
       > make: *** [Makefile:4418: check-am] Error 2
       For full logs, run 'nix log /nix/store/zmrxirkwscp1nwgqs3sg22wdfxm6yy6s-raft-canonical-0.11.2.drv'.
error: 1 dependencies of derivation '/nix/store/dz0qqwiwv46mr35h0ldx4q08f4j3fwgp-lxd-5.4.drv' failed to build
error: 1 dependencies of derivation '/nix/store/csvgmahd7zdl1rrxcrwmv8rrq0qzs315-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/zg5vx80sydkgiqva8v47cxgwnp9qglzv-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/f2wydfjwga8wzprlhxyjjjlqdcfn6fsj-unit-lxd.service.drv' failed to build
error: 1 dependencies of derivation '/nix/store/fgp0q68dvac5n07mxllb5cwc63vph27n-nixos-system-sabbath-22.11.20220730.7b9be38.drv' failed to build

I also still don’t understand how one can apply a patch from commit itself rather than copy-pasting the solution from there into their config or overriding all packages as suggested by @TLATER .

NobbZ · August 19, 2022, 5:08am

nix-channel is not relevant for flakes.

Can you please update your flake and try again?

nix build nixpkgs/nixos-unstable\#raft-canonical works for me and fetches from the cache.

ParetoOptimalDev · August 19, 2022, 5:09am

I disagree. You just make an overlay that applies the patch and uses a fixed output derivation.

NobbZ · August 19, 2022, 5:13am

An overlay is not “patching” as meant in this discussion.

Here “patching” means applying a patchfile to downloaded sources.

anpin · August 19, 2022, 5:44am

Oh thank you for explanation again, I’m very new to nixos and actually never did update the flake lock, so haven’t even thought about it. Now it downloads enormous amounts of updates and hopefully would work afterwards.

ParetoOptimalDev · August 19, 2022, 6:16am

Right and I literally mean an overlay that adds patches like the example OP had issues with for raft-canonical isn’t the wrong approach for flakes.

Out of these options:

Modifying a packages patches with an overlay would be the best/easiest approach imo.

anpin · August 19, 2022, 7:35am

indeed nix flake update fixed the issue for me

TLATER · August 19, 2022, 10:33am

Could you elaborate on why you hold that opinion?

Grabbing a rev from master is just like taking a nix flake update from the future. The only difference between doing this and not using the computer for a week and then updating is that some packages may not yet be built on hydra, but as long as the delta is small that should be a very minor concern.

On the other hand, making a patch means you have to rewrite the patch by hand. Or in the case of modules, do a significantly more complex operation to shove the updated module into the module system while replacing the old one. Unless there is a really good reason to, why would you go through that effort?

Modifying a packages patches with an overlay would be the best/easiest approach imo.

It’s easy, but not “someone has already done the work for me” easy.

ParetoOptimalDev · August 19, 2022, 3:38pm

Sure, I’d be glad to!

For me updating nixpkgs to master can mean recompiling 2-3 ghc versions since I use Haskell.nix, recompiling mongo, or in some cases I’ve noticed it meant recompiling webkit! Those all take on the order of hours to a day in webkit’s case.

Additionally updating to nixpkgs master typically breaks a lot of my packages in my admittedly huge flake system configuration. A workaround is using a different nixpkgs just for specific packages, but that can cause problems as well such as breaking the vscode terminal if nixpkgs mismatches.

I’m not sure what your experience is, but here’s a recent one of mine:

Emacs package org-ql has a bug
I debug the issue find the pull request fixing it
I lookup how to do a patch since I haven’t done one before

I put this in my overlays:

final: prev: {
    org-ql = epkgsPrev.org-ql.overrideAttrs (oldAttrs: rec {
      patches = [
        
        (prev.pkgs.fetchpatch {
          url = "https://patch-diff.githubusercontent.com/raw/alphapapa/org-ql/pull/237.patch";
          sha256 = "1j28plzxv8wp90gv6vv3sk7gybdy2js3vknyvx0kpwqim93yndn5";
        })
      ];
    });
}

So in my experience I didn’t need to create the patch. Even if I did though, it’s pretty simple to do with git or especially with magit (emacs git porcelain) and simply add the file to my flake.

I think you might agree that my example:

Had someone else already “do the work” by using the PR’s patch url
Affects less surface area and is less likely to break ones current configuration
Will be much faster than waiting for a full nixpkgs update

Hope this helps, but look forward to any more comments or questions!

NobbZ · August 19, 2022, 5:04pm

There is a difference between patching nixpkgs and patching a “packages” source. You did the latter. TLater says the former is between hard and impossible.

And even though I know it has been done already, with flakes and without, the resulting code to make use of it is ugly, and I think is IFD as well, though I don’t want to put my hand fire for that.

ryantm · August 20, 2022, 2:35am

Here’s how I patch my flake nixpkgs:

outputs = {
    self,
    nixpkgs,
  }: let
    config = {
      allowUnfree = true;
    };
    patches = [
       # patches like fetchpatch here.
    ];
    pkgsBoot = import nixpkgs {
      system = "x86_64-linux";
      inherit config;
    };
    nixpkgsPatched = pkgsBoot.applyPatches {
      src = pkgsBoot.path;
      inherit patches;
    };
    pkgs =
      if builtins.length patches == 0
      then pkgsBoot
      else
        import nixpkgsPatched {
          system = "x86_64-linux";
          inherit config;
        };

Lun · August 20, 2022, 5:33am

Can use nixpkgs.legacyPackages.${system}.applyPatches rather than having two imports for it (just outsourcing the extra import to the nixpkgs flake, not avoiding it. Feels cleaner but not important.)

To include an entire PR’s patch rather than just one commit, you can use a URL like https://github.com/NixOS/nixpkgs/compare/d536e0a0eb54ea51c676869991fe5a1681cc6302.patch (can also do nixos-unstable…hash.patch at the end iirc or staging…hash if need a different base)

Can be useful to add an outPath for the final pkgs with // { outPath = "${nixpkgsPatched}"; };

iwanb · August 20, 2022, 9:12am

There is a relevant issue to make it simpler/built-in: Support flake references to patches · Issue #3920 · NixOS/nix · GitHub

Last time I had trouble getting something like what ryantm suggests working, although this looks simpler than the examples in the issue.
So I forked nixpkgs, created a branch with my current locked revision + the PR commits applied on top, then changed my nixpkgs input to point to that.

andrevm · August 20, 2022, 10:44am

With flake-utils-plus, this is as easy as:

utils.lib.mkFlake {
  ...
  channels = {
    unstable = {
      patches = [ ./patches/PR.patch ];
    };
  };

You can then download a PR patch from https://github.com/NixOS/nixpkgs/pull/<num>.patch and refer to it in the channel’s patches, and it’ll get automatically applied.

NobbZ · August 20, 2022, 1:08pm

That requires FU+, and there are a lot of people arguing against usage of those frameworks. Even regular flake utils.