The problem I’m trying to solve: I want to use newer src from github to install a beta version of a package I maintain using an overlay in NixOS and nix-darwin.
I’ve been able to do it by copying the derivation into my dotfiles repo and modifying it, but shouldn’t it be possible to use overrideAttrs and override to modify the package?
Ok I took a look and it looks at first glance as though overrideAttrs might work on the results of buildRustPackage. I’m not entirely certain though because I don’t have a 100% grasp of the override system implementation.
In any case, you could certainly try creating an overlay that calls overrideAttrs to change both the src and cargoSha256.
overrideAttrs works on the attributes passed to stdenv.mkDerivation, not on the attributes passed to buildRustPackage, so at that point you already get the attributes that buildRustPackage is passing to stdenv.mkDerivation.
In order to get around this you need to look at the source of buildRustPackage and figure out how the mkDerivation attributes came to be.
For buildRustPackage the important part here is cargoDeps, which is the only location where cargoSha256 is used. So let’s say if you have a package foo and you’d like to change src and cargoSha256 you could override it like this:
foo.overrideAttrs (drv: rec {
src = ...;
cargoDeps = drv.cargoDeps.overrideAttrs (_: {
inherit src; # You need to pass "src" here again,
# otherwise the old "src" will be used.
outputHash = "... new cargoSha256 ...";
});
})
The nested overrideAttrs is for the fetchcargo function (source), which again overrides mkDerivation attributes, so instead of passing sha256, you need to pass outputHash.
To illustrate this with a more concrete (but probably stupid) example, here is how to downgrade loc to version 0.4.0:
{ pkgs ? import <nixpkgs> {}, lib ? pkgs.lib }:
pkgs.loc.overrideAttrs (drv: rec {
name = "loc-${version}";
version = "0.4.0";
src = pkgs.fetchFromGitHub {
owner = "cgag";
repo = "loc";
rev = "v${version}";
sha256 = "0cdcalfb0njvlswwvzbp0s7lwfqx4acxcmlsjw2bkanszbdz10s8";
};
cargoDeps = drv.cargoDeps.overrideAttrs (lib.const {
name = "${name}-vendor";
inherit src;
outputHash = "1qiib37qlm1z239mfr5020m4a1ig2abhlnwava7il8dqvrxzsxpl";
});
})
Here I also passed name along with the src attribute to make sure that the store path actually reflects the version change, so it’s less confusing than having a store path that says something like 0.4.1 but the actual version is 0.4.0.
Ok I wasn’t sure because I saw overrideAttrs was mentioned in the implementation of makeOverridable, but looking closer, I think all it’s doing is wrapping overrideAttrs such that the results of calling it is made overridable again.
It would be nice if functions like buildRustPackage had their own version of overrideAttrs. I guess we’d want a different name though.
Another option here is to use something like
self: super:
{
ffsend = super.callPackage <nixpkgs/pkgs/tools/misc/ffsend> {
rustPlatform = super.rustPlatform // {
buildRustPackage = args:
super.rustPlatform.buildRustPackage (args // {
src = newSrcValue;
cargoSha256 = newSha;
});
};
};
}
This re-invokes the original package (whose path you need to know of course) but overrides the call to rustPlatform.buildRustPackage such that it modifies the args before passing them along.
If anyone stumbles upon this and gets the following error type:
do not know how to unpack source archive /nix/store/1qiib37qlm1z239mfr5020m4a1ig2abhlnwava7il8dqvrxzsxpl-loc-0.4.0-vendor
You need to set the file type in name:
...
cargoDeps = drv.cargoDeps.overrideAttrs (lib.const {
name = "${name}-vendor.tar.gz";
inherit src;
outputHash = "1qiib37qlm1z239mfr5020m4a1ig2abhlnwava7il8dqvrxzsxpl";
});
...
In case anybody tries to do this with flakes:
final: prev: {
mdbook-multilang = prev.mdbook.overrideAttrs (oldAttrs: rec {
pname = "mdbook";
version = "pr1306";
src = prev.fetchFromGitHub {
owner = "Ruin0x11";
repo = "mdBook";
rev = "9d8147c52dd9d50047ba5b29e4af99f92577806e";
sha256 = "sha256-gJnQKHssO2ChiT4d037Lncd7hiOa5uh756p8TzPzbgQ=";
};
cargoDeps = oldAttrs.cargoDeps.overrideAttrs (prev.lib.const {
name = "${pname}-vendor.tar.gz";
inherit src;
outputHash = "sha256-QCEyl5FZqECYYb5eRm8mn+R6owt+CLQwCq/AMMPygE0=";
});
});
}
Hi folks, just to clarify: it appears that while you can override the outputs of a buildRustPackage derivation, it isn’t possible to override the inputs (i.e. dependencies). Please let me know if I’m wrong about this (and feel free to revert my edits to the wiki page which note this).
I struggled quite a bit with this before giving up, going so far as to reify the Cargo.lock using fromTOML, patching it, then writing it back out with remarshal, and using cargo-edit to patch the dependencies’ Cargo.toml files. My use case involves an A->B->C dependency where I need to replace “C”, so the Cargo.toml in “B” needs to be rewritten.
Specifically, I’m trying to swap out the ring cryptography library with IBM’s fork which has support for powerpc64le (PR to merge upstream). I picked tiny as an example case but pretty much anything written in Rust that uses cryptography or HTTPs relies on ring. The goal here was not to have to run cargo update for every single rust package that uses ring (or that can use it in place of openssl). This is a large and rapidly growing number of packages. It ought to be possible to calculate the new checksum for ring once, and then patch that value in using non-network-accessing operations, inside of nix derivations.
I ended up in never-ending fights with cargo, which really wants to do things its way. The closest I ever got always managed to end with something like
error: the lock file /build/source/Cargo.lock needs to be updated but --frozen was passed to prevent this
If there is a way around this please do let me know. Until then my working assumption is that cargo simply doesn’t support the kinds of magic that nix has gotten me addicted to using. The only way forward here appears to be to skip cargo and have nix drive the build process with crate2nix.
It turns out that this is possible, but requires changes to buildRustPackage in order to invoke cargo build with the weaker --offline flag rather than the --frozen flag. Once #187838 is merged that will be possible.
Here’s what an example override looks like.
This was a really motivating use for this ability. The ring crate is basically the Rust standard crypto library, and by now is a dependency of a huge part of the Rust ecosystem, as well as a big chunk of the Python ecosystem as well!
For reasons which remain mysterious, the ring maintainer has been ignoring a pull request from IBM to add the assembler primitives for PowerPC for almost two years now. These primitives are copied from BoringSSL, just like all the other ring primitives. It’s a pretty weird situation. No response of any kind, positive or negative.
In the example above I was able to replace (the non-openssl version of) the tiny IRC client with a version of ring that has IBM’s patches. It built, and I was able to successfully connect to libera.chat using TLS. Note that if you have powerpc64le hardware and want to try the same thing, you will also need #168983 because we’re still waiting for the bootstrap files to be uploaded.
This re-invokes the original package (whose path you need to know of course) but overrides the call to
rustPlatform.buildRustPackagesuch that it modifies the args before passing them along.
Packages created with callPackage automatically get a .override function (technically speaking, a functor, an attribute set which behaves like a function thanks to its __functor attribute), since lib.callPackageWith uses lib.makeOverridable to make packages overridable.
nix-repl> pkgs.ffsend.override
{ __functionArgs = { ... }; __functor = «lambda @ /nix/store/gzf4zwcakda1nykn6h0avh45xhjhvsz4-source/lib/trivial.nix:440:19»; }
Since lib.callPackageWith takes either a function or a path,
callPackageWith = autoArgs: fn: args:
let
f = if isFunction fn then fn else import fn;
one can use super.callPackage super.ffsend.override instead of super.callPackage <nixpkgs/pkgs/tools/misc/ffsend>. This avoids the lookup path impurity and allows overlaying packages that don’t have their own dedicated file (e.g. pkgs.xorg.xorgserver) or come from a flake.
I (ab)use this trick significantly when patching out /bin/sh. For example, tectonic can be overlayed like
final: prev:
{
tectonic-unwrapped = final.callPackage prev.tectonic-unwrapped.override {
rustPlatform = final.rustPlatform // {
buildRustPackage = args: final.rustPlatform.buildRustPackage (args // {
src = final.fetchFromGitHub {
owner = "tectonic-typesetting";
repo = "tectonic";
rev = "78fd97716ee111861bd981a45e6816589d16f504";
sha256 = "sha256-pKx2fzBllkv3fzUfhv9qKlPqD4JKdEN74+gsLbmwZ/o=";
};
cargoHash = "sha256-BcVepSoFogh/OU0DoQNnT8X9bNc74rMDT+3zelmfwkY=";
# https://github.com/NixOS/nixpkgs/pull/291770
buildFeatures = [ "external-harfbuzz" ];
});
};
};
}
If you’re using .override to replace rustPlatform, you don’t need or want the final.callPackage. At best it’s going to do nothing, but if the package already was overridden to change things it’s going to lose those previous overrides. Also override gives you access to the original values, so you can customize those instead of replacing them.
final: prev:
{
tectonic-unwrapped = prev.tectonic-unwrapped.override (old: {
rustPlatform = old.rustPlatform // {
buildRustPackage = args: old.rustPlatform.buildRustPackage (args // {
# override src/cargoHash/buildFeatures here
});
};
});
}
The solutions in this thread worked for me but I wanted to be able to apply potentially multiple patches to the code. By overriding src, I would have to do that manually. So came up with this: I take old source and create a new derivation by running patch on it. Then I continue as described in previous posts.
modifications =
final: prev:
let
# Rust packages need special handling: https://nixos.wiki/wiki/Overlays
addRustPatches =
pkg: patches: cargoHash:
pkg.overrideAttrs (oldAttrs: rec {
# take the original source and apply all patches before making it the new source
# we cannot use patches or patchPhase because all dependencies are vendored into
# a separate derivation before the patch phase resulting in mismatching Cargo.lock
# checksums
src = prev.runCommand "patched-source" { } ''
cp --no-preserve=mode -r ${oldAttrs.src} $out
cd $out
${prev.lib.concatMapStringsSep "\n" (patch: ''patch -p1 < "${patch}"'') patches}
'';
cargoDeps = oldAttrs.cargoDeps.overrideAttrs (
prev.lib.const {
inherit src;
outputHash = cargoHash;
}
);
});
in
{
fclones = addRustPatches prev.fclones [
(builtins.fetchurl {
url = "https://patch-diff.githubusercontent.com/raw/pkolaczk/fclones/pull/280.patch";
sha256 = "sha256:0inir6g158hfc4a1s2hwsbr887szb6mzpm961xjpisy1vgbjg9hy";
})
] "sha256-o+jsVnw9FvaKagiEVGwc+l0hE25X+KYY36hFhJwlcj0=";
};
By the way, nixpkgs already has applyPatches which does your patched-source logic.
I assume you meant cargoPatches? I’ve never seen applyPatches.
I meant applyPatches. It’s not widely used, but can be useful in some situations where the source needs to be transformed before the normal patches take effect.
You can find the implementation here. It’s undocumented in the manuals.
cargoPatches didn’t seem to work for me