Building package from local modified sources

Help
1

Hi. I’m hunting for any info on how to build a Nix package from local (and modified!) sources. To be clear, not the modified Nix package files from the Nixpkgs repos but the actual original program modified source code tweaked locally.

IDK whether I can’t quite word properly what I’m looking for or the topic is generally a grey area. But all I keep finding is:

  • How to force pull and build some Nixpkgs package locally as opposed to auto downloading pre-built bins from Nixpkgs cache.
  • How to clone Nixpkgs registry repo and build/install packages from that local Nixpkgs registry repo as opposed to online Nixpkgs registry repo on Github.
  • How to build a custom version of a package with some custom compiler flags.
  • How to build a custom version of a package and apply a source diff patch in the process.
  • etc

Some of these assorted shreds of info I found (discourse/google/reddit/blogs/youtube) are rather dated and no longer apply anyways. But regardless, none answer the two fundamental questions I’m trying to figure:

  1. Assume you are daily driving a certain program. The project is hosted on Github and is widely available for everything including NixOS via Nixpkgs. Now you want to improve certain things and potentially contribute back to the project. Meaning you want to clone the project locally and set up a dev environment so you can modify the code, build and run it locally on your Nixos box. What is the most efficient way of achieving this?
  2. Not to distract from point 1 above as it is a generic task on its own, but the program I want to tweak is an always on running service. As in actually running on the box I want to develop it on, originally installed via standard configuration.nix route. It is a popular Wayland compositor WM. So naturally, the next question after building it from sources would be how on earth do you run it to test/debug given that it is already running and in fact is one of the fundamental system pieces needed to do almost anything including Nix package management and the actual development. As in letting you run all other programs needed for development starting with multiple terminal instances and finishing with web browser for docs etc. A catch 22 situation…

Any advice please? Any pointers to the relevant Nixos doc/wiki pages or any external guides? Both of these sound like rather usual common place tasks in my head yet I’m struggling to find anything relevant.

Thanks

2

E.g. for sway:

{ pkgs, ... }: {
  programs.sway.package = pkgs.sway.overrideAttrs (_: {
    src = /path/to/sway;
  });
}

If you’re using pure eval you can’t use an absolute path there, so you either have to clone to a relative path, or use fethchurl with a file:// path; or a flake input with flake = false; if using flakes.

For the second part, consult the dev docs of your project, that’s not NixOS related. All projects need to deal with that question, and usually do it a bit differently; TDD is my favorite solution to this.

3

Thanks! Though this is still leaving way too much freedom for imagination… what exactly would that do? Would it:

  • Automatically pull all the relevant sway dependencies and dev environment?
  • Build sway package and install it as if it was installed from Nixpkgs rather than from locally built sources? Complete with honoring all the original nix package config entries and auto running this local build on the next boot?
  • What is the standard procedure for continuously rebuilding such a package? I suppose “rebuild switch” would be an overkill…
  • Where exactly would you even put that nix snippet into?
  • Do you really need to manually pull the sources given that the package manager can already do (and does for any online cache misses). I mean, isn’t there an official correct nix way of fetching the package sources and setting the dev shell environment ready for this package development?
4

Yes. It’s overriding the source of the sway package with your local code. Sway will be rebuilt from scratch locally and that new sway becomes the sway of your next NixOS generation.

You wouldn’t. That’s a question about the sway development process; refer to their contribution docs.

Typically when developing on something you don’t want to immediately test it in “production”.

With sway specifically, you probably want to build it with make and run it as a nested compositor for testing, rather than put it in your NixOS config straight away.

Overriding the package is mostly useful for if you’ve made some changes you want to daily drive before they are merged upstream (or if you make regular changes and want to avoid the merge overhead or if you fork it fully, …).

It’s a module, the programs.sway bit can go into e.g. configuration.nix. You might want to read the basic tutorials on https://nix.dev.

I mean, technically you can do this with nix develop or nix-shell by running the fetchPhase (I think), but this doesn’t necessarily give you a proper git repo to develop in.

The sources are often tarballs or otherwise not intended for development.

5

So there’s a distinction to be made here between a build environment (everything you need, and only what you need, to build a package from source) and a dev environment (the things you want to develop on a package). Nixpkgs is basically one big library for how to make build environments. It contains nothing about dev environments. You can use the Nixpkgs-provided build environment as a bare-bones dev environment if you want — that’s as easy as nix-shell '<nixpkgs>' -A some-package, in the standard (not flakes) way of doing things. But if you’re expecting incremental recompilation, language servers, any of that sort of thing, you’re going to be writing a shell.nix file and putting all that stuff in there. Here are some tips for getting started, though the details will vary based on the language ecosystem your project is using.

If you’re not actually asking a question about a dev environment, and instead just want to know how to build and use software that you’ve modified, that’s where TLATER’s advice above comes in handy.

1 Like
6

Ah wish it was Sway indeed. I’d feel so much more at home in a C project. It is Niri and Rust is mostly an unreadable gibberish to me at this point but you need to start with something. So here goes nothing.

The official Niri project NixOS info is pretty much non-existent. A one liner referring to some non-existent NixGL Github repo. There is also a flake inside Niri repo ( niri/flake.nix at b5640d5293ad8dca06cb447692ea7cbb21680eb1 · YaLTeR/niri · GitHub ) but it is considerably different vs what the Nixpkgs Niri package does. Not even sure which one to go with.

Thank you guys!

7

So I’m trying to dig the nix-shell route as per rhendric’s advice. You are supposed to put together a shell.nix for the project. Is there a way to auto-generate shell.nix out of stock Nixpkgs package? As it already has all the environment and deps in it anyway? Sounds like a reasonable thing to do, generate the shell.nix for a package you want to work on from a bullet proof known to work existing Nixpkgs nix and use that as a starting point, where you can then add the rest of stuff you may need. LSPs and what not? Or asking too much?

One thing that keeps puzzling me time and again why I can’t find any detailed tutorials that will guide through all the steps. Everything I stumble across is always how to set up a dev shell with certain packages in it that will let you use those packages inside the shell. Haven’t found a single one which will show the process of setting up the dev shell to work on a specific existing package, modify sources, incremental builds etc. With trial and error I’ll come up with the most twisted and needlessly complicated setup, thats for sure.

There are like hundreds of resources on the internet and all stop short of just how you e.g. fire a dev shell with a specific python version or specific python packages. But never on how e.g. to work on one of those Python packages to modify it to end up with a tweaked package.

It is as if modifying an existing package sources was the rarest thing on Earth which is hard to believe. I suppose if the original project maintainer were to develop on Nix there may potentially be a ready shell.nix in the project but I have yet to bump into one.

8

Yes, because it’s very project-specific. You’ll have to create subtly different shell.nix files for every project under the sun. Even generating shell.nix-es with LLMs will not generally work well.

You’re basically asking “how do I develop software without having to think”, which is kind of impossible.

Just generating shell.nix isn’t enough to answer your questions, either, because you’re seemingly additionally asking how to use the software you’re installing; that in turn boils down to reading the rust and wayland books, as well as the niri contribution docs, and understanding the intricacies of NixOS and how it interacts with all of that.

All of that is learnable, and docs for everything are out there, but you’ll have to do the work of connecting the dots; your questions are too specific.

Theoretically you can do something like:

{ pkgs ? import <nixpkgs> { }, ... }: pkgs.mkShell {
  inputsFrom = [
    pkgs.niri
  ];
}

… and then add any other packages you need to packages. In practice, this rarely works well, because development setups usually differ a lot from production build setups.

For rust, I basically just copy-paste this to every project:

{ pkgs ? import <nixpkgs> { }, ... }: pkgs.mkShell {
  packages = [
    pkgs.rust-analyzer
    pkgs.rustc
    pkgs.rustfmt
    pkgs.cargo
    pkgs.clippy
  ];
}

Rust is nice like that, as a modern ecosystem it doesn’t have all the insane FHS-hardcoding of ye olden days, so once you learn how to do Rust development most things are samey. At least if nightly isn’t involved.

That said, for niri you will likely need pkgs.pkg-config and some wayland development libs as well since it binds to C libraries. If you try to compile the project it will tell you what’s missing.

Still gotta read their contribution docs to learn how the dev/test flow works afterwards.

9

Thanks TLAILER! Yes, I’m trying to climb a bunch of hills simultaneously here. More in-depth Nix than my usual basic package installing and config, dev shells, rust dev, wayland compositors and what not.

Definitely not HelloWorld grade project I’d normally start anything new from. But there are these few little annoyances in otherwise perfect Niri that drive me mad. I absolutely must make it work the way I want or there will be no rest, it will be nagging me for good. A heavy form of OCD lol. So I have to start with something.

10

Further into my little Niri modding exercise as per above, I now have a modded version which does what I wanted it to. It builds and runs fine in a local nix shell. I also have it in my own Niri fork on GitHub to make it easier to keep with upstream.

Now the only outstanding issue I have is for whatever reason my Niri fork is failing to build when installed via nixpkgs.overlays+overrideAttrs (following Overriding a Version in Wiki) or by overriding programs.niri.package via overrideAttrs as per @TLATER’s suggestion above.

There is clearly some difference between building my fork in a dev shell vs normal nixos-rebuild. Nothing todo with building the project sources. The issue appears to be in the patchPhase build step with nix build trying to substitute shell shebang ‘#!/bin/sh’ in ‘resources/niri.service’ which is obviously not there. Why would it be there, it is a systemd service file after all? …

I’m staring at the logs for a good while now and struggling to make any sense out of it. Nor I can find anything useful online about it. Can you please help guys?

This is my program override in configuration.nix:

    programs.niri =
    {
        enable = true;
        package = pkgs.niri.overrideAttrs (_: 
        {
            src = pkgs.fetchFromGitHub
            { 
                owner = "iamvoidcoder";
                repo = "niri_mods";
                rev = "dbde8f4128fbc49a0823583494cbdabe90b809b6";
                sha256 = "P3z84Zd36gQ89zfpHaoDDSsAMKiHpP6CNq081N5AiGU=";
            };
        });
    };

And this is the build error:

> substituteStream() in derivation niri-25.11: ERROR: pattern /usr/bin doesn't match anything in file 'resources/niri.service'

Full log:

building the system configuration...
error: Cannot build '/nix/store/af5f3n2814j1mzv0y6y3s0iayqkiphsp-niri-25.11.drv'.
       Reason: builder failed with exit code 1.
       Output paths:
         /nix/store/0mp4j0r0bjkib4hls7g98lyrcd4zx6yp-niri-25.11
         /nix/store/4sm1l6nv8d06myw4l2bcxv0kbqxzcjz6-niri-25.11-doc
       Last 10 log lines:
       > Using versionCheckHook
       > Running phase: unpackPhase
       > unpacking source archive /nix/store/sd0q00zgb1d1g28dwzmp65g1iasaps70-source
       > source root is source
       > Executing cargoSetupPostUnpackHook
       > Finished cargoSetupPostUnpackHook
       > Running phase: patchPhase
       > patching script interpreter paths in resources/niri-session
       > resources/niri-session: interpreter directive changed from "#!/bin/sh" to "/nix/store/lw117lsr8d585xs63kx5k233impyrq7q-bash-5.3p3/bin/sh"
       > substituteStream() in derivation niri-25.11: ERROR: pattern /usr/bin doesn't match anything in file 'resources/niri.service'
       For full logs, run:
         nix log /nix/store/af5f3n2814j1mzv0y6y3s0iayqkiphsp-niri-25.11.drv
error: Cannot build '/nix/store/af5f3n2814j1mzv0y6y3s0iayqkiphsp-niri-25.11.drv'.
       Reason: builder failed with exit code 1.
       Output paths:
         /nix/store/0mp4j0r0bjkib4hls7g98lyrcd4zx6yp-niri-25.11
         /nix/store/4sm1l6nv8d06myw4l2bcxv0kbqxzcjz6-niri-25.11-doc
       Last 10 log lines:
       > Using versionCheckHook
       > Running phase: unpackPhase
       > unpacking source archive /nix/store/sd0q00zgb1d1g28dwzmp65g1iasaps70-source
       > source root is source
       > Executing cargoSetupPostUnpackHook
       > Finished cargoSetupPostUnpackHook
       > Running phase: patchPhase
       > patching script interpreter paths in resources/niri-session
       > resources/niri-session: interpreter directive changed from "#!/bin/sh" to "/nix/store/lw117lsr8d585xs63kx5k233impyrq7q-bash-5.3p3/bin/sh"
       > substituteStream() in derivation niri-25.11: ERROR: pattern /usr/bin doesn't match anything in file 'resources/niri.service'
       For full logs, run:
         nix log /nix/store/af5f3n2814j1mzv0y6y3s0iayqkiphsp-niri-25.11.drv
error: Cannot build '/nix/store/h64k3cpmd8liik79vhr0whgmhpdi50i2-desktops.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/4i91a7lags26dvsh0xk06a6zhy1xkgxd-desktops
error: Cannot build '/nix/store/gf1yb6rvkxpgpmp90hb2dhfvjq2dld21-system-generators.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/4npqgqbg74xq4k05pgzmap7mhinq2wcr-system-generators
error: Cannot build '/nix/store/fn37xajhdzkp0bfcqfmz9q3cgcdzlm41-system-path.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/b2bd7pi1nhwiwa3ijyjm1nc8mr9nnyx1-system-path
error: Cannot build '/nix/store/kb3n62y1834z6lvf1f8sy89rjwh7rvvh-nixos-system-nixopc-25.11.20260102.64049ca.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/qr3x90h2in3k0bj94v0jby1ihv26bycl-nixos-system-nixopc-25.11.20260102.64049ca
Command 'nix --extra-experimental-features 'nix-command flakes' build --print-out-paths '/etc/nixos#nixosConfigurations."nixopc".config.system.build.toplevel' --no-link' returned non-zero exit status 1.
11

I’m suspecting it has todo with substituteInPlace in nixpkgs/pkgs/niri/package.nix

postPatch = ''
    patchShebangs resources/niri-session
    substituteInPlace resources/niri.service \
      --replace-fail '/usr/bin' "$out/bin"
  '';

Looks like the intent is to replace ‘/usr/bin’ in ‘resources/niri.service’ with nix store path and fail if the string is not found.

Thing is ‘/usr/bin’ string is actually not present in ‘resources/niri.service‘ file. Which makes it unclear how on earth nixpkgs Niri package.nix builds fine when installed normally via environment.systemPackages or programs.niri.enable?

12

Nixpkgs builds the release version of Niri, which is before this commit which removes the /usr/bin.
You should be able to just set postPatch = ""; in your overrideAttrs call.
Generally, for patching software, I create .patch files with git format-patch and apply them to the package like here:

If you use flakes, I recommend using GitHub - sodiboo/niri-flake: Nix-native configuration for niri. It also sounds like that will be usable without flakes when cleaning up a bunch of tech debt by sodiboo · Pull Request #1548 · sodiboo/niri-flake · GitHub is done.

13

Thanks @bitbloxhub!

Tried stubbing postPatch as suggested and it fixed the binary substitute error.

Now it is failing on .cargoHash mismatch. Asking to set .cargoHash=”” to find the actual hash. Which I do but it is not making any difference, still getting the same error about .cargoHash mismatch and still being asked to set .cargoHash=””.

IDK, modding / overriding packages is proving to be one giant headache :face_with_spiral_eyes: Think I’ll just go back to building from local sources via .src = /local/path instead of .src = fetchFromGitHub and clone the fork manually. At least this way it builds and runs without any issues.

Thanks

14

Yes, with rust things are different because the cargo vendor dir is built differently. .overrideAttrs doesn’t work ootb.

You have to separately override the source for the cargo vendor dir. See this thread: Is it possible to override cargoSha256 in buildRustPackage?