Is it possible to edit the result of a mkDerivation, considering immutability?

Help
1

This is a generic question, but lets imagine a scenario where: we have some software which is built from source using mkDerivation; the build is placed in results/ with an executable binary and the source of the package. Now, for some reason, to rapidly test a change, we 1) need to edit part of the source (say a script file which the binary reads and executes directly), and 2) without rebuilding the package, which would take too long (or because of some other obscure reason), just run the binary to see the changes. Is this possible? What would be the proper development approach, considering NixOS package builds are immutable? Maybe there is some middle way, or a simple solution I am not seeing?

2

For rapid iteration on the software itself, I think it’s much more convenient to enter a development shell and use the project’s native build tools directly, than it is to build another Nix derivation for every change.

2 Likes
3

If the build output is immutable, then by definition you can’t edit the result. :stuck_out_tongue:

But you can actually remount /nix/store read-write then have your way with any store path. This voids your warranty. I wouldn’t recommend doing it as a habit.

If you’re trying repeatedly to make a package which successfully builds, then the nix-build --keep-failed option can help. But I don’t think this is what you’re asking.

I use the nix develop --redirect option sometimes. This lets you substitute a build dependency with a local directory while developing. But this probably isn’t quite what you’re asking.

I would probably add some “hooks” in the source so that the path to this shell script can be overridden at runtime. Prepend an environment variable like DEBUG_MY_SCRIPTS_PATH to your script’s PATH.

Try to write your scripts so that they can be run just as well without nix. This will let you debug them better in isolation. Strategies such as avoiding pkgs.writeScript and instead using substituteAllInPlace or wrapProgram can help with that.

1 Like
4

I wouldn’t recommend doing this at all if you don’t know what you’re doing. Source: I did this once when I didn’t know what I was doing. It didn’t quite lead to shark attacks but it was close. A corrupted store is no joke.

5

Also, if you know you are going to experiment editing a specific set of scripts, you can make a package version that replaces those scripts with symlinks elsewhere (and maybe even wrap the binary into a wrapper that deploys the default versions if they are not yet there).

Also, there are some black magic tools to copy the path and mostly rewrite self-references and apply some other changes…

6

Hello all, thanks for the replies :wink:

Maybe lets provide a more concrete example, on which I am currently working.

I am developing a software called magic-vlsi, which I can build from source. I have written a simple mkDerivation script, which fetches it from github, and builds it:

# default.nix
{
  pkgs ? import <nixpkgs> { }
}:
{
  magic = pkgs.callPackage ./magic.nix {   };
}

# magic.nix
{ lib
, stdenv
, fetchFromGitHub
, tcl
, git
, cairo
, tcsh
, python3
, tk
, m4
, ncurses
, libGLU
  # undocumented packages required
, freeglut
, mesa
}:
stdenv.mkDerivation {
  name = "magic";

  buildInputs = [
    tcl
    git
    cairo
    tcsh
    python3
    tk
    m4
    ncurses
    libGLU
    # undocumented packages required
    freeglut
    mesa
  ];

  makeFlags = [];

  configureFlags = [
    "--with-tcl=${tcl}/lib/"
    "--with-tk=${tk}/lib/"
  ];

  src = fetchFromGitHub {
    owner = "RTimothyEdwards";
    repo = "magic";
    rev = "8.3.440";
    sha256 = "1c94m6pcbvs3ppvrkbbccwh30lmyssvzm6hylf4jwrbyqll078sg";
  };
}

So, some of the solutions rely on flakes. Are flakes in general better for development? Moreover, what if some later package (lets say, pack_A) depends on magic, I make small change to magic, and want to test the change reflected in pack_A. What would be the best approach, to build this development environment, so that I do not need to rebuild the packages? I have done this in classic FHS-compliant linux, where I install magic and pack_A from source, and then just edit some files in /usr/share and all works quite fast. But I am not sure how to approach this in NixOS…

7

Keep it simple. If you want an environment where you can build both magic and pack_A, the simplest thing is to run nix-shell -p <long list of build inputs from both packages>. Do everything except for installation the same way you’d do on any other Linux. Install magic to a temp directory if you must, and then configure pack_A to point to that temp directory. If you don’t have to install magic, maybe you can configure pack_A to point to magic’s build directory and get rapider iteration that way. Save worrying about how to package all of this into Nix for the very end, once you’re no longer iterating on the projects themselves.

If you want something more sophisticated than keeping nix-shell -p <long list of build inputs from both packages> in your command history or in a script file somewhere, you can make a .nix file like this:

with import <nixpkgs> { };
mkShell {
  buildInputs =
    magic.buildInputs ++
    builtins.filter (p: p.pname != "magic") pack_A.buildInputs;
}

and call that with nix-shell. (That assumes you’re getting both magic and pack_A from Nixpkgs, which maybe you’re not; in that case, you’ll be importing or callPackageing some additional Nix files to get those references.)

Here’s the deal with flakes: if you want to pin the version of Nixpkgs that this file uses as an input, flakes may be helpful, if you’re willing to wade through the user experience trash fire that I think they are. I rarely bother but a lot of people seem to think they’re the best, so form your own opinion. But you certainly don’t need them, not even to pin to a particular version of Nixpkgs; for that, you can run nix-shell whatever.nix -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/some-particular-commit-sha.tar.gz and you’ll be using that version until you change it.

But all any of that is doing is still just getting you into a development shell where you have all your dependencies and build tools. Once you’re there, just do boring normal things and forget about Nix until it’s time to package up what you’re doing.

1 Like
8

Thanks @rhendric, you are right about keeping it simple. I used nix-shell as you advised to build and install all the software and it works well! :wink: