Unable to add a file to $out for a package

I’m trying to add a script ("qute-readlater", see below) to the qutebrowser package using a nixpkgs overlay:

final: prev:
{
  qutebrowser = prev.qutebrowser.overrideAttrs (old: let
    quteReadLater = ./qute-readlater;
  in
    {
      postInstall = ''
      cp  ${quteReadLater} "$out/share/qutebrowser/userscripts/qute-readlater"
      '' + old.postInstall;
    });
}

But when I try to build qutebrowser it fails with the error:

> cp: cannot create regular file '/nix/store/kvx8024gj7yb2pfvg5fh0c01n28glifg-qutebrowser-2.5.4/share/qutebrowser/userscripts/qute-readlater': Permission denied

Am I going about this the wrong way? Is it not allowed to add files to the $out directory during a build?


Edit: I just realized all files in the nix store (where $out points to) have 444 or 555 permission, so it makes sense that you can’t copy a file to a directory there. Then what is the correct way to package qutebrowser together with a custom script (additional file)?

Try tapping into the postUnpack phase. It runs after the source code has been extracted, so you can copy the script into ./misc/userscripts/. It should then be treated like any other bundled userscript. See qutebrowser/misc/userscripts at master · qutebrowser/qutebrowser · GitHub

I changed the postInstall part to

      postUnpack = ''
        cp ${quteReadLater} "./misc/userscripts/qute-readlater"
      '';

This returns:

> cp: cannot create regular file './misc/userscripts/qute-readlater': No such file or directory

I checked with nix develop 'nixkpgs#qutebrowser' + unpackPhase that the misc/userscripts directory really does exist. So I’m not sure what’s causing the above problem. Does the postUnpack run from inside the unpacked src archive?

Ah, check this out. This is an example from Nixpkgs:

 postUnpack = ''
    rmdir source/src/phazor/kissfft
    ln -s ${kissfft.src} source/src/phazor/kissfft

    rmdir source/src/phazor/miniaudio
    ln -s ${miniaudio.src} source/src/phazor/miniaudio
  '';

So it looks like the source code is extracted into a subdirectory named source. The directory should probably be source/misc/userscripts.

That doesn’t seem to be it either. Here’s the error with more context:

error: builder for '/nix/store/1i47sl89dm9lxargl4q9q741nck7qfqr-qutebrowser-2.5.4.drv' failed with exit code 1;
       last 10 log lines:
       > Sourcing python-imports-check-hook.sh
       > Using pythonImportsCheckPhase
       > Sourcing python-namespaces-hook
       > Sourcing python-catch-conflicts-hook.sh
       > qtPreHook
       > unpacking sources
       > unpacking source archive /nix/store/klbrs2cly0jwg5zh8ki046yh3xnn3pd3-qutebrowser-2.5.4.tar.gz
       > source root is qutebrowser-2.5.4
       > cp: cannot create regular file 'source/misc/userscripts/qute-readlater': No such file or directory

Okay, here’s what worked:

      postUnpack = ''
        cp ${quteReadLater} "${old.pname}-${old.version}/misc/userscripts/qute-readlater"
      '';

Qutebrowser built successfully, but I still have lots of questions about the build process.

  1. Why did cp ${quteReadLater} "$out/share/..." fail with a permission denied error?
  2. Why did running this cp in the postUnpack hook instead work fine?
  3. What directory does the postUnpack hook canonically run from? Shouldn’t it run from the source root directory?
  4. Is there a canonical way to refer to the source directory? The qutebrowser-2.5.4 format might break in the future.
  5. Why is the unpack directory named source in @emmanuelrosa’s example, but qutebrowser-2.5.4 in my case? Isn’t the unpackPhase a standardized operation?

Weird, this works just fine for me. What OS, Nix version and Nixpkgs revision do you use?

It runs from the /build directory inside the sandbox. The main body of unpackPhase sets sourceRoot variable but it is only cd’d into at the end of unpackPhase (after postUnpack).

$sourceRoot in postUnpack or later.

It depends on what is in the src attribute. fetchgit and derived fetchers like fetchFromGitHub use source, while archives fetched with fetchurl containing a single directory will just use the directory name as is.

Answering my questions for future readers:

Why did cp ${quteReadLater} "$out/share/..." fail with a permission denied error?

I don’t know yet.

Why did running this cp in the postUnpack hook instead work fine?

The file mode for the unpacked archive/directories has not yet been set. This makes sense since otherwise you couldn’t edit the source directory in any way, including by calling Make or patching, etc.

Is there a canonical way to refer to the source directory? The qutebrowser-2.5.4 format might break in the future.

From the unpackPhase function in stdenv.setup:

if [ -n "${setSourceRoot:-}" ]; then
    runOneHook setSourceRoot
elif [ -z "$sourceRoot" ]; then
    for i in *; do
        if [ -d "$i" ]; then
            case $dirsBefore in
                *\ $i\ *)
                    ;;
                *)
                    if [ -n "$sourceRoot" ]; then
                        echo "unpacker produced multiple directories"
                        exit 1
                    fi
                    sourceRoot="$i"
                    ;;
            esac
        fi
    done
fi

If a single new directory is produced (as in the qutebrowser case), it is set to the variable sourceRoot.

What directory does the postUnpack hook canonically run from? Shouldn’t it run from the source root directory?

The answer to this is in the stdenv.setup shell script of nixpkgs. Specifically in the unpackPhase and genericBuild shell functions defined in this script.

Specifically, in genericBuild we have

eval "${!curPhase:-$curPhase}" #eval each phase (including unpackPhase)

# omitted lines

        if [ "$curPhase" = unpackPhase ]; then
            # make sure we can cd into the directory
            [ -z "${sourceRoot}" ] || chmod +x "${sourceRoot}"

            cd "${sourceRoot:-.}"
        fi

So after unpackPhase + postUnpack hook runs, evaluation moves into the sourceRoot directory, which is where the files have been unpacked to.

Why is the unpack directory named source in @emmanuelrosa’s example, but qutebrowser-2.5.4 in my case? Isn’t the unpackPhase a standardized operation?

It is not standardized, except in that if the unpackPhase produces a single dir, the variable sourceRoot is set to that directory (relative file path).

nixos-version:
23.05.20230616.c7ff1b9 (Stoat)

nix --version:
nix (Nix) 2.13.3

nixpkgs (from flake.lock):
    "nixpkgs": {
      "locked": {
        "lastModified": 1686921029,
        "narHash": "sha256-J1bX9plPCFhTSh6E3TWn9XSxggBh/zDD4xigyaIQBy8=",
        "owner": "nixos",
        "repo": "nixpkgs",
        "rev": "c7ff1b9b95620ce8728c0d7bd501c458e6da9e04",
        "type": "github"
      },
      "original": {
        "owner": "nixos",
        "ref": "nixos-23.05",
        "repo": "nixpkgs",
        "type": "github"
      }
    },

Weird, does not look like there are any differences between the your commit and nixos-unstable I tried. We use the same Nix version as well. Maybe you are target of a curse or something.

Does the following give you any useful info?

final: prev:
{
  qutebrowser = prev.qutebrowser.overrideAttrs (old: let
      quteReadLater = ./qute-readlater;
    in
      {
        postInstall = ''
        ls -la "${quteReadLater}"
        ls -la "$out/share/qutebrowser/userscripts"
        cp  ${quteReadLater} "$out/share/qutebrowser/userscripts/qute-readlater"
        '' + old.postInstall;
      });
}

Here’s what I see:

-r-xr-xr-x 2 nobody nogroup 5030 Jan  1  1970 /nix/store/a1b3kzyl19xh1dvi4vz153cj5vgfn02f-qute-readlater
total 196
drwxr-xr-x 2 nixbld nixbld  4096 Jun 19 21:40 .
drwxr-xr-x 4 nixbld nixbld  4096 Jun 19 21:40 ..
-rwxr-xr-x 1 nixbld nixbld  5606 Jun 19 21:40 README.md
# more output omitted

line 1 is the qute-readlater userscript, lines 2 and later is the output of ls -la $out/share/qutebrowser/userscripts.

I don’t see a problem with the permissions for the nixbld user to copy the file in line 1 into the directory listed in line 2. What gives?

Yeah, that looks fine. No idea what is up with it.

It gets stranger. I created a separate file qutebrowser-mine.nix with just the qutebrowser derivation:

let
  pkgs = import <nixpkgs> { overlays = [
    (final: prev:
      {
        qutebrowser = prev.qutebrowser.overrideAttrs (old: let
          quteReadLater = ./qute-readlater;
        in
          {
            postInstall = ''
      cp  ${quteReadLater} "$out/share/qutebrowser/userscripts/qute-readlater"
      '' + old.postInstall;
          });
      }) ]; };
in
pkgs.qutebrowser

And ran nix-build qutebrowser-mine.nix - it built fine. But for some reason it fails when I build it from my home-manager flake.