Temporary Writable Directory for Builds

Some package builds need a temporary writable directory, be it because $HOME must be accessible, it relies on some caching or any other reason.

I went looking around on the nixpkgs repo and found at least three different solutions:

  1. mktempe. g., r2modman
  2. $TMPDIRe. g., build-support/go/module.nix
  3. $NIX_BUILD_TOPe. g., hare

Would there be a correct solution?

Regarding the ones above, I’ve managed to found at least an issue in regards to $NIX_BUILD_TOP:

https://github.com/NixOS/nixpkgs/issues/189691#issue-1361062317

However, its stated harmfulness is related to the use of nix-shell, not the build process.

1 Like

mktemp -d is definitely the way to go! Most notably mktemp -d ensures that it’s a fresh directory. So you don’t have to worry about the path potentially being polluted already.

2 Likes

Does Nix do anything to ensure that mktemp -d directories are cleaned up after a build? Or are package authors responsible for putting a trap 'rm -rf $whatever' EXIT in the relevant phase script?

Only the build directory ($NIX_BUILD_TOP, also the same as $TMPDIR) is writable by the build, and it’s definitely being cleaned up after the build :slight_smile:

2 Likes

nix-shell doesn’t have any clean-up phase like nix-build, so I think it’s best if temporary directories are created inside the build directory/current working directory and/or using trap 'rm -rf...' EXIT.

Thanks for the clarification!, @Infinisil.

Do nix-shell and nix-build have the same value for TMPDIR? Because if nix-shell sets it to a tmpfs partition — e. g., /run —, the created directory would be cleaned up after a reboot and, thus, there would be no need for the trap call.

Sorry for digging this up, but it still seems like the most relevant issue, and I’ve been thinking about how to deal with this.

It looks like nix-shell respects TMPDIR and if it’s set, will use it for both TMPDIR and NIX_BUILD_TOP inside the shell.

nix develop goes a step further and invokes the shell with a fresh mktemp -d set as both TMPDIR and NIX_BUILD_TOP, which it cleans up when the shell exits.

I don’t think traps are really an option because you need some way to compose them, and stdenv already sets an EXIT trap. It does expose failureHook and exitHook, but only calls one or the other, and I don’t see anything in nixpkgs using these for cleanup.

Based on all that, I think the best option is just to use mktemp and either use nix develop, or be careful about the TMPDIR used with nix-shell.

Expecting NIX_BUILD_TOP (or TMPDIR) to be empty seems like a bad idea, given how nix-shell works. So I would avoid doing things like:

            install -m 0600 /dev/null "$NIX_BUILD_TOP/env-vars" &&
            export 2>/dev/null >| "$NIX_BUILD_TOP/env-vars"
    PGDATA="$NIX_BUILD_TOP/postgresql"

Unfortunately there are tons of places that do this.

Edit: Actually I was wrong about nix develop cleaning up the temp dir. It doesn’t seem to, at least not in all cases.

I don’t have the best sense of whether working around trap’s limitations here is enough to meet your needs, but here’s a post showing how to work around this with a trap-shimming shell library I wrote:

Can someone explain how the hook writable-tmpdir-as-home.sh relates to this problem?

As per the docs:

This setup hook ensures that the directory specified by the HOME environment variable is writable. If it is not, the hook assigns HOME to a writable directory (in .home in $NIX_BUILD_TOP). This adjustment is necessary for certain packages that require write access to a home directory. This hook can be added to any phase.

By setting HOME to a writable directory, this setup hook prevents failures in packages that attempt to write to the home directory.

… it seems to be meant for this purpose, though I have yet to figure out to actually run it inside my mkDerivation, which needs a writable HOME, too.

I am very confused.

Just add writableTmpDirAsHomeHook to your nativeBuildInputs.

Demo:

$ cat $(nix-build --expr 'let inherit (import <nixpkgs> { }) stdenv writableTmpDirAsHomeHook; in stdenv.mkDerivation { name = "test"; nativeBuildInputs = [ writableTmpDirAsHomeHook ]; phases = [ "installPhase" ]; installPhase = "ls -ld $HOME > $out"; }' --no-out-link)
drwxr-xr-x 2 nixbld nixbld 40 Mar 26 00:04 /build/.home
2 Likes

You just need to add writableTmpDirAsHomeHook to the
nativeBuildInputs attribute of your derivation.

1 Like

thank you so much @rhendric and @onemoresuza for taking the time to help me along :pray:!

Does using writableTmpDirAsHomeHook instead of “manually” mktemping as mentioned above also make sense to you?

I mean I guess the difference is academic/tiny, but I like the idea of using an idiomatic way to deal with the home directory problem.

I mean I guess the difference is academic/tiny, but I like the idea of
using an idiomatic way to deal with the home directory problem.

I’d also add that using the idiomatic way makes more clear for users
what the derivation needs and what it is doing: if
writableTmpDirAsHomeHook is present a nativeBuildInputs, one can
quickly realize that the derivation is using $HOME and, therefore,
needs it to be writable.

1 Like

Glad to see this thing being used :slight_smile:

1 Like

since I had such trouble understanding how to use package setup hooks in general (including writable home dir), I logged this ticket to improve the docs.

Please chime in if you can (or close the ticket if I’m just being unreasonable dense :sweat_smile:).

1 Like