Permanently use a sandbox build directory outside of /tmp?

johanneskastl · February 15, 2025, 3:50pm

Dear all,

I noticed building my home-manager configuration on one of my machines fails due to a “no space left on device” errror.

Took me a while to realize, that there is plenty of space in /nix, but not enough in /tmp, where the sandbox build directory seems to be created.

       > rustc-LLVM ERROR: IO failure on output stream: No space left on device
       > error: could not compile `util` (lib)
       > warning: build failed, waiting for other jobs to finish...
       > error: failed to write bytecode to /build/source/target/x86_64-unknown-linux-gnu/release/deps/language-c16d310478aa5fb8.language.eef0e8f1ec19b467-cgu.0.rcgu.b
c: No space left on device (os error 28)

(Why it builds this package and does not just download it is another story).

As far as I can see, the build directory is placed in /tmp, which only has 4GB of free space due to being a tmpfs.

Question:
Is there a way to permanently put the sandbox and its build directory somewhere else? The btrfs partition where /nix and /var/home/ are located has 190GB free…
As this machine only has 8GB of RAM, a tmpfs might simply not be big enough.

I checked the manpage for nix.conf, but found no hint. Or maybe I am just overlooking it.

Thanks in advance!

Kind Regards,
Johannes

johanneskastl · February 15, 2025, 4:03pm

Hmm, I found build-dir which might be the right switch.

Setting it to /var/tmp in /etc/nix/nix.conf and testing it currently…

TLATER · February 16, 2025, 8:32am

Fyi, this is why NixOS normally doesn’t make /tmp a tmpfs.

Does feel a bit backwards, though, it’d make more sense to just give nix a separate build directory. Never thought this hard about why it’s done this way.

ElvishJerricco · February 16, 2025, 8:57am

It’s done this way because the most obvious way to get a temporary directory is to put it in $TMPDIR falling back to /tmp. Unfortunately, that is very much the wrong fallback for something like a Nix build, and it ought to default to /var/tmp, per the subset of the Filesystem Hierarch Standard that NixOS does adhere to.

johanneskastl · February 23, 2025, 5:17pm

Sorry, forgot to report back.

Just setting the build-dir in /etc/nix/nix.conf does not work, at least not with home-manager.

I need to use TMPDIR=/var/tmp/ home-manager ... for it to use /var/tmp/ instead of /tmp. Looking for a way to configure this in the home-manager configuration, so I do not need to remember this all the time…

johanneskastl · March 3, 2025, 12:23pm

Opened a bug report at home-manager to clarify why the TMPDIR=/var/tmp/ is needed…

github.com/nix-community/home-manager

bug: home-manager not respecting build-dir when set in /etc/nix/nix.conf?

opened 10:09AM - 02 Mar 25 UTC

johanneskastl

bug triage

### Are you following the right branch? - [x] My Nixpkgs and Home Manager versi…ons are in sync ### Is there an existing issue for this? - [x] I have searched the existing issues ### Issue description Dear maintainers, as discussed in https://discourse.nixos.org/t/permanently-use-a-sandbox-build-directory-outside-of-tmp/60353/, I am using home-manager on a Fedora-based device with only 8GB of RAM. Hence the `tmpfs` in `/tmp` is too small, while `/var/tmp/` has enough space. Setting `build-dir` in `/etc/nix/nix.conf` (or `/etc/nix/nix.custom.conf`) shows this setting when running `nix config show`: ``` $ nix config show|grep build-dir build-dir = /var/tmp sandbox-build-dir = /build ``` (As far as I found this obseletes setting the TMPDIR variable in the nix-daemon systemd unit file) However, when using `home-manager`, I always need to prefix the command with `TMPDIR=/var/tmp/`, otherwise the build fails due to missing space in `/tmp`. ``` # working $ nix-channel --update && TMPDIR=/var/tmp/ home-manager switch # failing $ nix-channel --update && home-manager switch $ ``` I searched a lot on how to set this in the home-manager configuration itself somewhere, but of course those are lousy search terms so I might just have missed it. Is there an option that I need to set so this is being respected? Or is this a bug that needs to be fixed? Thanks in advance! Kind Regards, Johannes ### Maintainer CC _No response_ ### System information ```markdown - system: `"x86_64-linux"` - host os: `Linux 6.12.10-200.fc41.x86_64, Aurora, 41.20250302 (Kinoite), ddd565a` - multi-user?: `yes` - sandbox: `yes` - version: `nix-env (Nix) 2.26.1` - nixpkgs: `/nix/store/w3ghsxcwcynh88fgr7j841drw08fyzn0-nixpkgs-24.11/nixpkgs` ```

doronbehar · March 3, 2025, 12:33pm

Probably relevant:

github.com/NixOS/nix

Running Nix as root ignores nix-damon's TMPDIR

opened 02:26PM - 10 Oct 22 UTC

Atemu

bug

**Describe the bug** A clear and concise description of what the bug is. N…ix builds ignore the `nix-daemon`'s TMPDIR environment variable when run as root. **Steps To Reproduce** 1. `TMPDIR=/var/tmp nix-daemon` 2. `nix-build '<nixpkgs>' -A hello --check` 3. `sudo nix-build '<nixpkgs>' -A hello --check` **Expected behavior** A clear and concise description of what you expected to happen. Both builds should run in `/var/tmp`. **`nix-env --version` output** nix-env (Nix) 2.11.0 **Additional context** Add any other context about the problem here. This happens all the time when you try to run `nixos-rebuild` because that command obviously needs to run as root. The kernel build notably uses more space than most people have space for in /tmp thanks to BTF. `sudo nixos-rebuild switch` with a custom kernel build is no bueno by default for the vast majority of systems. That's not a good UX. Honestly, I don't see why `/tmp` should be the default. Sure, it can be a useful optimisation if you know your builds are all tiny but that's an assumption you cannot make for all builds in general. `/var/tmp` is on a physical disk by default and therefore makes a much better default TMPDIR.

johanneskastl · March 4, 2025, 5:33pm

Thanks for the link. I have no clue how the build-dir is mapped internally. But as it replaces the “old” way of setting a TMPDIR in the nix-daemon unit file, I am not sure if this bug still applies…

johanneskastl · March 4, 2025, 5:34pm

Also, I am not running home-manager as root user. I am calling it as my normal unprivileged user.