Nixos-install fails building hwdb.bin

Help
1 
# nixos-install --flake 'git+https://git.sr.ht/~zackw/server-configs?dir=nix#tinka'
copying channel...
building the flake in git+https://git.sr.ht/~zackw/server-configs?dir=nix&ref=refs/heads/nix-and-guix&rev=74316117b86b4edc2259b3e51bd70060b260dc7a...
error: builder for '/nix/store/bnya0xz50m7kixgmmcw351b28djf63b4-hwdb.bin.drv' failed with exit code 1;
       last 8 log lines:
       > Adding hwdb files for package /nix/store/d84f8nm2na5cr53m4jk0qk2mj7lgr9fx-systemd-257.9
       > Adding hwdb files for package /nix/store/f040c9mhj9sm2mngzqpbsx6walbpqqnr-ipv6-privacy-extensions.rules
       > Adding hwdb files for package /nix/store/xgnd6bkasz0hb6817dg9jhjqifkd9jcd-ipv6-privacy-extensions.rules
       > Adding hwdb files for package /nix/store/8lnq3lb6mv0xf8vazljmyk5ghra6qmwx-bcache-tools-1.0.8
       > Adding hwdb files for package /nix/store/is5x55rsgcn5dw928b993vhanhd22mcq-extra-udev-rules
       > Adding hwdb files for package /nix/store/cajxzhhk2xv3vvkklx2alik2zdh9kfrw-extra-hwdb-file
       > Adding hwdb files for package /nix/store/mcwrmay5m4z0mbb3sd989jpx9j7rxllz-98-block-io-scheduler.rules
       > Generating hwdb database...
       For full logs, run:
         nix log /nix/store/bnya0xz50m7kixgmmcw351b28djf63b4-hwdb.bin.drv
       note: build failure may have been caused by lack of free disk space
error: 1 dependencies of derivation '/nix/store/9hhvmhi7zvwidyrqi0d6ahyvq95ghv5a-etc.drv' failed to build
error: 1 dependencies of derivation '/nix/store/yvld8yy5ksb4p6vjr8qpgr4ly0yj5qqz-nixos-system-tinka-25.05.20250923.d1d8831.drv' failed to build

It isn’t actually lack of disk space:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         99M     0   99M   0% /dev
tmpfs           986M     0  986M   0% /dev/shm
tmpfs           493M  2.8M  491M   1% /run
tmpfs           986M   27M  960M   3% /
/dev/root       1.7G  1.7G     0 100% /iso
/dev/loop0      1.6G  1.6G     0 100% /nix/.ro-store
tmpfs           3.0G  4.2M  3.0G   1% /nix/.rw-store
overlay         3.0G  4.2M  3.0G   1% /nix/store
tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
tmpfs           986M  1.2M  985M   1% /run/wrappers
tmpfs           1.0M     0  1.0M   0% /run/credentials/getty@tty1.service
tmpfs           1.0M     0  1.0M   0% /run/credentials/serial-getty@ttyS0.service
tmpfs           198M  4.0K  198M   1% /run/user/1000
none            1.0M  336K  688K  33% /mnt
/dev/vda1       254M     0  254M   0% /mnt/boot
/dev/vda3       766M   75M  692M  10% /mnt/nix/scratch
/dev/vda2       940K   52K  746K   7% /mnt/nix/state
tmpfs           198M  4.0K  198M   1% /run/user/0
/dev/vda5       8.0G  4.2G  3.9G  52% /mnt/nix/store

And /mnt/nix/var/log/nix/drvs/bn/ya0xz50m7kixgmmcw351b28djf63b4-hwdb.bin.drv.bz2 contains no additional information. Looking at the code to build this derivation, it seems like an invocation of systemd-hwdb is failing silently. Then I ran the same sequence of commands by hand (as root, not in a chroot), and they all succeeded, including the systemd-hwdb call.

Can anyone suggest how I can get unstuck here?

2

Are you using btrfs anywhere at all?
Also 8 gigs seems shockingly small for the nix store. I’ve struggled with even using 40 gigs in the past and would recommend 100.

Also keep in mind, builds happen in /tmp which is on-disk, and it really looks like you are running out of space.

3

No. Never have, probably never will.

8 gigs seems shockingly small for the nix store. I’ve struggled with even using 40 gigs in the past and would recommend 100.

This VPS only has 24 gigs of storage and I would like to use as much of that as possible for data, thank you.

Local test images indicated only about 2G would be needed for the store, but somehow the space requirement more than doubled when I moved it to the real installation. I plan to tinker a little and see if I can claw back some of the space once I have a working install.

builds happen in /tmp which is on-disk

You wouldn’t happen to know which /tmp that is, would you? There’s at least three possibilities.

4

If you are running very little you can fit your store into ~ 5G, but this is really a how long is a piece of string question. The stuff I am doing is basically running observability (opentelemetry) + discovery (consul) + one service. We (work) also retain no generations, and do a prebake on a larger machine so that 99% of the build is already done with as much pruned as possible s.t. rebuilds are pulling in small delta. It’s possible that if you can get your build done at all you can prune the store quite aggressively back down. Depends on what the live set of roots is.

/tmp is by definition mounted at only one spot (outside of namespaces / chroots), and that’s what is being referenced.

5

That’s just it - this is all running from within the installer CD, so there’s at least two namespaces involved, the installer’s environment and the environment being constructed, plus builder processes get sandboxed so they could have their own /tmp

To put it another way, the directory that appears as /tmp to a builder process, being run from within the installer, could be /tmp, or /mnt/tmp, or some third path I don’t know about, from the perspective of the installation shell.

6

Well in an attempt to make myself more informed and give a more complete answer I decided to deliberately break a build and run it with nix-build -K so I could show you it was under /tmp (I know this wouldn’t answer the installer question, but one thing at a time).

error: could not compile `XXX` (bin "XXX") due to 1 previous error; 4 warnings emitted
note: keeping build directory '/nix/var/nix/builds/nix-666327-2838463246/build'

So not /tmp.

~/src/work/xxx/xxx inline-pkgs +1 !1                                ▼  impure 16:40:35
❯ ll /nix/var/nix/builds/nix-666327-2838463246/build
drwxr-xr-x    - nixbld1 25 Sep 16:38 cargo-vendor-dir
.rw------- 8.0k nixbld1 25 Sep 16:38 env-vars
drwxr-xr-x    - nixbld1 25 Sep 16:38 xxx
.rw-------  111 nixbld1 25 Sep 16:38 tmp.4I0df9OtRn

I am pretty confident tmp.4I0df9OtRn is the private builder tmp.

I had forgotten that 2.30 moves default build-dir because of reasons. So ignore everything I said about builds in tmpdir it’s all out of date, and it happens in /nix. Which doesn’t answer the installer question.

EDIT: having spent 30 seconds reading the installer, the call to nix-build has an undocumented --store (afaict) option that uses the mount point, so I would assume builds happen under /mnt/nix.

7

Thanks, this is helpful. I am doing weird things with /nix/var (kitbashed ephemeral root; not using impermanence) and it’s quite possible that the builds directory wound up in a space that is very small and/or mounted noexec. I have to suspend working on this for the moment but I have some concrete things to try now, at least.