How to avoid nix.gc deleting so many .drv?

Atemu · August 22, 2022, 7:30am

Okay, that’s slow but not terribly slow.

The HDD is going to have a much greater effect on speed.

That should do it. With such little RAM, I’d recommend setting it to a high percentage (maybe 80-90% even; at least 75%). I’d also recommend lz4 for a slower system where zram swap will be used more often.

Do you have disk swap enabled? If so, I’d recommend disabling that. If you’re swapping to a HDD, that could very well cause simple workloads to take hours instead of minutes.

If swap wasn’t there, that must have been edging very close to the limit. What is memory usage like during a rebuild when there is no swap?

= 18Mbit/s

Slower than your tested speed but not unreasonably slow either.

135546345 · August 23, 2022, 4:30am

I just realized that the above level of resource usage are so high because in the background firefox was open, now I tested to download superTuxKart without swap and without firefox, the memory usage was 1.9GiB and when the rebuild finished it went down to 1.3GiB, what took the longest was the download.

[sudo] password for root: 
[23:43:52] building Nix...
[23:43:55] building the system configuration...
[23:45:08] these 13 derivations will be built:
[23:45:08]   /nix/store/0khggj3f1zssdb4xdbzhbhr9yzi3h7i9-system-path.drv
[23:45:08]   /nix/store/2x9af7g50pf66wb9nv4wqgrmnl1c5ii8-unit-accounts-daemon.service.drv
[23:45:08]   /nix/store/qp44s2glc8grf8gdzhc9ybpz6gkqx6mb-unit-polkit.service.drv
[23:45:08]   /nix/store/xp2h4k5hjnxwrk88dhdfrd60i0isnqc5-unit-systemd-fsck-.service.drv
[23:45:08]   /nix/store/164lc4q8cka7s3la903db9ck97pscxmb-dbus-1.drv
[23:45:08]   /nix/store/xpzkjr01k90c67ijvip2351mwvspm1jh-unit-dbus.service.drv
[23:45:08]   /nix/store/033kkjc67l4xjc0nrw7wgcs5nhz4g14c-system-units.drv
[23:45:08]   /nix/store/gq7krsi9cjjghak7dw0c88qdak01xrfn-system-crontab.drv
[23:45:08]   /nix/store/7wlb4436lrqxpdaqgl4b5fpqmrssia1f-crontabs.drv
[23:45:08]   /nix/store/8v0pzqz1pkwni48rc8xjnx9l9mkzr8z1-unit-dbus.service.drv
[23:45:08]   /nix/store/wa3d86ryg38v6fqf0c6v78kb0c7g794v-user-units.drv
[23:45:08]   /nix/store/axifxsizrsj5jhqsxfphr5h1cac5nypr-etc.drv
[23:45:08]   /nix/store/mgpxzhj29gy3d24ssdw71rz6jwck6rk9-nixos-system-nixos-22.05.2600.23534df34c1.drv
[23:45:08] these 6 paths will be fetched (620.18 MiB download, 743.86 MiB unpacked):
[23:45:08]   /nix/store/7zk47irvprynj02sbgm1rfa77cklb2zw-angelscript-2.35.1
[23:45:08]   /nix/store/81flppby8shis39f1h9mjrjm8hhqdggc-bluez-5.64-dev
[23:45:08]   /nix/store/f1bvlhphj0ffqdhwvb5iclfic0a7qpzf-stk-assets
[23:45:08]   /nix/store/knf98zrlhr3p7ccfkxdamri453bf7smw-libopenglrecorder-unstable-2020-08-13
[23:45:08]   /nix/store/p1jk8d1l0rkm141ybic69kqh6q84a3zy-WiiUse-0.15.5
[23:45:08]   /nix/store/qxmg4bkcshmjx0xpp9brgc999fjwxvgf-supertuxkart-1.3
[23:45:10] copying path '/nix/store/f1bvlhphj0ffqdhwvb5iclfic0a7qpzf-stk-assets' from 'https://cache.nixos.org'...
[23:45:10] copying path '/nix/store/7zk47irvprynj02sbgm1rfa77cklb2zw-angelscript-2.35.1' from 'https://cache.nixos.org'...
[23:45:10] copying path '/nix/store/knf98zrlhr3p7ccfkxdamri453bf7smw-libopenglrecorder-unstable-2020-08-13' from 'https://cache.nixos.org'...
[23:45:10] copying path '/nix/store/81flppby8shis39f1h9mjrjm8hhqdggc-bluez-5.64-dev' from 'https://cache.nixos.org'...
[23:45:10] copying path '/nix/store/p1jk8d1l0rkm141ybic69kqh6q84a3zy-WiiUse-0.15.5' from 'https://cache.nixos.org'...
[00:04:16] copying path '/nix/store/qxmg4bkcshmjx0xpp9brgc999fjwxvgf-supertuxkart-1.3' from 'https://cache.nixos.org'...
[00:05:06] building '/nix/store/0khggj3f1zssdb4xdbzhbhr9yzi3h7i9-system-path.drv'...
[00:06:45] warning: collision between `/nix/store/mg8b6z8wia9n58j4mmkxyjrdawb1h2m7-emacsclient.desktop/share/applications/emacsclient.desktop' and `/nix/store/b4s94xc5hkks8jrmbdd5js4s338hlxi0-emacs-with-packages-28.1/share/applications/emacsclient.desktop'
[00:06:53] created 31607 symlinks in user environment
[00:06:58] gtk-update-icon-cache: Cache file created successfully.
[00:08:22] building '/nix/store/164lc4q8cka7s3la903db9ck97pscxmb-dbus-1.drv'...
[00:08:22] building '/nix/store/gq7krsi9cjjghak7dw0c88qdak01xrfn-system-crontab.drv'...
[00:08:23] building '/nix/store/2x9af7g50pf66wb9nv4wqgrmnl1c5ii8-unit-accounts-daemon.service.drv'...
[00:08:23] building '/nix/store/qp44s2glc8grf8gdzhc9ybpz6gkqx6mb-unit-polkit.service.drv'...
[00:08:23] building '/nix/store/xp2h4k5hjnxwrk88dhdfrd60i0isnqc5-unit-systemd-fsck-.service.drv'...
[00:08:23] building '/nix/store/7wlb4436lrqxpdaqgl4b5fpqmrssia1f-crontabs.drv'...
[00:08:24] building '/nix/store/8v0pzqz1pkwni48rc8xjnx9l9mkzr8z1-unit-dbus.service.drv'...
[00:08:24] building '/nix/store/xpzkjr01k90c67ijvip2351mwvspm1jh-unit-dbus.service.drv'...
[00:08:24] building '/nix/store/wa3d86ryg38v6fqf0c6v78kb0c7g794v-user-units.drv'...
[00:08:26] building '/nix/store/033kkjc67l4xjc0nrw7wgcs5nhz4g14c-system-units.drv'...
[00:08:29] building '/nix/store/axifxsizrsj5jhqsxfphr5h1cac5nypr-etc.drv'...
[00:08:30] building '/nix/store/mgpxzhj29gy3d24ssdw71rz6jwck6rk9-nixos-system-nixos-22.05.2600.23534df34c1.drv'...
[00:09:05] leaving systemd-boot 250.4 in place (250.4 is not newer)
[00:09:09] stopping the following units: accounts-daemon.service
[00:09:09] NOT restarting the following changed units: systemd-fsck@dev-disk-by\x2duuid-91D3\x2dE366.service
[00:09:09] activating the configuration...
[00:09:10] setting up /etc...
[00:09:14] reloading user units for user...
[00:09:16] setting up tmpfiles
[00:09:16] reloading the following units: dbus.service
[00:09:17] restarting the following units: polkit.service
[00:09:18] starting the following units: accounts-daemon.service
[00:09:20] the following new units were started: \x2eswapfile.swap, dev-zram0.swap, sysstat-collect.service

Like this?

  zramSwap = {
    enable = true;
    algorithm = "lz4";
    memoryPercent = 85;
  };

nixinator · August 23, 2022, 10:50am

maybe think about using remote builders if your lack resources.

the kind people at nixbuild.net could probably help you, so you don’t melt your laptop.

Atemu · August 23, 2022, 11:20am

That looks good.

620.18 MiB/15min ~= 5.8mbit/s

Now we need to figure out whether it’s your internet or your disk. It’s probably the former.

Monitor download speed in real time while Nix is substituting something.

Their problem is binary substitution, not builds.

135546345 · August 24, 2022, 7:15pm

Substituting means installing a package?

NobbZ · August 24, 2022, 8:18pm

“Substitution” is the process of downloading a cached artefact instead of building it.

135546345 · August 28, 2022, 8:20pm

I’m reading the man page of nixos-rebuild but i can’t understand it very well, to make nixos-rebuild download but not build what command i have to use? --no-build-nix?

Atemu · August 28, 2022, 9:05pm

I don’t think you can make it not build but if you were to build a simple system with very few customisations, there would be very little to build afterwards and the time and resources that’d take are insignificant.

NobbZ · August 29, 2022, 7:22am

--max-jobs 0/--option max-jobs 0 will disable local builds. Usually this is used to check if remote/distributed builders work, though you can also use it to debug caching or enforce use of it.

This is very likely to break, as there is some kind of derivations, especially created by the system configuration, that are thus trivial to build, that they are marked as “local builds”. With setting the max jobs to 0, you disable even those.

Though, why do you want to disable local builds at all? They are not relevant to debugging the potential network problems, and if a local build happens already then I’m pretty sure that would be visible in the logs.

135546345 · August 29, 2022, 4:35pm

I misspoke, how do I make a “substitution”?

NobbZ · August 29, 2022, 4:38pm

Nix does those automatically.

A nix expression is evaluated and for all derivations that needs to be built to fully realize it the following happens in order:

Check if already in store, dont do anything if it is.
is it marked as local build? → build
check if available in cache? → download
build

135546345 · August 29, 2022, 4:51pm

Ok, so how do i do this?

Atemu · August 29, 2022, 10:39pm

Use a network monitoring tool like iftop. I personally like btop.

Also conduct additional tests while the download is running. Start another download from somewhere else or the cache for example and see what happens.

135546345 · August 31, 2022, 12:01am

Downloading brave with an average of 150KiB/s

[20:38:17] building Nix...
[20:38:19] building the system configuration...
[20:38:26] these 13 derivations will be built:
[20:38:26]   /nix/store/bqrbxfv57777d82vbar57nvzcazzjv24-system-path.drv
[20:38:26]   /nix/store/9vsgjhqc5hjbrhmfbbar623gyxjmbwd6-unit-systemd-fsck-.service.drv
[20:38:26]   /nix/store/hm4g3r3mnpcq015hqsf69qrgfp3iyznl-unit-accounts-daemon.service.drv
[20:38:26]   /nix/store/5wz297vb31z23ya1s50q70nr45pb1jd6-dbus-1.drv
[20:38:26]   /nix/store/ydich8ppv5xgmzbvkha2gm4zkxcgh17a-unit-dbus.service.drv
[20:38:26]   /nix/store/yf1kjspllccinmbppb1xpq1f9wrf23kl-unit-polkit.service.drv
[20:38:26]   /nix/store/5dz37gh8h5lnz8qdr4mdpxjj772aad34-system-units.drv
[20:38:26]   /nix/store/cx3ix0zb9icpaqn36a6hazzg2nmslq3h-system-crontab.drv
[20:38:26]   /nix/store/ai8y3616k0mv7mv8zxk1w2dqvrw8dp26-crontabs.drv
[20:38:26]   /nix/store/l4m7r7chb61gxc6q3wfbk2lkbkwdql0y-unit-dbus.service.drv
[20:38:26]   /nix/store/gypylgccbaknhp9mk8sw7nbbzsjnnvqh-user-units.drv
[20:38:26]   /nix/store/rrzqfwdnh1i37pkvgr55x3zd016245c5-etc.drv
[20:38:26]   /nix/store/r9sigfnjxmcga1gd2ysgn1509iwy5d7s-nixos-system-nixos-22.05.2785.cc634d9aa08.drv
[20:38:26] these 4 paths will be fetched (95.41 MiB download, 280.40 MiB unpacked):
[20:38:26]   /nix/store/ikb7a6ii4qsq3wkjhy9sapzn23xbiwj9-wayland-1.20.0-bin
[20:38:26]   /nix/store/mnyyr92l9bhacnxvkfacig6iszna933q-brave-1.38.115
[20:38:26]   /nix/store/n5bgxkb9wwhvmq4qkh6hxmrhj7zrlz2d-pango-1.50.7-bin
[20:38:26]   /nix/store/vyqadlg9wn7v0hjymzvg00wa99fr95ca-libdrm-2.4.110-bin
[20:38:26] copying path '/nix/store/vyqadlg9wn7v0hjymzvg00wa99fr95ca-libdrm-2.4.110-bin' from 'https://cache.nixos.org'...
[20:38:26] copying path '/nix/store/n5bgxkb9wwhvmq4qkh6hxmrhj7zrlz2d-pango-1.50.7-bin' from 'https://cache.nixos.org'...
[20:38:26] copying path '/nix/store/ikb7a6ii4qsq3wkjhy9sapzn23xbiwj9-wayland-1.20.0-bin' from 'https://cache.nixos.org'...
[20:38:27] copying path '/nix/store/mnyyr92l9bhacnxvkfacig6iszna933q-brave-1.38.115' from 'https://cache.nixos.org'...
[20:53:01] building '/nix/store/bqrbxfv57777d82vbar57nvzcazzjv24-system-path.drv'...
[20:53:01] warning: creating dangling symlink `/nix/store/a3qfjlglmnbw4pbn40w0smg1fifi3b25-system-path//share/icons/hicolor/22x22/apps/brave-browser.png' -> `/nix/store/mnyyr92l9bhacnxvkfacig6iszna933q-brave-1.38.115/share/icons/hicolor/22x22/apps/brave-browser.png' -> `../../../../../opt/brave.com/brave/product_logo_22.png'
[20:53:03] warning: collision between `/nix/store/mg8b6z8wia9n58j4mmkxyjrdawb1h2m7-emacsclient.desktop/share/applications/emacsclient.desktop' and `/nix/store/pny2bh6bz9bdpb3gdmka1sc7z45a5v5r-emacs-with-packages-28.1/share/applications/emacsclient.desktop'
[20:53:04] created 31607 symlinks in user environment
[20:53:06] gtk-update-icon-cache: Cache file created successfully.
[20:53:11] building '/nix/store/5wz297vb31z23ya1s50q70nr45pb1jd6-dbus-1.drv'...
[20:53:11] building '/nix/store/cx3ix0zb9icpaqn36a6hazzg2nmslq3h-system-crontab.drv'...
[20:53:11] building '/nix/store/hm4g3r3mnpcq015hqsf69qrgfp3iyznl-unit-accounts-daemon.service.drv'...
[20:53:11] building '/nix/store/yf1kjspllccinmbppb1xpq1f9wrf23kl-unit-polkit.service.drv'...
[20:53:12] building '/nix/store/9vsgjhqc5hjbrhmfbbar623gyxjmbwd6-unit-systemd-fsck-.service.drv'...
[20:53:12] building '/nix/store/ai8y3616k0mv7mv8zxk1w2dqvrw8dp26-crontabs.drv'...
[20:53:12] building '/nix/store/l4m7r7chb61gxc6q3wfbk2lkbkwdql0y-unit-dbus.service.drv'...
[20:53:12] building '/nix/store/ydich8ppv5xgmzbvkha2gm4zkxcgh17a-unit-dbus.service.drv'...
[20:53:12] building '/nix/store/gypylgccbaknhp9mk8sw7nbbzsjnnvqh-user-units.drv'...
[20:53:13] building '/nix/store/5dz37gh8h5lnz8qdr4mdpxjj772aad34-system-units.drv'...
[20:53:14] building '/nix/store/rrzqfwdnh1i37pkvgr55x3zd016245c5-etc.drv'...
[20:53:15] building '/nix/store/r9sigfnjxmcga1gd2ysgn1509iwy5d7s-nixos-system-nixos-22.05.2785.cc634d9aa08.drv'...
[20:53:19] leaving systemd-boot 250.4 in place (250.4 is not newer)
[20:53:32] stopping the following units: accounts-daemon.service
[20:53:32] NOT restarting the following changed units: systemd-fsck@dev-disk-by\x2duuid-91D3\x2dE366.service
[20:53:32] activating the configuration...
[20:53:32] setting up /etc...
[20:53:34] reloading user units for user...
[20:53:35] setting up tmpfiles
[20:53:35] reloading the following units: dbus.service
[20:53:35] restarting the following units: polkit.service
[20:53:36] starting the following units: accounts-daemon.service

135546345 · August 31, 2022, 12:05am

While brave was downloading i run curl 'https://cache.nixos.org/nar/0kx52cfa4iiqzj892g3nxccljvszxwxb24qhql622q6f75i6ja4p.nar.xz' > /dev/null

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 56.1M  100 56.1M    0     0  86683      0  0:11:19  0:11:19 --:--:-- 51665

I used kde system monitor, while curl was running i didn’t notice an increase in the download rate

135546345 · August 31, 2022, 12:58am

I have tried both but I don’t see the difference, do you know what “keep-outputs” does? I didn’t understand what the manual says.
https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-keep-outputs

uep · August 31, 2022, 1:36am

The simplest explanation is with a common use-case, though there are others:

Some packages need source tarballs to be downloaded and added to the store manually, typically due to some manual step (maybe login or accepting terms). It’s annoying when these get gc’d, because you have to do the manual step again, so this stops them getting removed.

Other fixed output derivations also don’t get removed, which can help avoid re-downloading large tarballs too even if they can be done automatically.

Atemu · August 31, 2022, 5:50pm

Well that’s only 680Kbit/s, so there’s your problem.

It may sound weird but try a different DNS server like Cloudflare’s.

At what times does this usually happen? Who’s your ISP?

135546345 · September 1, 2022, 7:27am

I will try it.

I don’t notice a pattern with the time. I’m not in usa, so you won’t recognize it, but is a new/small company. I will later call them.

Thanks everyone for the help!!