Nixpkgs-review running out of memory

Development
1

I am trying to run nixpkgs-review on my own PR just to learn things.

Unfortunately, my machine runs out of memory very quickly and starts killing processes. This happens even if I try to limit packages to build using following command:

nixpkgs-review pr -p qgis 204148

I have no problem to build any package using nix-build -A. My laptop is fairly new Lenovo T14s with 16G of RAM. What I am doing wrong ?

Thanks

2

Does the OOM happen during one of the evaluation phases or during build?

I’m just trying to run a review against the PR you specified. Memory usage at start was roughly 8 of 16 RAM and 7 of 16 swap.

Currently there is a huge substitution going on, fetching roughly 3.5 GiB from the caches.

3

When running nixpkgs-review pr -p qgis 204148 OOM starts during the build process. I saw following lines just about the time when my machine went in to troubles:

[10/0/30 built (3 failed), 0.0 MiB DL] building mapnik-unstable-2022-10-18 (buildPhase):       |                       [10/0/30 built (3 failed), 0.0 MiB DL] building mapnik-unstable-2022-10-18 (buildPhase):       |

From what I saw before, it looks like it tried to build multiple packages at once.

4

For me only 19 of 104 have been built so far, so it might take a while until I reach the state you are in.

So far highes memory usage has been at 13/10 RAM/swap.

5

That’s strange. Looks like you are building all packages (without filtering them using -p). In my case running nixpkgs-review pr -p qgis 204148 runs out of memory in very few minutes .

6

Indeed, I forgot to add the filter. Let me restart.

For the restart the new “mem before” was 6/10 RAM/swap.

And I still have to build 84 packages overall…

So you probably had built a lot before and therefore can “fail fast”

PS: Before canceling the prev. build, I actually observed a peak of 14/10. Not directly before, but between last report and cancelling.

7

In my case, without filter it was only worse. Can some nix daemon configuration make any difference ? Thank you for trying to help me.

8

PS: If you use tmpOntempfs OOMs are actually easier to achieve than without. Therefore: Is /tmp a tmpfs or written physically?

9

Actually, I have services.earlyoom.enable = true;, but checking memory consumption during the build my machine was really going out of memory.

I don’t set boot.tmpOnTmpfs, so it should be false by default.

mount | grep "tmp"
devtmpfs on /dev type devtmpfs (rw,nosuid,size=779508k,nr_inodes=1944518,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=7795060k)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=3897532k,mode=755)
tmpfs on /run/wrappers type tmpfs (rw,nodev,relatime,size=7795060k,mode=755)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=1559008k,nr_inodes=389752,mode=700,uid=1000,gid=100)
10

df -h /tmp will tell you stats about the drive that bakes the folder, though in the snippet you pasted, there is nothing obvious to see, unless some linking happens…

What I am wondering though is, might this perhaps be related to many jobs running at once? Would it work if you were using a lower cores/max-jobs setting for nix?

And when you do nix-build, do you make sure you build from your local clone or do you build from your channel, which then might be easily substitutable.

As you say, there is still the build happen on nixpkgs-review, I doubt you ever nbuilt the exact thing that review is building now, or review wouldn’t try to rebuild it again…

11

The evaluation is the problem. 16G is not enough for that, I believe. (and you probably want to keep some other things in RAM, too)

12

I disabled earlyoom and trying again.

I went to my local nixpkgs clone and ran nixpkgs-review pr -p qgis 204148. Now it wants to download 3GB and to build 535 which it didn’t wanted one hour ago. Is it because some big change was merged to master in a meanwhile ?

13

Thanks @vcunat . How it works on other people’s machines then ? Everybody has 32GB of RAM ?

14

Yes, rebuild of everything was merged to master in the meantime, but binaries should be available for everything (perhaps not updates newer than 24h).

15

If ofborg has evaluated your PR (the action “ofborg-eval” has succeeded), then nixpkgs-review will automatically download ofborgs’ evaluation result and use that. Then you don’t need that much RAM.

16

I never* had memory problems during eval, especially did those never come up while there was already build logs streamed.

*Assuming normal conditions

17

Thank you @raphi . There is “All checks have passed” statement in my PR for few days already (including ofborg-eval).

18

I am sorry for bothering you guys here, but I wanted to learn how this thing works on simple PR until I submit something much bigger.

1 Like
19

With I earlyoom disabled, my machine crashed during build process as before - just little bit later. It ran until all 16GB of RAM and 16gb of swap was taken then crashed.

@vcunat , I think, I am running out of memory during the build not during evaluation.

20

I suppose I missed the mentioned possibility of caching evals from OfBorg. Builds can also get expensive, of course.

Personally I simply… don’t develop on machines under 32G RAM and add zramSwap.enable = true; as a cushion, but I understand that it’s not trivial for everyone.

1 Like