I can't update NixOS anymore because of webkitgtk manual compilation

Help! I cannot update NixOS anymore because suddenly webkitgtk-2.42.3+abi=4.1 and webkitgtk-2.42.3+abi=6.0 needs to get manually compiled. My laptop does not have enough RAM (only 32 GB) for it to be compiled. What should I do? This started happening suddenly.

I am running a flake based system. Here is the whole flake repo.

When I update, I do:

nix flake update
sudo nixos-rebuild switch --flake .#asus_gnome

I have good luck using the --max-jobs (otherwise known as -j) flag to control the concurrency that Nix uses to rebuild.

Using -j 1 will usually get out of these jams, albeit slowly.

Let me try this. But I thought the limiting factor was memory, not CPU.

Reducing concurrency also reduces the amount of memory required. A ratio of about 2 GiB of RAM per assigned thread/core works well for most cases. I have no clue about WebKit though

Well, using --max-jobs 1 is also causing a failure.

(I did not press any key, it just died on its own)

Created a GitHub issue here.

Usually when something needs to be compiled downstream your nixpkgs version is simply not one for which hydra has artifacts:

url = "github:NixOS/nixpkgs/release-23.05";

Don’t use the release branches, use nixos-23.05. Afaik the release branches are what people apply patches to, you should not expect artifacts to be built for that branch, that happens on the nixos-* branches, along with all testing.

Those branches should maybe be renamed to dev-23.05 or something, then it will be more clear that you will from time to time hit things that haven’t been built by hydra yet.

Also, 23.11 is out now, I think 23.05 will be EOL next week, so upgrade. I don’t think artifacts for older nixpkgs versions are usually deleted, at least not currently, but still, 23.05 doesn’t really get security patches anymore.

8 Likes

Yes this did it. Thank you so much. Updated to 23.11 as well.

So, this issue has been solved by picking the right nixpkgs branch, but I still think this is worth commenting on. There’s an important difference between --max-jobs and --cores. The max-jobs setting controls how many individual derivations Nix will build at a time simultaneously. The cores setting will inform (not control) those derivations how many threads they ought to launch. For instance…

My personal Hydra configures its builder with cores = 16 and 2 max-jobs. Theoretically, a Threadripper 1950X is capable of more than this (because most builds don’t use every core they can), but containing it to 2GiB of RAM per thread is the only way to keep it from crashing when it builds a few webkits at the same time.

Sorry, I know this is not OP’s issue. Just some interesting stuff to think about.

3 Likes

Do you know how to specify the RAM limit?

There basically isn’t one. The best you can do is systemd restrictions on nix-daemon.service, but that’s not very nuanced. It helps though

1 Like

Borrowing a term used elsewhere, releng-XX.YY would indicate this is the release engineering branch, not the release branch, and perhaps save a little of this confusion.

2 Likes

There is no RAM limit that you need to specify. The “2 GiB per thread” is a rule of thumb to get you an estimate of the allowed concurrency.

When we say this, we mean, that if you have 4 GiB of RAM, then you can have a concurrency of 2, if you have 16 GiB of 8, and so on. You should never increase this above your CPUs thread count though.

And as someone already sayd, you have 2 values to adjust the concurrency, and depending on the usecase, its worth to play with them under different circumstances.

Try mounting an external SSD as an extra swap partition. Your build will be slow, but you should be able to overcome your RAM limitations.

So why is cache.nixos.org missing a build of webkitgtk? The latest version is webkitgtk-2.42.4+abi=4.0