Slow NixOS rebuild

Help
1

I’ve been a NixOS user for over 2 years and with the growth of my NixOS configuration, I found that my system rebuild is becoming slower (takes around 3-5 mins and a lot of memory to just evaluate my system top-level derivation) but I can’t figure out the reason behind it. I’ve followed the advice in other posts to avoid instantiating many nixpkgs but it still doesn’t help.

I’m wondering if there’s a way that I could get more verbose statistics to help to find which part causes the slow rebuild. I would appreciate any ideas and advice.

1 Like
2

Are you doing any import from derivation (IFD)?

3

Have you tried --verbose?

NixOS only rebuilds parts that change. Might try smaller incremental changes.

Commands like nix flake update can cause much building.

5

Yes I did. The --verbose flag just shows what it is currently building. But my problem is that the first step to evaluate the system.build.toplevel derivation takes most of time and memory. It hasn’t started building anything yet. Building itself is fast enough comparing to evaluation

That’s also why even a very small change can take nearly the same amount of time.

1 Like
6

I don’t think so. But I’m doing something like imports = listFiles ./modules, where listFiles calls builtins.readDir to import all files in modules directory, though I don’t think it needs extra instantiation as /etc/nixos will always be added to store.

7

What you feel is probably true, at least to some extent, and is unavoidable in your side.

NixOS includes a large module system that is not completely lazy, which means you have to go through all options (including ones you never use) to get the final configuration. We might try to resolve this, considering that new, discrete modules can have similar properties to packages, i.e. have explicit dependency paths. (Related: NixOS evaluation speed regression · Issue #79943 · NixOS/nixpkgs · GitHub NixOS evaluation speed regression · Issue #79943 · NixOS/nixpkgs · GitHub)

But I still doubt the possibility of IFD. On my 7840HS, evaluation tends to take no more than 30 seconds (although that’s considerably slow).

1 Like
8

Yeah I also doubt it’s due to IFD but I just couldn’t find it in my config files. Is there a way to show the evaluation traces so I can find out what really happened?

9

--option allow-import-from-derivation false? Not sure.

1 Like
10

No, that shouldn’t cause IFD. Best disable the feature as @Aleksanaa recommends to find out though.

Could you post your config? Feel free to redact private information but it’s important to see what you’re setting and how.

Are you using nixos-containers?

11

Unfortunately, there’s no difference in performance after enabling the option. I also tried nix --extra-experimental-features nix-command build /etc/nixos#nixosConfigurations."xxx".config.system.build.toplevel --no-allow-import-from-derivation --verbose --dry-run but still no difference.

12

The option would make your build barf if IFD was used. If it evals fine, then you’re not using IFD.

13

Yes I’m using around 6-8 nixos-containers. Now that you mention it, it is indeed likely that these containers are slowing down the evaluation.

As for the config, It may take some time to clean them up before posting it as there are too many files. And there’s no IFD as it evals fine.

14

Each nixos-container is its own nixos config and eval time would therefore grow linearly. Could you try disabling all the nixos-containers and evaling the config? (Eval is enough, no need to build.)

1 Like
15

It seems these containers are indeed the real culprit. The evaluation is around 5x faster and uses much less memory after disabling them!

But nixos-containers are really convenient to run services in isolated environments. Is there any way to speed them up? The config of each container is usually just enabling a service.

16

If you are just worried that your service is not isolated enough, you can add additional hardening features to your systemd service. I am certainly not saying that nixos containers cannot be used, but they may be a bit too heavy for this scenario.

1 Like
17

I still think systemd-nspawn fits my use cases better, especially for the network isolation. It’s hard (though still doable) to set up the namespace and join them using veths and bridge just using systemd service config.

Or are you suggesting that I configure systemd-nspawn from scratch? (I haven’t tried it yet but maybe it’s doable, but I need to figure out how to convert nix config to systemd-nspawn config)

18

Finally I found a solution here: Very slow evaluation and huge RAM usage with many nixos-containers · Issue #65690 · NixOS/nixpkgs · GitHub. I pass pkgs from the host to avoid evaluating nixpkgs multiple times and the performance was greatly improved!

Thanks to all who helped me with this issue.