Profiling/optimising NixOS Configuration Derivation

[peterc-s]

For a while now my rebuilds have been taking longer than I’d really like. I use nh os switch for rebuilding and switching configurations, and it takes about a minute and a half on average for '.#nixosConfigurations.laptop.config.system.build.toplevel' to be built. However, I don’t know how to profile any further and figure out what is causing the slow down.

nh does provide a handy view of the dependency tree but that comes after the minute or so from the nix build I mentioned above. I have already pruned software that takes a while to set up which I don’t use often enough to warrant keeping it in my configuration.

If there isn’t a way to profile further, are there any general rules-of-thumb for things that may take a while and ways to optimise them?

[therivercass]

check how many times you’re evaluating nixpkgs. each flake you set as an input is likely pulling in it’s own copy of nixpkgs. use inputs.${input}.inputs.nixpkgs.follows = "nixpkgs"; judiciously to avoid evaluating nixpkgs over and over.

[TLATER]

It’s probably worth using nix eval to check if evaluation or build time is the main issue first, but yeah, that’s a common culprit.

[peterc-s]

@therivercass

use inputs.${input}.inputs.nixpkgs.follows = "nixpkgs";

All my other inputs follow the main nixpkgs input as per all of their documentations, I’m not sure that this is the issue.

@TLATER

It’s probably worth using nix eval to check if evaluation or build time is the main issue first

~1m 24s on nix eval. Not getting any output even with --verbose.

edit: Just ran through with --debug and it doesn’t get stuck on anything. Would be nice to figure out what is taking up most of the time though.

[Shawn8901]

FYI: the follows derivative just sets that they point to the same rev, AFAIK there are likely still two instances.

So if a flake does something like pkgs = import inputs.nixpkgs {} or similar, that is still a new evaluated instance, nevertheless if you follow and input or not.

A lot of those will likely result in a longer eval time.

Tho gathering metrics is the best to analyze it.

[peterc-s]

Ah I mean that makes sense. Is there a way to list out the instances?

What tools are there for gathering metrics? Anything akin to standard profiling tools?

[zendo]

>  nix shell nixpkgs#flamegraph
>  nix eval --no-eval-cache .\#nixosConfigurations.host.config.system.build.toplevel \
    --option eval-profiler flamegraph \
    --option eval-profile-file host
>  flamegraph.pl host > host.svg

[peterc-s]

Thank you! Pretty much exactly what I needed

For whatever reason, eval time has dropped to ~25 seconds which isn’t awful but I have found that a single package (haskell-language-server) is taking up a lot more time than other packages to eval (~10% of overall eval time).

edit: it is now back to ~1m30. 24% of which is haskell-language-server. I have no idea why:

Anyone know if I should be expecting these two big humps too? I don’t really know how the eval process works.

[rvl]

@peterc-s

For whatever reason, eval time has dropped to ~25 seconds

Is it possible that you forgot the --no-eval-cache option?

@peterc-s

I have found that a single package (haskell-language-server) is taking up a lot more time than other packages to eval (~10% of overall eval time).

I think because hls is a big package in terms of dependencies. For me it takes about 8.5s to eval when tested in isolation, compared to 0.6s for pkgs.hello (nixpkgs rev 00c21e4c9).

NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 NIX_SHOW_STATS_PATH=haskell-language-server-stats.json nix eval -f default.nix haskell-language-server --no-eval-cache --eval-profiler flamegraph --eval-profile-file haskell-language-server.profile && flamegraph.pl haskell-language-server.profile > haskell-language-server-profile.svg

If you have multiple GHC versions in the supportedGhcVersions argument of pkgs.haskell-language-server then it will take still longer to eval. I think this could explain the two humps in your flamgraph.

However, if you have other Haskell packages in your NixOS closure, then they will potentially share dependencies with HLS, effectively spreading the evaluation cost.

For a while now, I have installed HLS into my ~/.nix-profile (i.e. with nix profile add my-config#my-haskell-env), keeping it separate from NixOS configuration. This was for other reasons, but I now realise that it helps reduce nixos eval time.

---

For viewing the profile and listing slow packages, I also use the speedscope.app Sandwich View, press Ctrl-F to filter on derivationStrict, then sort by Total samples. It’s not ideal because derivation inputs need to be considered when judging packages – flamegraph.pl shows this on the y-axis.

[peterc-s]

@rvl

Is it possible that you forgot the --no-eval-cache option?

Unfortunately not the issue (as it went back up to ~1m30 afterwards, for whatever reason).

@rvl

I think because hls is a big package in terms of dependencies

Is it really that bloated? It’s 4-5x any other package on my system and I thought I had a few pretty dependency heavy packages.

@rvl

I think this could explain the two humps in your flamgraph.

I mean the two humps seem to not just be because of hls. I checked the flamegraph with and without my Haskell module. I also don’t think I have multiple GHC versions:

{
  config,
  lib,
  pkgs,
  inputs,
  ...
}: {
  environment.systemPackages = with pkgs; [
    ormolu
    haskell-language-server
    (
      ghc.withPackages (
        hsPkgs:
          with hsPkgs; [
            turtle
          ]
      )
    )
  ];
}

@rvl

For a while now, I have installed HLS into my ~/.nix-profile

I’m somewhat new to Nix and haven’t had a chance to play around with profiles yet. Will have to look into them

@rvl

For viewing the profile and listing slow packages, I also use the speedscope.app Sandwich View, press Ctrl-F to filter on derivationStrict, then sort by Total samples.

So much of it is Haskell, wow. I’ll have to figure out why etc and dbus are taking long too.

edit: looks like a lot of etc is just… Haskell.