I have a flake config for my raspberry pi 5 on my laptop and a local script that runs nixos-rebuild switch --flake ".#rpi5zfs" --build-host root@foo.me --target-host root@foo.me that I wrote to build everything on the raspberry pi. Sadly it takes a lot of time before even starting the SSH connection (maybe like 30 seconds even when I don’t change anything) and then when I do change stuff it copies lot’s of file from my laptop to the raspberry pi. Why is the startup time that long and how can I really delegate everything to the raspberry pi, so that my laptop gets no extra files in its /nix folder at the end of the compilation?
Evaluation always happens locally. So if you have IFD for example you’ll have such delays.
1 Like
@waffle8946 Arg too bad, thanks for the answer. Can’t we also delegate the evaluation? (with nixos-rebuild ideally or another tool if needed) Can we “debug” what is causing this delay? (I don’t remember using IFD myself and nixpkgs does not allow them I think) And IFD aren’t supposed to be run on the build-host since they might involve compilation?
There was some flamegraph nix profiler mentioned on this forum before, but I can’t find it atm. Maybe you’ll have better luck with those keywords?
That’s precisely why IFD is best avoided, nix can’t build a nicely parallelizable build graph if it needs to build to be able to evaluate an expression.
The derivation will be built on the remote, then downloaded and included in the evaluation, with all the delay that implies.
@waffle8946 do you mean nom GitHub - maralorn/nix-output-monitor: Pipe your nix-build output through the nix-output-monitor a.k.a nom to get additional information while building. · GitHub ? If so I don’t think it will change anything because in the first loading part it really don’t print anything. That being said, I also have a long delay bore starting the compilation also when switching my local config while it was must faster before… Could it be a different bug? Do you also need to wait around 30s before anything happens when switching to a new derivation?
@TLATER but then I would see a ssh connection made much earlier no? What do you think of the comment in my previous paragraph?
Yes, that is the evaluation time. nom would show that information and how much time exactly is passed since you started it.
Did you have this thread in mind? Nix flamegraph or profiling tool?
1 Like
That’s the one, thanks!
Ok so I did a quick check: on my system (also flake), switching to a no-op takes 8s while doing the same kind of operation on the flake that will be deployed to the raspberry pi takes ~1.10mn, with 55s spend during the evaluation!! So it seems like something is really wrong with my raspberry pi setup, while I don’t think it contains anything crazy. Any idea how to troubleshot this? I tried nom but I don’t see much helpful operations there (it helps only during the compilation part), I’ll check the thread you mention, but it’s a bit confusing to see many methods that are a bit old, so if you have some recommandations to try that work with flake + remote builders that would be great. I also see that nix now has an official profiler A look at NixOS/Nixpkgs evaluation times over the years - #39 by purepani can I use it in this context?
Aha, so I tried:
❯ nix run github:crabdancing/nix-flamegraph -- -t .#1rpi5zfs
Started `nix eval` against target...
but this is printed right after starting, so without evaluating anything. Seems like it won’t help here (or certainly that I’m doing it wrong).
Did you get any output?