I started investigating why my personal PC is underperforming compared to some benchmarks and ended up spinning up AMD 16 core machine on Hetzner. I switched back and forward a couple of times between distros using nixos-infect and Hetzner rebuild machine from Ubuntu image and the results were more or less the same (so it’s unlikely that it is caused by cloud scheduling). Here are a couple of runs:
This is probably not measured by a benchmark, but I bet it causes a significant real world slow down in NixOS compared to more traditional distros.
One recurring pattern in NixOS is isolating files under many different store paths – one or more for each program – and then combining them via a path-like variable consisting of huge list of paths. For example, the dynamic loader looks at the rpath, shell looks at the PATH, Qt and GTK applications look for themes/icons/whatever using XDG_DATA_DIRS, …
If you trace the system calls of any program, you will almost surely see it go though all these paths, making hundreds or even hundred of thousands of access or stat calls searching for a shared library, or an icon, or whatever before it eventually finds it and start all over again with another library, and on and on.
I always thought that, while this looks dramatic in an strace, it’s probably not that impactful for most programs because things like libraries are generally loaded only once. Do you know of examples of things like this that need to be looked up again and again over the life of a program?
If so, maybe we could improve those cases by building a big symlink directory in the package for that program instead of a big environment variable that is resolved at run time?
Do you know of examples of things like this that need to be looked up again and again over the life of a program?
XDG_DATA_DIRS has been a recurring source of trouble[1][2][3]. I know for a fact that it’s looked up every time I open the (Qt) file picker, but there are probably more examples.
we could improve those cases by building a big symlink directory
I think that could work, but there may be a reason it hasn’t been done already.
Its made even worse, that in XDG_DATA_DIRS and other path environment variables have the same folder duplicated. In fact, it grows every time one opens a shell (being duplicated first, open bash they become triplicated, etc).
Tested by watch env | wc -l grow.
Here is a topic about it and a possible workaround, but I don’t know how to make the “kde launcher” clean the duplicated variables automatically.
Most likely caused by compiler flags, for example, nixpkgs does not enable LTO by default, but Arch Linux, Ubuntu, Fedora, OpenSUSE and more enable it by default.
More examples:
Issue 3 has been resolved, but 1 and 2 and possibly other packages are slower on NixOS than other distributions
I tired KDE Linux in a virtual machine, and it’s even faster than my host os (NixOS).
This might be caused by the explosion of env vars like XDG_DATA_DIRS, though most of the time, it should only slow down startup.
EDIT: I think “disable pgo for clang” is not entirely correct, pgo is not enabled for clang in most other distros either. Python is indeed slower.
I just tried running this benchmark on NixOS, an Arch Linux container, and an Ubuntu container. The containers are running in Podman on NixOS.
In the Ubuntu container, the score is about 7400; in the Arch Linux container, it’s about 6500; and in NixOS, it’s about 6400.
Note that the CPU benchmark is written in C, so the result is unrelated to Lua.
$ nix run nixpkgs#sysbench -- --threads="$(nproc)" cpu run
Running it on other system container
Arch Linux as example
# to enter arch linux
$ podman run -it --name arch-container archlinux /bin/bash
# then install sysbench and run benchmark
# exit
$ exit
# to enter the containr nex time
$ podman start -ai arch-container