Hey,
After an update in August, firefox (and chrome, slack, discord, spotify…) become so slow that I had to revert to the old version to avoid being driven insane by the high UI response time.
I have no idea how to debug this. Can someone give me any pointers on where to begin?
what version were you on, and what version did you ‘upgrade’ (or 'sidegrading as i sometimes say)…
glad your roll backed worked…however… i think that’s a win in it’s in own right.
if your system is worse after an upgrade… it’s a sidegrade. ;-).
maybe a graphics driver problem? or networking issue if it’s just stuff loaded from the internet? or the dns resolver is having problems?
Do you feel up to bisecting the problem?
@nixinator I’m actually not even sure how to figure out what version I’m on. All I know is my current kernel (5.7.16. Upgrading to 5.8.x broke my system), but I probably should figure out what revision of nixpkgs I’m on? The issue is that there’s noticeable delay when doing things like hovering links (waiting for the underlining effect to show up), in addition to pages loading really slowly. In general it just feels like I’m running firefox on a 20 year-old computer.
@twoolie I’m a technical user (software dev), but a newbie when it comes to NixOS. If you can give me some pointers on how I could go about bisecting it, that would be greatly appreciated. 
Sure…
nix-shell -p nix-info --run “nix-info -m”
nix-channel --list
will tell you what channel your subscribed to, remember that this is per user… caveat each user can have it’s own channel configuration, thus it’s own version of everything running, in unix/linux land this is crazy talk, but in nixos world it’s fine.
I.e root can have a different channel settings, to say a normal user…
If you run unstable (the master branch in git world), you’ll find that things move very very quickly. one reason, that system rebuilds can be done cheaply (fairly), so rebuilding the entire system is (fairly) trivial… because nix/nixos is a build system primarily…
you maybe better off running stable rather than beyond bleeding edge, (over the edge), while your learning, so you don’t loose all hope. You can always have the system (root) running stable, and mix and match stable/unstable packages in your non root users…
this thread here that goes into some details on how to determine what channels you are subscribed, and to find versions.
has a few idea on getting your current channel
Things move very fast in nixos world. mainly a side effect that rebuilding a complete system from scratch is easy , and by that i mean the entire system. This continuous integration means new things like kernels can be integrated a much faster rate , that your normal ‘non-rolling’ linux distributions.
… mixing is nice BUT be careful with apps that have an GUI/QT [or writing state!] … golden rule → don’t mix those
if you do, be aware of thing that will be unhappy at some point in time
@nixinator When I said I downgraded my system, I actually just meant that I’m booting an old generation (I didn’t rollback the nixpkgs channel). I’ve only added channels for the root user (nixos-unstable and home-manager). And yeah, maybe I should consider switching back to stable.
The version I’m booting now is “20.09pre239318.c59ea8b8a0e” (which is the last generation I installed without the problem), and the first one with the problem is “20.09pre242769.61525137fd1”
well, your broken, but at least you got a way of going back to and older system to get working again.
Few options i can think of?
1.Get someone who has the same , or similar hardware configuration to you build a system at 20.09pre242769.61525137fd1 , and see if they experience similar problems.
2.You have to find out exactly what has changed between your working system and your not working. system. You can do that be comparing the between the working commit and the non-working git tree. You may find something has changed?
I’d certainly investigating if there has been a kernel bump, and maybe running an older kernel if for some reason you need 20.09pre242769.61525137fd1” specifically.
an interesting problem, with a bit of work you can find out what causing this…
4.GitHub - timokau/nix-bisect: Bisect nix builds. Status: alpha/proof of concept. You'll probably have to dig into the implementation if you want to use it. Built for personal use, lightly maintained. PRs welcome. Issues welcome, but I make no promises regarding responses or fix is a cool tool, but in your case, it’s not actually a show stopping build or compilation error, it’s performance :-(. it would be interesting to expand this tool to run benchmarking tests, and log the difference. Maybe that feature already exists…
I’ve never personally done a bisect for nixos, but have done git bisects before and it should work the same.
Clone Nixpkgs somewhere accessible.
$ git clone https://github.com/NixOS/nixpkgs.git
Start Bisecting
$ git -C nixpkgs bisect start
$ git -C nixpkgs bisect bad 61525137fd1
$ git -C nixpkgs bisect good c59ea8b8a0e
Git will checkout a commit halfway between these two, and ask you to check if it’s good or bad
Start Testing!
$ sudo -i nixos-rebuild -I nixpkgs=$(pwd)/nixpkgs boot -p testing
$ reboot # sadly this part will be annoying :(
The -p testing will put these intermediate test builds into their own section of the grub menu
Once you boot this config, poke around at it and see if it’s working the way you expect it to.
Mark the build.
$ git -C nixpkgs bisect good.$ git -C nixpkgs bisect bad.nixos-rebuild properly, $ git -C nixpkgs bisect skip to move on with testing.Git will guide you through the process until it arrives at the commit(s) that caused the issue you’re seeing.
Thanks a lot! I’ll start bisecting, and I’m running Speedometer 2.0 to test, just to avoid relying on “feel”. It’s scoring 50 on my good config, and 15 on the bad one, so it definitely isn’t just in my head. 
I’ll have to run 11 steps in order to pin-point the offending commit, so I’ll probably not get done in one sitting. I’ll report back in a few days.
@nixinator: I’m not sure how I’d go about tracking down someone with the same specs. I’m running i7-4930K and GTX 970. I’m assuming all other HW is most likely irrelevant.
This is taking a long time because git bisect seems to be doing a sequential search, instead of binary. It originally said it would take about 11 steps, but 30 steps later (not counting the 150+ broken builds), it’s telling me that 7 steps remain.
Every time I run git bisect good/bad it outputs something like “X revisions left to test, 7 steps remain”, where X is 1 less for each time, and is at 140 now, so I’m assuming I have 140 tests left, which would take weeks, so I think I should just stop here.
Since you mention you’re new- I’ll suggest you pin all your productivity software (for me vi + plugins, firefox, fcitx, i3) and only upgrade them occassionally. Firefox has less than 3% share and a good thing about that is nobody would bother to “hack” it. (In fact I believe nix now allows pinning specific extensions too)
Could you provide an output of git bisect log?
Pinning Firefox version is a bad idea, Firefox is very much exposed to potentially malicious JavaScript code. If you want to avoid Firefox changing too often, it’s probably a better idea to use firefox-esr package instead which stays at 78 for one year and does get security updates.
Additionally, the usage share of Firefox is larger than 3%. It’s true that NetMarketShare says that the usage share of Firefox is 3%, however if you were to exclude mobile devices it gets closer to 7%. If you consider Linux desktops only it gets as high as 30% (!). I personally wouldn’t consider 30% usage share to be small.
But why would anyone attack Linux specifically? First: Tor Browser. Second: It’s a pretty popular operating system among programmers, and Linux machines may store SSH keys, AWS keys, and so on which can be used to exploit servers they have access to.
And in fact, there are actual attacks on Firefox, attacks exploiting CVE-2019-17026 vulnerability were seen in the wild.
Here you go (I’m running a script to automatically call git bisect skip whenever the nixos-rebuild command returns a non-zero exit status):
Thanks. From what I can tell git bisect found a possible range of commits, albeit it is fairly large because the issue started occuring in a commit that doesn’t actually build, and git bisect tries every single commit in that range to find a commit that will build (it won’t).
This can be dealt with by manually fixing the issue for each commit within that range. The issue preventing the build was likely fixed by f6286dea8881e446f33a9b2faca8db43ec55aa45, so we can bisect while applying this particular patch.
First, we can restore the bisection state without skips (I’ve removed skips from bisect log as commits may have been skipped due to LVM issue):
git bisect start
git bisect bad 61525137fd1002f6f2a5eb0ea27d480713362cd5
git bisect good c59ea8b8a0e7f927e7291c14ea6cd1bd3a16ff38
git bisect bad 7c379ea8c9298f7281a8017ec5f6d2ab5ae6ad75
git bisect good 372edacef218efe37e222ada723e100e43ba272f
git bisect bad 214cfd9c45445f90d48378bf129078b6f26f983a
git bisect good e097fbbe1179a837bec318df4ae8011e94727ef5
git bisect bad df2317a6e3bc49f8a31d1da5c9694e317820df01
git bisect good d61f1a4a5f6b76e333c4dc09f027556e4d9e8357
git bisect good 67f989dc5ed9d7ca7265b268bc0037f737318389
git bisect good 28f2797a81e738234cdc57a688afb136f692672b
git bisect good 4460a3700bf36da6fb900e921c90436d29dd9852
git bisect good 1dde18f6c5bee3f5fac62479701372bb0afdf8a0
git bisect good e34b5375cd9b256ce7030fec4430dde275fe140f
git bisect good 9d7a579b2e8fd4c28861e4a3d4082e458bad68bf
git bisect good 4aa493a8290cbf7f4ee98edac82d240b988f1f26
git bisect good 0c69cd70066a0ebd5aa957babbb97c424a691128
git bisect good a27273294f3998d2349d69fefb66413da10c24aa
git bisect good 6072bac139d421e84de0032ef0a3e9f3db257803
git bisect good ddbd436dc4644aa2c95730b3be8e1a3de9154acc
git bisect bad f6286dea8881e446f33a9b2faca8db43ec55aa45
git bisect good e40dd8d4b95a2c213ad1b16969cc018af45454f7
git bisect good b979b12128870c7c214b591fd737b9ad958a2cea
git bisect good 5d36e00b7df86c6675ff3e81349c6badf5f0a0c1
git bisect good 2647686ca16545142da3978e09c61a67255d8ee4
git bisect good df267a4cca6814261e10e9e3cb87fb706519307d
git bisect good e1f177fab25841f6f94bc55c04b5af43ce6b9f1b
git bisect good 9913501999d8289963b3ed91aab48fa2d671da0c
git bisect good 548bd1e04f97ad4fb27953e5c580e6a159e74a82
git bisect good 1621b5173a012bd5fb728375fe7a2d24e84895a8
git bisect good a0afc29460af23eae8a19c452acd87cb4bf1b6df
git bisect good c35a75c28cd090893452e97d6b2112ee44b42f58
Then for each step we can apply patch from f6286dea8881e446f33a9b2faca8db43ec55aa45 commit.
git cherry-pick --no-commit f6286dea8881e446f33a9b2faca8db43ec55aa45
sudo -i nixos-rebuild -I nixpkgs=$(pwd) boot -p testing
reboot
git reset --hard
git bisect good/bad
When I recently wanted to update, I also noticed a considerable slowdown, not only of Firefox, but the whole computer (though most noticeable on Firefox).
In my case, I had
boot.kernelPackages = pkgs.linuxPackages_latest;
in my config, opting me into the latest Linux kernel. Removing that line solved the slowdown!
In my case, linuxPackages_latest is 5.8.14, while the default one is 5.4.70. So some kernel update in that range must’ve been the problem, will probably bisect that eventually.
If you experience problems with a kernel version after 5.4, you might want to try downgrading to see if that fixes it for you too.
I tried this, but the build still failed (the first commit I tried building was 231e2ea1ff0).
Here’s the tail of the build output:
/nix/store/ysxchaa5rjjg28vqlfsy4k479wwqviqp-qtbase-5.14.2-dev/bin/qmake -install qinstall /build/qtquickcontrols-everywhere-src-5.14.2/src/controls/Styles/Base/images/needle.png /nix/store/i9lljvzpgfxwj79nka6lb8cids7pgyvb-qtquickcontrols-5.14.2/lib/qt-5.14.2/qml/QtQuick/Controls/Styles/Base/images/needle.png
mkdir: cannot create directory '/nix/store/w8k3lq0mxlwkjxqvak956p3jfrhhbmx7-qtdeclarative-5.14.2-bin/lib/qt-5.14.2/qml/QtQuick/Controls': Permission denied
make[2]: *** [Makefile:1642: install_qmlcacheinst] Error 1
make[2]: Leaving directory '/build/qtquickcontrols-everywhere-src-5.14.2/src/controls'
make[1]: *** [Makefile:64: sub-controls-install_subtargets] Error 2
make[1]: Leaving directory '/build/qtquickcontrols-everywhere-src-5.14.2/src'
make: *** [Makefile:61: sub-src-install_subtargets] Error 2
builder for '/nix/store/682c2y33slg99jznhp1nzcymjs8jxdnb-qtquickcontrols-5.14.2.drv' failed with exit code 2
cannot build derivation '/nix/store/7h32a8mlw3y566swsk5m6v4an2pqx22n-qtwayland-5.14.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/fgb7yzfif8j363xx95yzj7cl63385mdb-hook.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/kmbxray5ivb28d2m35plsayir6dip60r-flameshot-0.6.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/p71d7n8npl7pkxnp4ja0chh3ar1gbgxa-flameshot-0.6.0-fish-completions.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/z7746h4rfrpyijd4n3l6id42gbzsws0m-home-manager-path.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/0qvmr0bj2gp7ycfgwjnd8y0fr01jr7s4-home-manager-generation.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/4iw4rqnvpi8x7s3mzbijkisy7gxcrypj-activate-jlo.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/8v35csrgk9cic698av6cwvyb6kcxfc4z-unit-home-manager-jlo.service.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/b80ja01nfg9hynnjk10jbj0fl2ql4ifq-system-units.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/3kxm83fqwffkysrgj346llkbkc6s0id9-etc.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/0wph0agng800azrikjpsnqzhjmw8lnsd-nixos-system-jlo-pc-20.09.git.231e2ea1ff0M.drv': 1 dependencies couldn't be built
error: build of '/nix/store/0wph0agng800azrikjpsnqzhjmw8lnsd-nixos-system-jlo-pc-20.09.git.231e2ea1ff0M.drv' failed
Should I just skip it and keep trying?
@Infinisil
In my experience while bisecting the issue, the issue seems to always appear on 5.8., while never on 5.7., though I have avoided jumping to conclusions since the kernel is far from the only thing changing. Thanks for the suggestion. I’ll try downgrading only the kernel and see what happens.
Edit: Downgrading (or should I say upgrading?) to pkgs.linuxPackages instead of pkgs.linuxPackages_latest fixes the issue. (I’m scoring 44 points in Speedometer 2.0 on kernel 5.4.69, compared to 50 on kernel 5.7, and 15 on kernel 5.8).
Now I feel a bit silly for not realizing that downgrading the kernel was so easy. Even though the issue is fixed for me, I’ll might try bisecting the kernel until I find the commit causing the issue (though that might require me building the kernel from source?).
Thanks a lot for the help so far! This community is quite a bit more helpful than what I’m used to.
Well, at this point you have found out that kernel update is the cause, so no need to bother with rebasing nixpkgs anymore, it’s unlikely to find anything useful.
First, feel free to check whether the issue still exists in Linux 5.9. If it doesn’t, chances are there is no real need to report a bug, as 5.8 is not an LTS release and is very close to EOL at this point. linuxPackages_5_9 package is available in NixOS 20.09.
It is possible to bisect Linux kernel, albeit keep in mind this is going to be a very long process (Linux takes a long time to build). If you want to try, consider cloning Linux kernel (git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git) and configuring kernelPackages to use it.
boot.kernelPackages = linuxPackagesFor (linux_5_8.override { argsOverride.src = /path/to/git/clone; });
Then you can start bisecting the kernel and rebuilding after every step to determine whether an issue exists in a given commit.
nixos-rebuild boot -p kernel-bisect
As for starting points, considering the issue occurs on 5.8 but not 5.7 you may try using
git bisect good v5.7
git bisect bad v5.8.14
5.9 did not fix the issue (built from master @ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git, using linux_5_8_override, overriding src and modDirVersion. I hope that’s fine).
The build took about 30 minutes, which is not too terrible. I’ll start by testing the two tags you mentioned, then start bisecting.
Edit: I think maybe I should be using git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git. It’s really not obvious to me which one is the “right” one.