NFS slows down after upgrading to 23.11

Help
1

Hi All,
After upgrading from NixOS 23.05 to NixOS 23.11 the NFS performance from server to NixOS client dropped to around 3 to 4 MB/secs. From NixOS client to server works fine.

There has been no change in the NFS configuration. The mount options are the same as before:
x.x.x.x:/svr/hans on /svr/hans type nfs4 (rw,nosuid,nodev,noexec,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=xx.xx.xx.xx,local_lock=none,addr=xx.xx.xx.xx,user=xxx)

The kernel from 23.11:

Linux mercurius 6.6.4 #1-NixOS SMP PREEMPT_DYNAMIC Sun Dec  3 06:33:10 UTC 2023 x86_64 GNU/Linux

There are no errors in dmesg and journalctl.

I would like that NFS works as before because I work with large files or a lot of small files.

Does any know this a bug or knows how to proceed?

2

Hi there,
I made more analyses.

  1. The flake configuration of this machine forces the most possible kernel version 6.6.5.
  2. In a more simple setup with nixos 23.11, it uses initially kernel 6.1.x at which NFSv4 works fine as before. If you use config boot.kernelPackages = pkgs.linuxPackages_latest NFSv4 has the same problem.
  3. Checking the NFSv4 setup including troubleshooting did not result in any remark.
3

Based on this info, this smells like an upstream kernel bug.

Could you try linuxPackages_6_5?

4

Hi there,

I found the kernel and kernel packages using nixos package search:

  • linuxKernel.kernels.linux_6_5
  • linuxKernel.packages.linux_6_5.zfs

I added the kernel directive:
boot.kernelPackages = pkgs.linuxKernel.kernels.linux_6_5

nixos returns with errors:

[root@mxxxxx:/etc/nixos]# nixos-rebuild dry-build --flake flake.nix
building the system configuration...
path '/etc/nixos/flake.nix' does not contain a 'flake.nix', searching up
error:
       … while calling the 'head' builtin

         at /nix/store/z0513hax0dsbix0x89wr7fca8sa99sxc-source/lib/attrsets.nix:850:11:

          849|         || pred here (elemAt values 1) (head values) then
          850|           head values
             |           ^
          851|         else

       … while evaluating the attribute 'value'

         at /nix/store/z0513hax0dsbix0x89wr7fca8sa99sxc-source/lib/modules.nix:807:9:

          806|     in warnDeprecation opt //
          807|       { value = builtins.addErrorContext "while evaluating the option `${showOption loc}':" value;
             |         ^
          808|         inherit (res.defsFinal') highestPrio;

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: attribute 'extend' missing

       at /nix/store/z0513hax0dsbix0x89wr7fca8sa99sxc-source/nixos/modules/system/boot/kernel.nix:43:31:

           42|       type = types.raw;
           43|       apply = kernelPackages: kernelPackages.extend (self: super: {
             |                               ^
           44|         kernel = super.kernel.override (originalArgs: {

There is an attribute extend missing in the definition. I would like to know how to proceed.

5

The correct config is:

 boot.kernelPackages = pkgs.linuxPackages_6_5;

The error message you see here seems like and unrelated issue with your flake. I don’t use flakes, so I can’t help you with that and it’s a topic for another discussion as this issue would happen regardless of your actual NixOS config.
(My guess would be the (IMHO) needless tight coupling of flakes with git; make sure you git add all the files, dot all the is, cross the ts and whatever other new dance flakes might require by now.)

6

Hi there,
I applied the kernel config for kernel 6.5 and got kernel 6.5.13. As a bonus I got A correct nvidia (unfree) driver. With this kernel the NFSv4.2 problem does not occur any more.
Finally I made some adjustments to cleanup the build. The commands to build with flakes in my environment is:

export NIXPKGS_ALLOW_UNFREE=1
nixos-rebuild boot --flake "git+file":///etc/nixos --impure

Thanks to Atemu I could apply a workaround.

7

Note that kernel 6.5 won’t be around for much longer; it’s already marked as EOL on kernel.org.

I’d highly recommend sticking to the regular LTS kernel instead (6.1 for 23.11, simply unset
boot.kernelPackages).

8

Hi there,
Good point to use the LTS kernel version 6.1. This could be easy changed in my configuration.nix and works ok with NFSv4.2:

boot.kernelPackages = pkgs.linuxPackages_6_1;

In my situation this overrides the flake configuration :slightly_smiling_face:.

9

Hello,
Did you find out what the root cause of the problem was ?
I use NFS between a client(with a mpd server) and a server with the music files.
I also noted that the performance was quite low…

It is sufficient for my purposes now, though…

Regards,
CM

10

Hi,

Currently I still use Linux kernel 6-1. I did not look into the root cause, because it experienced it as too complicated. In Q2 2024 I reinstalled NixOS without ZFS encryption and without flakes to keep the configuration simple. After reinstallation the NFS performance was normal. Possibly I have to move to higher version of the linux kernel. I think I will look into it if I upgrade to 24.11. :slightly_smiling_face:

Hans