Unable to fix "too many open files" error

Help
1

Hello everyone,

I will be brief, after playing around with custom CPU flags I am unable to rebuild my system.
The process fails outputing a known error: “too many open files”.

After a quick internet search I have found this issue: Cannot raise maximum number of open file descriptors inside of nix build environment · Issue #4018 · NixOS/nix · GitHub
In which temporally augmenting the value of systemd.services.nix-daemon.serviceConfig.LimitNOFILE is suggest as a fix.

My problem is that even after raising the number up to a ludicrous amount the error does not disappear.

As a note:
I had to define it with lib.mkForce because the value was already defined somewhere else. (Not by me.)
My default value was already higher than the one suggested in the issue.

What am I missing?

Thanks in advice. :D

3 Likes
2

Possibly you accidentally altered an OS limit somehow. If you cannot rollback to before your “playing around”, you may have to investigate all the bits which can restrict the number of open files. Ideally you can find something to tell you what is limiting the build – maybe in build logs or in the system journal.

Poking around on my system, I found these interesting bits

  • cat /proc/sys/fs/nr_open returns 1073741816
  • ulimit -Hn returns 524288
  • sudo grep -R NOFILE /etc returns many files including
/etc/static/systemd/system/nix-daemon.service.d/overrides.conf:LimitNOFILE=1048576
/etc/static/systemd/system/nix-daemon.service:LimitNOFILE=4096

I have not set LimitNOFILE anywhere in my config; so these must be default values.

Here are some links which helped me understand more about NOFILE

2 Likes
3

I’ve managed to solve this issue, thanks for your reply. :slight_smile:

1 Like
4

Could you share what you did to fix the problem?

1 Like
5

After setting nixpkgs.hostPlatform.gcc.arch the build is failing with “Too many open files”. What did you do to resolve this?

6

Sorry for the late response but I’ve been away lately.

Right now I can exactly recall what steps I’ve took in order to fix the issue.

But I can tell you that if you want to build certain stuff you should disable boot.tmp.useTmpfs because if not building stuff will eat up your RAM.

You can also change where nix keeps its tmp files. I don’t know why do they use /tmp by default.

I’m not sure but you can try that and see if it helps.

7

At the risk of bumping an old thread, I encountered this issue while running a machine with quite a number of nixos containers.
I was able to fix it by increasing the systemd open files limit with:

  systemd.extraConfig = "DefaultLimitNOFILE=2048"; # defaults to 1024 if unset

This will require a reboot on the machine

5 Likes
8

Also sorry to necrobump, but this is one of the most prominent search results for the problem.

If you’re encountering this issue whilst doing nixos-rebuild switch, you might have tried to up the limit on nix-daemon.service (which is ludicrous by default anyway) and realised that it doesn’t work.

Turns out: it seems like nixos-rebuild switch does not use the Nix daemon to do the build (I tried disabling the daemon and noticing that builds don’t fail).
Apply a new limit in your shell that is doing the rebuild instead: ulimit -n 4096 for example. Then run the rebuild.

I am no expert but I think the Nix daemon is just there for multi-user support and probably only is there to make building possible by unprivileged users without letting them inject fake files into the store and making sure the builds are pure.
When you are the root user, I guess this does not apply.
This worked for me.

4 Likes
9

Not sorry to necrobump, because I’ve been struggling with this issue for months and no resource on the internet seems to get me further. The initial poster said that raising the limit did not help. That is also my experience. I did what jonboh said and raised the limits. I set:

systemd.extraConfig = "DefaultLimitNOFILE=1024:1048576";

And this does take effect:

:> ulimit -nS
1024
:> ulimit -nH
1048576

The investigations ericgundrum suggested don’t report anything more useful:

:> cat /proc/sys/fs/nr_open # I don't know what this number means but it's the same as in their post.
1073741816
:> sudo grep -R NOFILE /etc # Most of these numbers are decently high and the smaller ones seem to only affect NetworkManager.
/etc/static/systemd/system/systemd-logind.service:LimitNOFILE=524288
/etc/static/systemd/system/dbus-org.freedesktop.login1.service:LimitNOFILE=524288
/etc/static/systemd/system/systemd-journald.service:LimitNOFILE=524288
/etc/static/systemd/system/systemd-journald@.service:LimitNOFILE=524288
/etc/static/systemd/system/sysinit.target.wants/systemd-journald.service:LimitNOFILE=524288
/etc/static/systemd/system/multi-user.target.wants/systemd-logind.service:LimitNOFILE=524288
/etc/static/systemd/system/nix-daemon.service:LimitNOFILE=1048576
/etc/static/systemd/system/NetworkManager.service:LimitNOFILE=65536
/etc/static/systemd/system/nix-daemon.service.d/overrides.conf:LimitNOFILE=1048576
/etc/static/systemd/system/dbus-org.freedesktop.NetworkManager.service:LimitNOFILE=65536
/etc/static/systemd/system/network.target.wants/NetworkManager.service:LimitNOFILE=65536
/etc/static/systemd/system.conf:DefaultLimitNOFILE=1024:1048576
/etc/systemd/system/systemd-logind.service:LimitNOFILE=524288
/etc/systemd/system/dbus-org.freedesktop.login1.service:LimitNOFILE=524288
/etc/systemd/system/systemd-journald.service:LimitNOFILE=524288
/etc/systemd/system/systemd-journald@.service:LimitNOFILE=524288
/etc/systemd/system/sysinit.target.wants/systemd-journald.service:LimitNOFILE=524288
/etc/systemd/system/multi-user.target.wants/systemd-logind.service:LimitNOFILE=524288
/etc/systemd/system/nix-daemon.service:LimitNOFILE=1048576
/etc/systemd/system/NetworkManager.service:LimitNOFILE=65536
/etc/systemd/system/nix-daemon.service.d/overrides.conf:LimitNOFILE=1048576
/etc/systemd/system/dbus-org.freedesktop.NetworkManager.service:LimitNOFILE=65536
/etc/systemd/system/network.target.wants/NetworkManager.service:LimitNOFILE=65536
/etc/systemd/system.conf:DefaultLimitNOFILE=1024:1048576

Still, mostly steam games but on some occasions other programs, fail reporting too many open files.

If anyone has any more experience or insight here, I would be very grateful.

Edit:
I did a watch cat /proc/sys/fs/file-nr while starting a steam game that runs into the “too many open files” issue and the number of open files never went above 16k, which is way below the smallest number in the output above.

1 Like
10

I was encountering the same issue, and also tried a few of the suggestions including systemd.extraConfig = "DefaultLimitNOFILE=1024:1048576"; without any luck.

What fixed it in the end for me was adding the following to my nix config:

  security.pam.loginLimits = [
    { domain = "*"; type = "soft"; item = "nofile"; value = "65536"; }
    { domain = "*"; type = "hard"; item = "nofile"; value = "1048576"; }
  ];