I did some troubleshot and here is what I’ve found:
At 4:40 AM, NixOS was updated, which is normal.
At 17:05 PM, httpd was reloaded by logrotate
At 17:05 PM, I get PHP error messages in logs
PHP logs:
[Wed Feb 02 17:05:13.246275 2022] [http2:warn] [pid 710] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
PHP Warning: PHP Startup: Unable to load dynamic library '/nix/store/yp5x3f784nynkja7jh8hjs3dkb0y5kiz-php-posix-8.0.14/lib/php/extensions/posix.so' (tried: /nix/store/yp5x3f784nynkja7jh8hjs3dkb0y5kiz-php-posix-8.0.14/lib/php/extensions/posix.so (/nix/store/yp5x3f784nynkja7jh8hjs3dkb0y5kiz-php-posix-8.0.14/lib/php/extensions/posix.so: cannot allocate memory in static TLS block), /nix/store/5ixs2csccm589x0zqyj6ih0zbv0wjzsk-php-8.0.14/lib/php/extensions//nix/store/yp5x3f784nynkja7jh8hjs3dkb0y5kiz-php-posix-8.0.14/lib/php/extensions/posix.so.so (/nix/store/5ixs2csccm589x0zqyj6ih0zbv0wjzsk-php-8.0.14/lib/php/extensions//nix/store/yp5x3f784nynkja7jh8hjs3dkb0y5kiz-php-posix-8.0.14/lib/php/extensions/posix.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
I suspect this problem is somehow related to NixOS upgrades.
Any help would be appreciated, this happens many 2-3 times per week. I disabled auto-upgrades to see how it goes.
My initial thoughts are resource usage. How much RAM and swap does the server have? What is the regular load? Are you running any monitoring software like sysstat, netdata, zabbix, etc…? If so, what is your RAM and swap usage like around that time? Is this physical or virtual, and if virtual do you have access to the vm farm resource logs?
A couple more questions:
do you have any other services running large jobs around that time?
did you set environment.memoryAllocator.provider in your configuration?
how long have you been running this server for? when did this start happening?
Hopefully your answers will shed some light on the situation?
Thanks !
I can also wait a few days, the problem happens 2-3 times per week.
Right now, I’ll see how it does when the auto upgrades are disabled.
If that’s the root cause, I’ll try to find a way to quickly trigger the problem and open an issue.
Thanks!
There is no monitoring because it’s a server to host my blog, I don’t mind if it goes down for hours, I can live with it. I haven’t seen any RAM or swap issue so far.
But why do you think this is related to memory ? From what I see in logs, it looks like PHP fails to find posix.so.so which is why I suspect a problem with upgrades. I assume it is not properly switched to the new /nix/store layer.
There is no other service, just httpd.
I did not set environment.memoryAllocator.provider because I don’t know what it does.
This server have been running for about a month. Always had this problem.
Does this always happen the same time as when logrotate runs? Does it happen when there hasn’t been an upgrade? Maybe I’ll load this config in a container and let it run for a while… Can you provide me the most full copy of your configuration.nix as your able to?
It’s not related to upgrades for sure, because I disabled auto-upgrades.
I’m not sure this is related to logrotate but from what I can see, when PHP dies, it matches a logrotate execution. But the later is ran several times per day, and most of the time it’s okay.
I’d like to share my configuration.nix but I always change things.
Current change: disabled zramSwap just in case. I’ll wait for the next crash if any.
Next change: switch to another httpd mpm module (currently prefork).
I’ll try to disable logrotate later.
prefork is the only MPM which is supported with mod_php - enablePHP = true; enables mod_php. My practical recommendation without digging further is to abandon mod_php (a recommendation from upstream in fact) and switch to php-fpm using our NixOS module (services.phpfpm).
There are a number of examples in NixOS which demonstrate using apache and php-fpm together. If you get stuck with that I’d be happy to answer any questions and help get you running with a php-fpm based setup.
Ok so I guess there is no point to switch to another MPM.
The problem happened again today so it’s not related to zramSwap either. I just disabled logrotate to see how it goes. Next I’ll try to disable auto garbage collection.
I could indeed switch to php-fpm, in fact I have a working Nginx + fpm configuration ready but I find Apache + mod php easier and more readable.
Despite services.logrotate.enable = false; I can see that httpd gets reloaded every ~ 2 hours and logs have been rotated. I suspect this line overrides my settings and enables logrotate.
According to systemctl list-timers, garbage collection did not run.
That was my last option before switching to php-fpm, but I think I should create a bug report for this.
I’m going to try and reproduce your issue. Can you please paste your relevant configuration? I noticed that you have an incorrect definition of services.httpd.virtualHosts listed - virtualHosts takes an attribute set where the top level attribute is the host name.
Sorry about that, bad copy paste. In fact, I use a function to generate my vhosts, but I wont show this part for readability. Here is the relevant configuration:
Please log the issue on github, with steps to reproduce , that would at least give people with the same problem a chance of fixing it.
Might even be able to fix it.
This is a great example of flakeify your system, as if you did, and put it on a public repo, i could just build and switch to your configuration in a matter of seconds.