Httpd + php = blank pages

src386 · February 2, 2022, 6:42pm

Hey !

I run a simple webserver (httpd + php) that works well, but sometimes my websites just display a blank page.

First, here is my configuration:

services = {
    httpd = {
      enable = true;
      mpm = "prefork";
      adminAddr = "hostmaster@example.org";
      enablePHP = true;
      phpPackage = pkgs.php80;
      phpOptions = "display_errors = off";
      extraConfig = ''
        SetOutputFilter DEFLATE

        ServerSignature Off
        ServerTokens prod
        
        <FilesMatch "\.(ico|pdf|jpg|jpeg|JPEG|png|gif|js|css)$">
          Header set Cache-Control "max-age=3600, public"
        </FilesMatch>
      '';
      extraModules = [
        "deflate"
      ];
      virtualHosts =
        { documentRoot = "/var/www/html";
          forceSSL = true;
          enableACME = true;
          locations."/" = {
            index = "index.php index.html";
          };
          extraConfig = ''
            <Directory "/var/www/html">
              AllowOverride All
            </Directory>
          '';
        };
    };
  };

I did some troubleshot and here is what I’ve found:

At 4:40 AM, NixOS was updated, which is normal.
At 17:05 PM, httpd was reloaded by logrotate
At 17:05 PM, I get PHP error messages in logs

PHP logs:

[Wed Feb 02 17:05:13.246275 2022] [http2:warn] [pid 710] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
PHP Warning:  PHP Startup: Unable to load dynamic library '/nix/store/yp5x3f784nynkja7jh8hjs3dkb0y5kiz-php-posix-8.0.14/lib/php/extensions/posix.so' (tried: /nix/store/yp5x3f784nynkja7jh8hjs3dkb0y5kiz-php-posix-8.0.14/lib/php/extensions/posix.so (/nix/store/yp5x3f784nynkja7jh8hjs3dkb0y5kiz-php-posix-8.0.14/lib/php/extensions/posix.so: cannot allocate memory in static TLS block), /nix/store/5ixs2csccm589x0zqyj6ih0zbv0wjzsk-php-8.0.14/lib/php/extensions//nix/store/yp5x3f784nynkja7jh8hjs3dkb0y5kiz-php-posix-8.0.14/lib/php/extensions/posix.so.so (/nix/store/5ixs2csccm589x0zqyj6ih0zbv0wjzsk-php-8.0.14/lib/php/extensions//nix/store/yp5x3f784nynkja7jh8hjs3dkb0y5kiz-php-posix-8.0.14/lib/php/extensions/posix.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0

I suspect this problem is somehow related to NixOS upgrades.

Any help would be appreciated, this happens many 2-3 times per week. I disabled auto-upgrades to see how it goes.

nixinator · February 3, 2022, 12:39am

write a script that continuously curls the web server.

write another script that restarts httpd constantly.

you may be able to reproduce the problem that way.

If you can reproduce it, open an issue on github, so it can be reproduced.

have fun. (reproducing)

aanderse · February 3, 2022, 3:08am

My initial thoughts are resource usage. How much RAM and swap does the server have? What is the regular load? Are you running any monitoring software like sysstat, netdata, zabbix, etc…? If so, what is your RAM and swap usage like around that time? Is this physical or virtual, and if virtual do you have access to the vm farm resource logs?

A couple more questions:

do you have any other services running large jobs around that time?
did you set environment.memoryAllocator.provider in your configuration?
how long have you been running this server for? when did this start happening?

Hopefully your answers will shed some light on the situation?

src386 · February 3, 2022, 10:10am

Thanks !
I can also wait a few days, the problem happens 2-3 times per week.
Right now, I’ll see how it does when the auto upgrades are disabled.
If that’s the root cause, I’ll try to find a way to quickly trigger the problem and open an issue.

src386 · February 3, 2022, 10:15am

Thanks!
There is no monitoring because it’s a server to host my blog, I don’t mind if it goes down for hours, I can live with it. I haven’t seen any RAM or swap issue so far.
But why do you think this is related to memory ? From what I see in logs, it looks like PHP fails to find posix.so.so which is why I suspect a problem with upgrades. I assume it is not properly switched to the new /nix/store layer.

There is no other service, just httpd.
I did not set environment.memoryAllocator.provider because I don’t know what it does.
This server have been running for about a month. Always had this problem.

nixinator · February 3, 2022, 11:15am

You can see if those /nix/store paths actually exist.

I also found this, but it’s probably just a red herring, this will need more debugging if it can be reproduced.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964141. (a very quick glance it looks like it’s arm specific).

If those files actually exist, then perhaps open an issue, and give more information about your system

src386 · February 5, 2022, 10:16am

I disabled NixOS auto upgrades, but the problem happened again, so that was not the root cause.

The .so files that PHP is not able to find are present on the given paths, so I don’t understand.

I’ll dig into logs to find out what happened, and try to tweak other parameters.

aanderse · February 5, 2022, 12:06pm

Does this always happen the same time as when logrotate runs? Does it happen when there hasn’t been an upgrade? Maybe I’ll load this config in a container and let it run for a while… Can you provide me the most full copy of your configuration.nix as your able to?

src386 · February 5, 2022, 1:14pm

It’s not related to upgrades for sure, because I disabled auto-upgrades.
I’m not sure this is related to logrotate but from what I can see, when PHP dies, it matches a logrotate execution. But the later is ran several times per day, and most of the time it’s okay.
I’d like to share my configuration.nix but I always change things.
Current change: disabled zramSwap just in case. I’ll wait for the next crash if any.
Next change: switch to another httpd mpm module (currently prefork).
I’ll try to disable logrotate later.

aanderse · February 5, 2022, 1:35pm

prefork is the only MPM which is supported with mod_php - enablePHP = true; enables mod_php. My practical recommendation without digging further is to abandon mod_php (a recommendation from upstream in fact) and switch to php-fpm using our NixOS module (services.phpfpm).

There are a number of examples in NixOS which demonstrate using apache and php-fpm together. If you get stuck with that I’d be happy to answer any questions and help get you running with a php-fpm based setup.

Let me know.

src386 · February 7, 2022, 10:19am

Ok so I guess there is no point to switch to another MPM.

The problem happened again today so it’s not related to zramSwap either. I just disabled logrotate to see how it goes. Next I’ll try to disable auto garbage collection.

I could indeed switch to php-fpm, in fact I have a working Nginx + fpm configuration ready but I find Apache + mod php easier and more readable.

src386 · February 9, 2022, 10:11pm

Happened again, blank PHP pages.

Despite services.logrotate.enable = false; I can see that httpd gets reloaded every ~ 2 hours and logs have been rotated. I suspect this line overrides my settings and enables logrotate.

According to systemctl list-timers, garbage collection did not run.

That was my last option before switching to php-fpm, but I think I should create a bug report for this.

aanderse · February 10, 2022, 3:43am

I’m going to try and reproduce your issue. Can you please paste your relevant configuration? I noticed that you have an incorrect definition of services.httpd.virtualHosts listed - virtualHosts takes an attribute set where the top level attribute is the host name.

src386 · February 13, 2022, 10:35pm

Sorry about that, bad copy paste. In fact, I use a function to generate my vhosts, but I wont show this part for readability. Here is the relevant configuration:

security = {
  acme = {
    email = "hostmaster@example.org";
    acceptTerms = true;
  };
};

services = {
  httpd = {
    enable = true;
    mpm = "prefork";
    adminAddr = "hostmaster@example.org";
    enablePHP = true;
    phpPackage = pkgs.php80;
    phpOptions = "display_errors = off";
    extraConfig = ''
      SetOutputFilter DEFLATE

      ServerSignature Off
      ServerTokens prod
      
      <FilesMatch "\.(ico|pdf|jpg|jpeg|JPEG|png|gif|js|css)$">
        Header set Cache-Control "max-age=3600, public"
      </FilesMatch>
    '';
    extraModules = [
      "deflate"
    ];
    virtualHosts = {
      "myblog.example.org" = {
        documentRoot = "/var/www/html/myblog.example.org";
        forceSSL = true;
        enableACME = true;
        locations."/" = {
          index = "index.php index.html";
        };
        extraConfig = ''
          <Directory "/var/www/html/myblog.example.org">
            AllowOverride All
          </Directory>
        '';
      };
    };
  };
};

Then chown -R wwwrun:root /var/www/html/myblog.example.org where I installed Pluxml.

I will move all of this to a test environment. I will try php-fpm on the main server instead.

src386 · February 16, 2022, 8:13am

So far no problem since I moved to httpd + php-fpm.

However I’ve been able to reproduce it on my test environment, using httpd + mod_fpm.

nixinator · February 18, 2022, 2:55am

Please log the issue on github, with steps to reproduce , that would at least give people with the same problem a chance of fixing it.

Might even be able to fix it.

This is a great example of flakeify your system, as if you did, and put it on a public repo, i could just build and switch to your configuration in a matter of seconds.