Nginx logging to journald

Guides
1

Nginx, by default, logs to /var/spool/nginx. However, this can be changed per vhost:

  services.nginx.virtualhosts.<name>.extraConfig = ''
    error_log syslog:server=unix:/dev/log;
    access_log syslog:server=unix:/dev/log combined_host;
  '';

Perhaps we should update the default nginx config to do this, too?

A similar problem is capturing stderr from PHP-FPM. I think this is all that is required:

    services.phpfpm.pools.<name> = {
      extraConfig = ''
        catch_workers_output = yes
      '';
8 Likes
2

Does this differentiate between errors and non-errors? Usually journald add some metadata if output goes to stdout vs stderr. I think journald logging is better then our current log files since we do not have log-rotating in place. There are some people that migh rely on some externals tools that parse log files so a release note will be required.

3

We are using nginx quite a bit but that all goes to the journal. I have just checked a handful of servers and none of them have anything outside the journal.

That being said, I’m a huge fan of the principle of least surprise, and if we are shipping services that by default or in their typical usecases outside of the default write logs directly to disk, we definitely should ship sane defaults for log rotation.

1 Like
4

Those services should IMHO be fixed to log to systemd-journald.

6 Likes
5

globin,

That sounds like a fine solution too. Obviously there is high value in keeping things consistent, and pretty much everything else logs to journald so that makes sense.

The only exceptions would be rsyslog and syslog-ng. Would you see a problem with having those services add automatic rotation?

1 Like
6

We are using nginx quite a bit but that all goes to the journal. I have just checked a handful of servers and none of them have anything outside the journal.

Try /var/spool/nginx :wink:

1 Like
7

So I learned something new then… OK, I’m totally in favor of sane logrotate defaults and/or fixing nginx.

8

This was actually logged as an issue a while back nginx module doesn't log to journal · Issue #30732 · NixOS/nixpkgs · GitHub and here nginx -t -c: default configuration configured to write to read-only fs · Issue #34378 · NixOS/nixpkgs · GitHub

I don’t use nginx so won’t get around to fixing it likely. Anyone going to take this on before 19.03?

9

One thing about journald though: it throws away log lines when the throughput is too high. If people want to do accounting on the logs that won’t work, so it should be optional (but maybe on by default)

1 Like
10

Here’s an alternative configuration which logs to journald for all virtualhosts.

I also log errors to stderr rather than syslog, because it feels simpler (one fewer piece involved on the way to journald?). I don’t know if this has any practical consequences though.

Logging to stdout isn’t available for the access_log directive, sadly.

services.nginx.appendHttpConfig = ''
    error_log stderr;
    access_log syslog:server=unix:/dev/log combined;
'';
2 Likes
11

This is a very important point.
People may run with the assumption that nginx logs are always complete (as it was in the past).

If nginx suddenly started swallowing logs like systemd does, I’d certainly be very upset, as they are even more business-critical than normal system logs.

12

that is by default. Journald can be configured to not rate limit jobs, however malicious users may abuse this in certain ways. How To Change Log Rate Limiting In Linux

rsyslog has a rate limiting feature as well, however it is not enabled by default

1 Like
13

Oh wow, I never considered that option! The setting is NixOS - NixOS 21.05 manual

1 Like