Nginx crashes when trying to use modsecurity

I’m trying to use nginx with modsecurity by using the following config:

  services.nginx = {
    enable = true;
    package = (pkgs.nginxMainline.override { modules = with pkgs.nginxModules; [ modsecurity-nginx moreheaders dav ];});
 ...
  virtualHosts = {
      "${www_forumtest}" = {
        enableACME = true;
        forceSSL = true;
        locations."/" = { proxyPass = "http://localhost:4567/"; };
        extraConfig = ''
          modsecurity on;
          modsecurity_rules_file /etc/modsecurity/modsecurity_includes.conf;
        '';
      };
    };

and I have /etc/modsecurity/modsecurity_includes.conf written via

  environment.etc."modsecurity/modsecurity_includes.conf".text = ''
    Include /etc/modsecurity/modsecurity.conf
    Include /etc/modsecurity/crs-setup.conf
    # Include ${pkgs.modsecurity-crs}/rules/*.conf
  '';

(The last line refers to a package override which itself appears to work, i.e. installing it results in the necessary conf files)

As soon as I activate modsecurity and start nginx, it crashes with a coredump. Part of the log shows:

...
                                                     Found module libpthread.so.0 with build-id: 14ae19ad5edd0ebd4278dd6d7ed4d55175e9ce8d
                                                     Found module libdl.so.2 with build-id: 4f36ec2784ad2c6ebd30541dda8373f1626473ca
                                                     Found module nginx without build-id.
                                                     Stack trace of thread 83003:
                                                     #0  0x00007fa75e5645aa shmget (libc.so.6 + 0xfc5aa)
                                                     #1  0x00007fa75f00dc98 _ZN11modsecurity5utils11SharedFiles15add_new_handlerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPS7_ (libmodsecurity.so.3 + 0x1bcc98)
                                                     #2  0x00007fa75f00e0ee _ZN11modsecurity5utils11SharedFiles4openERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPS7_ (libmodsecurity.so.3 + 0x1bd0ee)
                                                     #3  0x00007fa75ef9f456 _ZN11modsecurity9debug_log14DebugLogWriter4openERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPS7_ (libmodsecurity.so.3 + 0x14e456)
                                                     #4  0x00007fa75ef2d515 _ZN2yy14seclang_parser5parseEv (libmodsecurity.so.3 + 0xdc515)
                                                     #5  0x00007fa75ef7c00e _ZN11modsecurity6Parser6Driver5parseERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_ (libmodsecurity.so.3 + 0x12b00e)
                                                     #6  0x00007fa75ef7c369 _ZN11modsecurity6Parser6Driver9parseFileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE (libmodsecurity.so.3 + 0x12b369)
                                                     #7  0x00007fa75ef9bd71 _ZN11modsecurity5Rules11loadFromUriEPKc (libmodsecurity.so.3 + 0x14ad71)
                                                     #8  0x00007fa75ef9bf33 msc_rules_add_file (libmodsecurity.so.3 + 0x14af33)
                                                     #9  0x000055e53a04806c ngx_conf_set_rules_file (nginx + 0xf306c)
                                                     #10 0x000055e539f99dc3 ngx_conf_parse (nginx + 0x44dc3)
                                                     #11 0x000055e539fc233c ngx_http_core_server (nginx + 0x6d33c)
                                                     #12 0x000055e539f99dc3 ngx_conf_parse (nginx + 0x44dc3)
                                                     #13 0x000055e539fbbcd3 ngx_http_block (nginx + 0x66cd3)
                                                     #14 0x000055e539f99dc3 ngx_conf_parse (nginx + 0x44dc3)
                                                     #15 0x000055e539f971e1 ngx_init_cycle (nginx + 0x421e1)
                                                     #16 0x000055e539f85231 main (nginx + 0x30231)
                                                     #17 0x00007fa75e48f780 __libc_start_main (libc.so.6 + 0x27780)
                                                     #18 0x000055e539f83c4a _start (nginx + 0x2ec4a)
░░ Subject: Process 83003 (nginx) dumped core
...

EDIT:

… after having been to hell and back to add debugging symbols to nginx and libmodsecurity (or maybe it’s just me to have a really hard time with the docs about this?!) I ended up with the following overrides:

{
  nixpkgs.overlays = [( self: super: {
   ...
    # apparently nixpkgs has no ready to deploy nginx-modsec, so we implement like here: https://git.selfprivacy.org/Izorkin/selfprivacy-nixos-config/src/branch/nginx-add-modsecurity/
    modsecurity-crs = pkgs.callPackage (pkgs.fetchurl {
        url = "https://git.selfprivacy.org/Izorkin/selfprivacy-nixos-config/raw/branch/nginx-add-modsecurity/generic/pkgs/modsecurity-crs/default.nix";
        sha256 = "0c10ywnwz9zaarvl2zmh3m3ds5bjmdn7c48v991kwgpiqncvjmk7";
    }) {};
    libmodsecurity = super.enableDebugging super.libmodsecurity;
    nginx = (super.enableDebugging super.nginxMainline).override {
      withDebug = true;
      withStream = true;
      modules = with super.nginxModules; [ modsecurity-nginx moreheaders dav ];
    };
  })];
}

and I was able to get a bit better backtrace:

Reading symbols from /nix/store/2diijh2ydl756snd5aghvqda3lfkwds8-nginx-1.21.4/bin/nginx...
[New LWP 3054597]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/kq5mfvarbjar95xahcf7kd5mfkc64qz3-glibc-2.33-78/lib/libthread_db.so.1".
Core was generated by `/nix/store/2diijh2ydl756snd5aghvqda3lfkwds8-nginx-1.21.4/bin/nginx -c /etc/ngin'.
Program terminated with signal SIGSYS, Bad system call.
#0  0x00007fe4cbab55aa in shmget () from /nix/store/kq5mfvarbjar95xahcf7kd5mfkc64qz3-glibc-2.33-78/lib/libc.so.6
(gdb) bt
#0  0x00007fe4cbab55aa in shmget () from /nix/store/kq5mfvarbjar95xahcf7kd5mfkc64qz3-glibc-2.33-78/lib/libc.so.6
#1  0x00007fe4cc55b7ab in modsecurity::utils::SharedFiles::add_new_handler (
    this=this@entry=0x7fe4cc6003d0 <modsecurity::utils::SharedFiles::getInstance()::instance>,
    fileName="/var/log/nginx/modsec_debug.log", error=error@entry=0x7ffd45a3a860) at utils/shared_files.cc:73
#2  0x00007fe4cc55bc48 in modsecurity::utils::SharedFiles::open (
    this=this@entry=0x7fe4cc6003d0 <modsecurity::utils::SharedFiles::getInstance()::instance>,
    fileName="/var/log/nginx/modsec_debug.log", error=error@entry=0x7ffd45a3a860) at utils/shared_files.cc:137
#3  0x00007fe4cc4f001a in modsecurity::debug_log::DebugLogWriter::open (
    this=this@entry=0x7fe4cc6003f8 <modsecurity::debug_log::DebugLogWriter::getInstance()::instance>,
    fileName="/var/log/nginx/modsec_debug.log", error=error@entry=0x7ffd45a3a860) at ../src/utils/shared_files.h:66
#4  0x00007fe4cc4ef439 in modsecurity::debug_log::DebugLog::setDebugLogFile (this=this@entry=0x5574ca6f7000,
    fileName="/var/log/nginx/modsec_debug.log", error=error@entry=0x7ffd45a3a860)
    at ../src/debug_log/debug_log_writer.h:40
#5  0x00007fe4cc46ab3d in yy::seclang_parser::parse (this=this@entry=0x7ffd45a3ac70) at seclang-parser.yy:1513
#6  0x00007fe4cc4cd32b in modsecurity::Parser::Driver::parse (this=this@entry=0x5574ca7330e0,
    f="Include /etc/modsecurity/modsecurity.conf\nInclude /etc/modsecurity/crs-setup.conf\nInclude /nix/store/3x7kpnihy0a3yf4acrhmawyyfpc1zyyf-modsecurity-crs-3.3.2/rules/*.conf\n", ref="/etc/modsecurity/modsecurity_includes.conf")
    at parser/driver.cc:154
#7  0x00007fe4cc4cd6f0 in modsecurity::Parser::Driver::parseFile (this=this@entry=0x5574ca7330e0,
    f="/etc/modsecurity/modsecurity_includes.conf") at parser/driver.cc:185
#8  0x00007fe4cc4ed04f in modsecurity::Rules::loadFromUri (this=this@entry=0x5574ca6f6220,
    uri=uri@entry=0x5574ca7319f1 "/etc/modsecurity/modsecurity_includes.conf") at rules.cc:98
#9  0x00007fe4cc4ed1fe in modsecurity::msc_rules_add_file (rules=0x5574ca6f6220,
    file=file@entry=0x5574ca7319f1 "/etc/modsecurity/modsecurity_includes.conf", error=error@entry=0x7ffd45a3b020)
    at rules.cc:346
#10 0x00005574c93c1d3e in ngx_conf_set_rules_file (cf=0x7ffd45a3b830, cmd=<optimized out>, conf=0x5574ca72f948)
    at /nix/store/pqbx61nmspn0n1smy3g32m56wrq2g9bw-modsecurity-nginx/src/ngx_http_modsecurity_module.c:342
#11 0x00005574c92f94f4 in ngx_conf_handler (cf=cf@entry=0x7ffd45a3b830, last=last@entry=0)
    at src/core/ngx_conf_file.c:463
#12 0x00005574c92f9977 in ngx_conf_parse (cf=cf@entry=0x7ffd45a3b830, filename=filename@entry=0x0)
    at src/core/ngx_conf_file.c:319
#13 0x00005574c9328484 in ngx_http_core_server (cf=0x7ffd45a3b830, cmd=<optimized out>, dummy=<optimized out>)
    at src/http/ngx_http_core_module.c:2938
#14 0x00005574c92f94f4 in ngx_conf_handler (cf=cf@entry=0x7ffd45a3b830, last=last@entry=1)
    at src/core/ngx_conf_file.c:463
#15 0x00005574c92f9977 in ngx_conf_parse (cf=cf@entry=0x7ffd45a3b830, filename=filename@entry=0x0)
    at src/core/ngx_conf_file.c:319
#16 0x00005574c9323012 in ngx_http_block (cf=0x7ffd45a3b830, cmd=<optimized out>, conf=<optimized out>)
    at src/http/ngx_http.c:239
#17 0x00005574c92f94f4 in ngx_conf_handler (cf=cf@entry=0x7ffd45a3b830, last=last@entry=1)
    at src/core/ngx_conf_file.c:463
#18 0x00005574c92f9977 in ngx_conf_parse (cf=cf@entry=0x7ffd45a3b830, filename=filename@entry=0x5574ca6b62b8)
    at src/core/ngx_conf_file.c:319
#19 0x00005574c92f6ae1 in ngx_init_cycle (old_cycle=old_cycle@entry=0x7ffd45a3b9e0) at src/core/ngx_cycle.c:284
#20 0x00005574c92e4493 in main (argc=4, argv=0x7ffd45a3bd88) at src/core/nginx.c:292
(gdb)

I’m however not sure how to read anything suspicious in there, except for /var/log/nginx/modsec_debug.log maybe, but that’s writable by nginx but remains at 0 bytes…

The only thing I could think of was some permission issues in files that the module is trying to handle, but I couldn’t find anything suspicious. Any ideas?

Solution by @Izorkin (thanks again!) add:

systemd.services.nginx.serviceConfig.SystemCallFilter = lib.mkForce "~@cpu-emulation @debug @keyring @mount @obsolete @privileged @setuid";

:pray:

You want to use the following option instead of overwriting the package. moreheaders and dav are already included by default.

services.nginx = {
  package = pkgs.nginxMainline;
  additionalModules = with pkgs.nginxModules; [ modsecurity-nginx ];
};

That is already the default by now and can be removed.

1 Like