Systemd config recommendations

Help
1

Background: I’m packaging some closed-source software to run on NixOS, and I’m having trouble with the systemd configuration options. I originally asked this question here, but I realize that the unorthodox handling of /opt/CrowdStrike makes this a more relevant question for NixOS users.

My configuration is below. Note: env is a buildFHSUserEnv drv related, and crowdstrike is an unpackged .deb with a few binaries I patchElf’d.

I’m having trouble primarily with two things:

  • /var/log/falconctl.log - Something (I’m not sure what exactly) seems to be symlinking this to /dev/stdout, which the systemd unit later complains about being “unable to open”:

Unable to open falconctl log file /var/log/falconctl.log, errno = 6

  • /opt/CrowdStrike - If I mkdir -p /opt/CrowdStrike in ExecStartPre I can get this to work, but this feels a bit too imperative to pass my smell test; I also don’t want /opt/CrowdStrike at the root of my filesystem if I can avoid it… What are the NixOS/systemd idioms for ensuring this directory exists (and persists) at /opt/CrowdStrike from the perspective of the unit? Could something like BindPaths be the answer? I halfheartedly tried this option, but I ran into “Namespace” errors with systemd, and I did not debug further in case I was traversing the wrong subtree…
systemd.services.falcon-sensor = {
  description = "CrowdStrike Falcon Sensor";                                                                                                                                               
  unitConfig.DefaultDependencies = false;                                                                                                                                                  
  after = [ "local-fs.target" ];                                                                                                                                                           
  conflicts = [ "shutdown.target" ];                                                                                                                                                       
  before = [ "shutdown.target" ];                                                                                                                                                          

  serviceConfig = {
    EnvironmentFile = cfg.envFile;                                                                                                                                                         
    StandardOutput = "journal";                                                                                                                                                            
    ExecStartPre = pkgs.writeShellScript "crowdstrike-prestart" ''
      mkdir -p /opt/CrowdStrike
      touch /var/log/falconctl.log
      ${env}/bin/setup -c "${crowdstrike}/opt/CrowdStrike/falconctl -s -f --cid=$CID"
    '';                                                                                                                                                                                    
    ExecStart = ''
      ${env}/bin/setup -c ${crowdstrike}/opt/CrowdStrike/falcond
    '';                                                                                                                                                                                    
    Type = "forking";                                                                                                                                                                      
    PIDFile = "/run/falcond.pid";                                                                                                                                                          
    Restart = "no";                                                                                                                                                                        
    TimeoutStopSec = "60s";                                                                                                                                                                
    KillMode = "control-group";                                                                                                                                                            
    KillSignal = "SIGTERM";                                                                                                                                                                
  };                                                                                                                                                                                       

  wantedBy = [ "multi-user.target" ];                                                                                                                                                      
};

Any suggestions or drive-by comments welcome :slight_smile:

2

I’m not convinced that that’s what’s happening here. /dev/stdout should be the stdout of the process (i.e. /proc/self/fd/1), if something relinks that then something’s going seriously wrong - I don’t even think it’s possible to change that as a non-root user. If the application is doing that, good luck packaging this ;p You might fare better running this as a non-root user.

The error you’re showing seems more like falconctl doesn’t have enough permissions to read from (and presumably write to) /var/log/falconctl.log, which may be due to it running through buildFhsUserEnv, which in turn will put your process into a namespace.

You have more or less two options.

The basic simple one of just creating /opt/CrowdStrike is to use systemd.tmpfiles, and using DependsOn and After on the unit for the tmpfiles (don’t recall the name off the top of my head). They’re called “tmpfiles”, but if you set the correct settings the directory won’t be cleaned up on reboot. This is how NixOS in general creates directories that are required at runtime.

The more complex, but IMO nicer, option for specific units rather than general runtime directories is to use RuntimeDirectory, which is systemd’s integrated way of managing per-unit directories. You can even combine it with DynamicUser to create a sandboxed environment for the process. This is very cool, but it might interfere with your fhs env, and your proprietary application might just not be flexible enough to bend to a sysadmin’s will. It’s worth a try, though.

This is similar to RuntimeDirectory, the reason you’re running into that error is most likely the namespacing I’ve mentioned. Both buildFhsUserEnv and systemd (when you’re using that setting) will create user namespaces, and they probably won’t give their subprocesses permissions to create more namespaces. You can convince systemd to give its child more permissions, but I’m not sure what the real effect of that is.

It’s possible that RuntimeDirectory doesn’t require a namespace, so perhaps that just works. I’ve never used it without DynamicUser, tell me if it does :slight_smile:

3

Thanks for all of the pointers.

The error you’re showing seems more like falconctl doesn’t have enough permissions to read from (and presumably write to) /var/log/falconctl.log , which may be due to it running through buildFhsUserEnv , which in turn will put your process into a namespace.

It seems that the binary is symlinking the logfile to /dev/stdout:

λ sudo rm /var/log/falconctl.log
λ systemctl restart falcon-sensor.service
λ la /var/log/falconctl.log
lrwxrwxrwx 11 root 20 Jul 08:49 /var/log/falconctl.log -> /dev/stdout

The binary remains unable to open the logfile, and I’m not sure how to further debug it…

It’s possible that RuntimeDirectory doesn’t require a namespace, so perhaps that just works. I’ve never used it without DynamicUser , tell me if it does

Unfortunately the binary depends on /opt/CrowdStrike existing at /opt/CrowdStrike (not /run/crowdstrike or somewhere else). systemd.tmpfiles seem to work fine for this use-case :slight_smile:

4

I ended up solving this with systemd.tmpfiles for /var/log/falconctl.log with 0660 perms, which I pulled from the openat syscall I saw in strace

2 Likes
5

That’s some behavior. Good that you’ve got it working at least :slight_smile:

6

I would love to see what you did here, if you’re still around.

I’m running into a lot of trouble with this app. Between this and the app wanting to write data to /opt/CrowdStrike, I have no idea how I’m going to get it working, if ta all.

7

Something like this should be what they did:

systemd.tmpfiles.settings = {
  "10-crowdstrike" = {
    "/opt/CrowdStrike" = {
      d = {
        group = "crowdstrike";
        user = "crowdstrike"
        mode = "0770";
      };
    };
    "/var/log/falconctl.log" = {
      f = {
        group = "crowdstrike";
        user = "crowdstrike";
        mode = "0660";
      };
    };
  };
}

Assuming you make your crowdstrike run as the user crowdstrike rather than root (might need to run it as root and change the owner/group if hat doesn’t work).

As long as you don’t specify an age, those files will not be cleaned up.

YMMV, it’s been a while, who knows if the tool still works the same.

8

Has anyone had any success with this?
I’m trying to debug this with nixosTest, and I’m still getting errno = 6 for falconctl.log with the group and user set to "root" in systemd.tmpfiles.settings.

Update: I got it to work by adding after = ["local-fs.target" "systemd-tmpfiles-setup.service"]; to the falcon-sensor service.

9

Any chance you’d be willing to post a redacted config for this? I’m working on the same thing in the next few days here and want to see a working example.

10

Yes, I am hoping to write a post on it (maybe on the NixOS wiki) if my employer doesn’t oppose.

Here’s a tip: falcon-sensor has a whitelist of supported kernels. So even if you get it working, it will likely run in reduced-functionality mode (essentially making it useless).
There’s a falcon-kernel-check binary, which I used with nixosTest to quickly try out different kernels and find a supported one.