Issue with snx-rs service running differently as systemd vs manual sudo

[martinhrvn]

I’m trying to set up snx-rs as a systemd service in NixOS, but I’m encountering an unusual issue with client access.

The problem

When I run snx-rs manually with sudo snx-rs -m command, everything works perfectly:

• The daemon starts
• I can run snxctl connect as my regular user
• The VPN connection works using my config in ~/.config/snx-rs/snx-rs.conf

However, when I set up the exact same command as a systemd service running as root:

• The service itself starts without errors
• When I try to run snxctl connect as my user, I get Error: No such file or directory (os error 2)
• I have even tried to run snx-rs daemon using sudo and root account, in both cases it works fine.
• It is not possible to run it as my user

What I’ve tried

My current service configuration:

systemd.services.snx-rs = {
  description = "SNX-RS Service";
  wantedBy = [ "multi-user.target" ];
  after = [ "network.target" ];
  serviceConfig = {
    Type = "simple";
    ExecStart = "${pkgs.snx-rs}/bin/snx-rs -m command";
    User = "root";
    Restart = "on-failure";
  };
};

I suspect this has to do with systemd service isolation in NixOS, since the same command works fine when run manually as root but fails when run through systemd.

Questions

1. How can I make the systemd service behave the same as when I run the command manually with sudo?
2. What specific systemd isolation settings might be preventing the client from connecting to the daemon?

I am not sure how snx-rs actually work internally, but I would assume the snxctl reads the config and sends the config options to the daemon, but it very well may be that the daemon tries to read the config.

Any suggestions would be appreciated!

[TLATER]

@martinhrvn

What specific systemd isolation settings might be preventing the client from connecting to the daemon?

None, since you didn’t configure any.

Many NixOS modules create systemd services that do, but since you’re doing this yourself there is no hidden magic that would break this.

@martinhrvn

How can I make the systemd service behave the same as when I run the command manually with sudo?

That’s quite hard. The difference is that sudo will retain some of your user’s environment variables, and execute the root user’s rcfiles. This means that there are a bunch of things set which systemd can’t reasonably have (and should not).

This is the service defined by the project, by the way, making yours as close as possible to this is probably helpful: snx-rs/assets/snx-rs.service at 107d7ae26f640da1012b83594e8a5a5bc5b55fee · ancwrd1/snx-rs · GitHub

@martinhrvn

I am not sure how snx-rs actually work internally, but I would assume the snxctl reads the config and sends the config options to the daemon

Looking at the project readme, that appears to be accurate.

Having looked a little at their code, it appears to connect to the local server via UDP. This is unlikely to throw any file access errors.

Something seems very broken here, to the point that I would suspect you’re using a different snxctl in the two scenarios. It would probably help to strace this to figure out what file it’s trying - and failing - to access.

Note that the error happens on the client side, so I don’t think the issue is necessarily with the service.

[martinhrvn]

Thanks for the great reply.

I have run strace for both when I run the daemon manually vs using systemd, but was not able to figure out any major differences.

But I’m not sure what to do about it. It’s not a deal breaker if it doesn’t work as systemd service, but I believe it’s a great learning opportunity so I want to understand why is this the case.

Yes, it is true that it’s happening client side, but when I stop the service and run the snx-rs manually then it works.

[TLATER]

@martinhrvn

I have run strace for both when I run the daemon manually vs using systemd, but was not able to figure out any major differences.

It’s not about differences. Somewhere near the end of the strace log you’ll have a system call to open a file, and that system call fails. It will tell you which file was attempted to be opened.

If you can figure out what the file is, we can work backwards from there.

You can use strace -e trace=file to limit it to only file-related syscalls (though if you don’t find anything in there, it might be that you’re running a binary not built by nix somehow and the interpreter is wrong).

[martinhrvn]

Alright I’m not sure if that’s what I can say from the two straces but it seems to fail during this:

statx(AT_FDCWD, "/etc/pki/tls/certs/ca-bundle.crt", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID|STATX_SUBVOL, stx_attributes=0, stx_mode=S_IFREG|0555, stx_size=520166, ...}) = 0
statx(AT_FDCWD, "/etc/pki/tls/certs", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID|STATX_SUBVOL, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=26, ...}) = 0
openat(AT_FDCWD, "/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = 9
newfstatat(AT_FDCWD, "/nix/store/954l60hahqvr0hbs7ww6lmgkxvk8akdf-openssl-3.4.1/etc/ssl/certs/0feb9fd6.0", 0x7ffc3bc6c0c0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/etc/pki/tls/certs/0feb9fd6.0", 0x7ffc3bc6c0c0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 10
+++ exited with 1 +++

(the errors for openssl opening the cert happens in manual as well) so I just assume it’s the /etc/lcoaltime

here is the strace for when I run manual snx-rs:

statx(AT_FDCWD, "/etc/pki/tls/certs", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID|STATX_SUBVOL, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=26, ...}) = 0
openat(AT_FDCWD, "/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = 9
newfstatat(AT_FDCWD, "/nix/store/954l60hahqvr0hbs7ww6lmgkxvk8akdf-openssl-3.4.1/etc/ssl/certs/0feb9fd6.0", 0x7ffe21a6d150, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/etc/pki/tls/certs/0feb9fd6.0", 0x7ffe21a6d150, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 10
statx(AT_FDCWD, "/.dockerenv", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7ffe21a6ff70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/proc/self/cgroup", O_RDONLY|O_CLOEXEC) = 10
statx(10, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0444, stx_size=0, ...}) = 0
openat(AT_FDCWD, "/proc/sys/kernel/osrelease", O_RDONLY|O_CLOEXEC) = 10
statx(10, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0444, stx_size=0, ...}) = 0
openat(AT_FDCWD, "/proc/version", O_RDONLY|O_CLOEXEC) = 10
statx(10, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0444, stx_size=0, ...}) = 0
openat(AT_FDCWD, "/dev/null", O_RDONLY|O_CLOEXEC) = 10
openat(AT_FDCWD, "/dev/null", O_WRONLY|O_CLOEXEC) = 11
openat(AT_FDCWD, "/dev/null", O_WRONLY|O_CLOEXEC) = 12
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=437052, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
statx(AT_FDCWD, "/etc/localtime", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID|STATX_SUBVOL, stx_attributes=0, stx_mode=S_IFLNK|0777, stx_size=31, ...}) = 0
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 10
statx(10, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID|STATX_SUBVOL, stx_attributes=0, stx_mode=S_IFREG|0444, stx_size=2301, ...}) = 0
+++ exited with 0 +++

in both cases the binary for the snxctl and snx-rs comes from the package snx-rs…
/run/current-system/sw/bin/snx-rs -> /nix/store/i4cch9wwakc1m8yrm97712bf7ips0dk2-snx-rs-3.1.1/bin/snx-rs
/run/current-system/sw/bin/snxctl -> /nix/store/i4cch9wwakc1m8yrm97712bf7ips0dk2-snx-rs-3.1.1/bin/snxctl