Ordering: filesystem mount before systemd.tmpfiles (before systemd-nspawn container)

Help
1

I have a config on my NixOS VPS with three relevant components that need to run in order:

  1. A file system mount
fileSystems."/persistent" = {
  device = ...;
  fsType = ...;
  options = ...;
};
  1. A systemd-tmpfiles config relating to that mount
systemd.tmpfiles.rules = [
  "d /persistent/my-data 700 myuser myuser -"
  "d /persistent/my-data-backup 700 myuser myuser -"
];
  1. A systemd-nspawn container that binds those directories
containers.my-service = {
  autoStart = true;
  ephemeral = true;

  bindMounts = {
    "/some/dir" = {
      hostPath = "/persistent/my-data";
      isReadOnly = false;
    };

    "/another/dir" = {
      hostPath = "/persistent/my-data-backup";
      isReadOnly = false;
    };
  };

  config = ...
};

How can I guarantee that these will run in order (if possible/necessary)? I know all three of these have systemd units. Is it sufficient / a good idea to override each of the units with systemd ordering directives? (See e.g. this SE issue.)

1 Like
2

You can see the dependencies of a service at runtime with systemctl list-dependencies --after container@my-service.service. In this case, it will be ordered after mounts and tmpfiles because it is a service unit and it has the default dependencies of service units described in man systemd.service, which includes After=sysinit.target, which itself is ordered after both local-fs.target and systemd-tmpfiles-setup.service.

2 Likes
3

Thanks @ElvishJerricco! That guarantees 3 runs after 1 & 2, but is 2 (tmpfiles) guaranteed to run after 1 (mounting)? I don’t see that in the systemd dependency tree using the command you gave.

EDIT: never mind, silly mistake on my part. Configuring it the way I said I did does result in the correct dependency tree.

1 Like
4

Actually, there still does seem to be an issue. I see

[username@nixos:~]$ sudo systemctl list-dependencies --after systemd-tmpfiles-setup.service
systemd-tmpfiles-setup.service
...
● └─local-fs.target
●   ├─-.mount
●   ├─persistent.mount
●   ├─run-keys.mount
●   ├─run-user-1000.mount
●   ├─run-wrappers.mount
○   ├─systemd-fsck-root.service
●   ├─systemd-growfs-root.service
●   ├─systemd-remount-fs.service
●   └─local-fs-pre.target
●     ├─systemd-remount-fs.service
●     ├─systemd-tmpfiles-setup-dev-early.service
●     └─systemd-tmpfiles-setup-dev.service

But when actually booting the system the tmpfiles commands do run before before persistent.mount (the directories are created and then overmounted). How can I fix the ordering?

5

Uh that definitely should not be happening. Can you share logs?

6

Weirdly, having rebooted the system several times, I’m unable to reproduce the issue (though I’m certain it happened—I used different directory names in the tmpfiles rules specifically to test for it).

7

Hello @Invariance-NaN, I was just debugging a very similar issue and noticed this:

Is it possible that you previously had /persistent configured with the "nofail" mount option? That would exclude persistent.mount from local-fs.target (and therefore systemd-tmpfiles-setup.service could have started before /persistent was mounted).

8

I’m having a similar issue and I think it’s because systemd-tmpfiles-setup.service is not set to run after remote-fs.target so there is no guarantee that tmpfiles are created after NFS (or other) is mounted (although it might happen just by chance).

Before-dependencies of my NFS mount:

~$ sudo systemctl list-dependencies --before mnt-remote.mount 
mnt-remote.mount
● ├─remote-fs.target
● │ └─systemd-user-sessions.service
○ └─umount.target

After-dependecies of systemd-tmpfiles-setup:

~$ sudo systemctl list-dependencies --after systemd-tmpfiles-setup.service
systemd-tmpfiles-setup.service
● ├─system.slice
● ├─systemd-journal-flush.service
● ├─systemd-journald.service
● ├─systemd-journald.socket
○ ├─systemd-sysusers.service
● └─local-fs.target
●   ├─-.mount
●   ├─run-keys.mount
●   ├─run-user-1000.mount
●   ├─run-wrappers.mount
○   ├─systemd-fsck-root.service
●   ├─systemd-growfs-root.service
●   ├─systemd-remount-fs.service
●   └─local-fs-pre.target
●     ├─systemd-remount-fs.service
●     ├─systemd-tmpfiles-setup-dev-early.service
●     └─systemd-tmpfiles-setup-dev.service

So systemd-tmpfiles-setup.service needs to be configured to run after remote-fs.target. Does anyone know how to do it?

9

Try this:

systemd.services.systemd-tmpfiles-setup = {
    wants = [ "remote-fs.target" ];
    after = [ "remote-fs.target" ];
};
10

Hello @lw91tnkac, this is a different issue, and it’s working as documented in systemd.mount(5). See also systemd.special and systemd-fstab-generator for related background.

I have a feeling that ordering systemd-tmpfiles-setup.service after remote-fs.target could be asking for trouble. For example, if some part of NFS needs tmpfiles.d setup, then you have a dependency cycle.

What might be better is to create your own systemd-tmpfiles-setup-mnt-remote.service with a separate config file given on the systemd-tmpfiles command line. I haven’t tried this myself yet though.

11

Yes, you are right! just tried it out and it does cause a dependency loop :frowning:

12

Ordering systemd-tmpfiles-setup.service before remote-fs.target does not work (Dependency cycle), so I ended up creating a new service instead:

systemd.services.remotefs-tmpfiles = {
      description = "Ensure System Files and Directories are created after remote fs are up.";
      path = ["systemd-tmpfiles"];
      script = "systemd-tmpfiles --create --remove --exclude-prefix=/dev";
      after = ["remote-fs.target"];
      requires = ["remote-fs.target"];
      before = ["my-service.service"];
      requiredBy = ["my-service.service"];
      serviceConfig = {
        Type = "oneshot";
        RemainAfterExit = "yes";
      };
    };

Maybe it would be also worth passing a different config file to systemd-tmpfiles.