LXC configuration that persists across rebuilds?

Help
1

Update: I have tmpfs on root, and in order to make this work, all of /var has to be on disk, not on tmpfs. Nor can /var/lib be on disk and the rest of /var on tmpfs, all of /var must be on disk. I’ve tried all sorts of configurations and this is the only one that works that keeps / in tmpfs.

I’m using LXC/LXD on my NixOS host to run some things in an Alpine Linux guest.

This works fine, except when I rebuild NixOS, all my LXC configuration is lost. I have to rerun lxd init and set it all back up again from scratch.

LXC stores some of its configuration in files, and some in a database. It’s not clear from NixOS docs how to configure it to persist across rebuilds. Has anyone figured this out?

My current LXC setup in configuration.nix:

virtualisation = {
  containers.enable = true;
  containerd.enable = true;
  libvirtd = {
    enable = true;
    qemuRunAsRoot = false;
  };
  lxd = {
    enable = true;
    zfsSupport = true;
    recommendedSysctlSettings = true;
  };
  lxc = {
    enable = true;
    lxcfs.enable = true; 
    systemConfig =  
      ''
        lxc.lxcpath = /var/lib/lxd/storage-pools  # The location in which all containers are stored.
        lxc.bdev.zfs.root = rpool/safe/lxd  # Default ZFS root dataset name.
      '';
  };
};
2

From what I can see the from the available configuration options, the LXD configuration is not managed by NixOS. If something is not covered by the configuration module you have two choices:

  1. Manage configuration the configuration with LXD own tools and backup and restore it on your own;
  2. Extend the configuration options so that its configuration is integrated with NixOS. The manual can tell you how to do that. If you stumble upon something, we are here to help :wink: .

I must say that the latter may be complicated by the fact that LXD does not use plain files but instead some “distributed” variant of SQLite but I see that it allows to initialize its configuration from a YAML file so you may be lucky.

1 Like
3

Thank you, yes I’m hoping I can specify enough of the config using virtualization.lxc.{systemConfig,defaultConfig,usernetConfig} to point lxc/lxd to that database on startup, and just persist that database across rebuilds. Or actually to persist /var/lib/{lxc/,lxcfs/,lxd/} across rebuilds.

Speaking of which, is there a simple way to persist /var/lib/{lxc/,lxd/,lxcfs/} to /persist/var/{lxc/,lxd/,lxcfs/}? I already do this with /etc config files using environment.etc."config-loc".source = "/persist/etc/config-loc/";

But it’s not clear how to do this with something outside of /etc. There’s filesystems, but that seems more for specifying which filesystems to mount, eg overkill for this.

4

Are you using an erase your darlings setup?

Usually /var and /lib as well as everything else is persistent across rebuilds and reboot.

5

I’m using a hybrid Erase Your Darlings setup, except with / on tmpfs instead of a zfs dataset. Everything else on zfs.

However, Graham doesn’t persist all of /var in his example, just /var/lib/bluetooth and /var/lib/acme, using systemd.tmpfiles.rules:

{
  systemd.tmpfiles.rules = [
    "L /var/lib/bluetooth - - - - /persist/var/lib/bluetooth"
  ];
}

I tried that with /var/lib/{lxc,lxcfs,lxd}:

  systemd.tmpfiles.rules = [
    "L /var/lib/bluetooth - - - - /persist/var/lib/bluetooth" 
    "L /var/lib/acme - - - - /persist/var/lib/acme"
    "L /var/lib/lxc - - - - /persist/var/lib/lxc" 
    "L /var/lib/lxcfs - - - - /persist/var/lib/lxcfs"
    "L /var/lib/lxd - - - - /persist/var/lib/lxd"
  ];

But there’s a problem. lxd/lxc can’t write to any of those three directories if they’re persisted to /persist/var/lib/{lxc,lxcfs,lxd} - permission denied. Even when running as root.

Haven’t had time to figure that one out yet.

6

This would have been relevant information from the beginning. Your system is built to not have state beyond the state that is explicitely mentioned elsewhere.

What are the permissions on the link targets?

Running what exactly as root?

7

This would have been relevant information from the beginning. Your system is built to not have state beyond the state that is explicitely mentioned elsewhere.

Yes sorry, every time I checked /var/lib after a reboot/rebuild, the directory structure was still there, as if NixOS was automatically persisting it. I didn’t realize all files in the /var subdirectories were wiped though.

What are the permissions on the link targets?

I got this fixed, seems the link targets need to be 755.

8

So for now I’ve moved /var out of tmpfs and onto its own zfs dataset, rpool/local/var. That, along with persisting /etc/lxc/{default.conf,lxc-usernet,lxc.conf} to /persist/etc/lxc/{same} has solved the problem. LXC config now persists across reboots/rebuilds.

I would still like to find a way to keep /var in tmpfs, and explicitly state what gets persisted. findmnt now shows this topology for the working setup:

├─/var rpool/local/var zfs rw,relatime,xattr,posixacl
│ ├─/var/lib/lxcfs lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
│ ├─/var/lib/lxd/shmounts tmpfs tmpfs rw,relatime,size=100k,mode=711
│ ├─/var/lib/lxd/devlxd tmpfs tmpfs rw,relatime,size=100k,mode=755
│ └─/var/lib/lxd/storage-pools/lxdpool/containers/ae-test rpool/safe/lxd/containers/ae-test zfs rw,noatime,xattr,posixacl

Looks like /var/lib/lxcfs needs to be mounted to its own fuse.lxcfs filesystem, which is probably why it didn’t like being persisted to zfs earlier.

And instead of persisting all of /var/lib/lxd, it may be enough to persist only /var/lib/lxd/storage-pools/lxdpool/containers/, or preferably just /var/lib/lxd/storage-pools/.

Will try this later and update.

9

Ok, gave that a try but still can’t get it to work. I’ve tried everything I can think of at the moment, but need a deeper understanding of LXC to figure this out.

For now I will just go with the simple setup that worked - move /var out of tmpfs and onto its own ZFS dataset.

For future reference, in case anyone else has this problem, here’s a wall of text of what I tried. Don’t feel obligated to read it unless you’re just really curious or are working on the same problem yourself.

  1. Objective: put all of root / on tmpfs, including /var, while persisting enough of lxc/lxd to rpool/safe/persist so that lxc/lxd continues to work across reboots/rebuilds
  2. the crucial lxc/lxd locations are:
    • /var/lib/lxc
    • /var/lib/lxcfs (fuse.lxcfs filesystem for providing host system information to running containers)
    • /var/lib/lxd (holds the actual storage pools, containers, and vms)
    • /var/log/lxd
    • /etc/lxc (system config)
    • ~/.config/lxc (unprivileged config)
  3. Plan:
    • persist /var/lib/{lxc,lxd} to /persist/var/lib/{lxc,lxd}
    • allow NixOS and lxc/lxd to automatically manage /var/lib/lxcfs (it is auto mounted to device: lxcfs of type: fuse.lxcfs)
    • persist /etc/lxc/{default.conf,lxc-usernet,lxc.conf} to /persist/etc/lxc/{default.conf,lxc-usernet,lxc.conf}

Relevant lines in configuration.nix:

{ config, pkgs, ... }:

{
...
  virtualisation = {
    containers.enable = true;
    containerd.enable = true;
    libvirtd = {
      enable = true;
      qemuRunAsRoot = false;
    };
    lxd = {
      enable = true;
      zfsSupport = true;
      recommendedSysctlSettings = true;
    };
    lxc = {
      enable = true;
      lxcfs.enable = true;
      systemConfig = ''
        lxc.lxcpath = /var/lib/lxd/containers
        lxc.bdev.zfs.root = rpool/safe/lxd
      '';
    };
  };
...
  environment.etc = {
    "lxc/default.conf".source = "/persist/etc/lxc/default.conf";
    "lxc/lxc-usernet".source = "/persist/etc/lxc/lxc-usernet";
    "lxc/lxc.conf".source = "/persist/etc/lxc/lxc.conf";
  };
...
  systemd.tmpfiles.rules = [
    "L /var/lib/lxc - - - - /persist/var/lib/lxc"
    "L /var/lib/lxd - - - - /persist/var/lib/lxd"
  ];
...
};

hardware-configuration.nix for this is pretty standard.

lxd init completes without error and creates the ZFS dataset (rpool/safe/lxd), the lxd storage pool (/var/lib/lxd/storage-pools/lxdpool), and the network bridge (lxdbr0).

But attempting to launch a container results in an error that does not occur when all /var is on disk (on its own ZFS dataset, rpool/local/var).

full details

$> sudo lxd init                             
[sudo] password for bgibson: 
Would you like to use LXD clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: lxdpool
Name of the storage backend to use (btrfs, dir, lvm, zfs) [default=zfs]: 
Would you like to create a new zfs dataset under rpool/lxd? (yes/no) [default=yes]: no
Create a new ZFS pool? (yes/no) [default=yes]: no
Name of the existing ZFS pool or dataset: rpool/safe/lxd
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]:  
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the LXD server to be available over the network? (yes/no) [default=no]: yes
Address to bind LXD to (not including port) [default=all]: 
Port to bind LXD to [default=8443]: 
Trust password for new clients: 
Again: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes] 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

$> sudo lxc launch images:alpine/edge ae-test
Creating ae-test
Starting ae-test                            
Error: Failed to run: /nix/store/9kp22pvvgn376q6jqhvi8agqwqzbg3a2-lxd-4.14/bin/.lxd-wrapped forkstart ae-test /var/lib/lxd/containers /var/log/lxd/ae-test/lxc.conf: 
Try `lxc info --show-log local:ae-test` for more info

$> sudo lxc info --show-log local:ae-test    
Name: ae-test
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/08/10 04:44 UTC
Status: Stopped
Type: container
Profiles: default

Log:

lxc ae-test 20210810044427.200 WARN     conf - conf.c:lxc_map_ids:3007 - newuidmap binary is missing
lxc ae-test 20210810044427.201 WARN     conf - conf.c:lxc_map_ids:3013 - newgidmap binary is missing
lxc ae-test 20210810044427.203 WARN     conf - conf.c:lxc_map_ids:3007 - newuidmap binary is missing
lxc ae-test 20210810044427.203 WARN     conf - conf.c:lxc_map_ids:3013 - newgidmap binary is missing
lxc ae-test 20210810044427.204 WARN     cgfsng - cgroups/cgfsng.c:fchowmodat:1293 - No such file or directory - Failed to fchownat(43, memory.oom.group, 65536, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc ae-test 20210810044427.249 ERROR    conf - conf.c:lxc_setup_rootfs_prepare_root:3437 - Failed to setup rootfs for
lxc ae-test 20210810044427.249 ERROR    conf - conf.c:lxc_setup:3600 - Failed to setup rootfs
lxc ae-test 20210810044427.249 ERROR    start - start.c:do_start:1265 - Failed to setup container "ae-test"
lxc ae-test 20210810044427.249 ERROR    sync - sync.c:sync_wait:36 - An error occurred in another process (expected sequence number 5)
lxc ae-test 20210810044427.256 WARN     network - network.c:lxc_delete_network_priv:3621 - Failed to rename interface with index 0 from "eth0" to its initial name "veth2065dbfa"
lxc ae-test 20210810044427.257 ERROR    start - start.c:__lxc_start:2073 - Failed to spawn container "ae-test"
lxc ae-test 20210810044427.257 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:868 - Received container state "ABORTING" instead of "RUNNING"
lxc ae-test 20210810044427.257 WARN     start - start.c:lxc_abort:1016 - No such process - Failed to send SIGKILL via pidfd 44 for process 7938
lxc ae-test 20210810044427.399 WARN     conf - conf.c:lxc_map_ids:3007 - newuidmap binary is missing
lxc ae-test 20210810044427.400 WARN     conf - conf.c:lxc_map_ids:3013 - newgidmap binary is missing
lxc 20210810044427.419 ERROR    af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:207 - Connection reset by peer - Failed to receive response
lxc 20210810044427.420 ERROR    commands - commands.c:lxc_cmd_rsp_recv_fds:129 - Failed to receive file descriptors

By comparison, here is what it lxd init looks like when all of /var is mounted to disk (rpool/local/var) instead of tmpfs:

> sudo lxd init                                                           master [81c71d8] deleted modified untracked
[sudo] password for bgibson: 
Would you like to use LXD clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: lxdpool
Name of the storage backend to use (dir, lvm, zfs, btrfs) [default=zfs]: 
Would you like to create a new zfs dataset under rpool/lxd? (yes/no) [default=yes]: no
Create a new ZFS pool? (yes/no) [default=yes]: no
Name of the existing ZFS pool or dataset: rpool/safe/lxd
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the LXD server to be available over the network? (yes/no) [default=no]: yes
Address to bind LXD to (not including port) [default=all]: 
Port to bind LXD to [default=8443]: 
Trust password for new clients: 
Again: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes] 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

bgibson@z10pe-d8:~ (*)
> sudo lxc launch images:alpine/edge ae-test                              master [81c71d8] deleted modified untracked
Creating ae-test
Starting ae-test                            

bgibson@z10pe-d8:~ (*)
> sudo lxc list                                                           master [81c71d8] deleted modified untracked
+---------+---------+----------------------+----------------------------------------------+-----------+-----------+
|  NAME   |  STATE  |         IPV4         |                     IPV6                     |   TYPE    | SNAPSHOTS |
+---------+---------+----------------------+----------------------------------------------+-----------+-----------+
| ae-test | RUNNING | 10.227.20.113 (eth0) | fd42:93b1:cd1:18e7:216:3eff:fe6e:b3bd (eth0) | CONTAINER | 0         |
+---------+---------+----------------------+----------------------------------------------+-----------+-----------+
1 Like