Unprivileged lxc container on NixOS

Help
1

I am trying to setup lxc (not lxd) on NixOS to run some unprivileged Ubuntu containers for development work. I have a working setup on an Ubuntu machine running multiple Ubuntu containers, but having trouble to replicate this on NixOS.

Here is the configuration.nix

{ config, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
      ./cachix.nix
    ];

  boot.loader.grub.device = "/dev/nvme0n1";

  networking.hostName = "nixos";
  networking.networkmanager.enable = true;
  networking.useDHCP = false;
  networking.interfaces.enp0s31f6.useDHCP = true;

  time.timeZone = "Pacific/Auckland";

  nixpkgs.config = { 
    allowUnfree = true;
  };

  environment.systemPackages = with pkgs; [
    wget emacs 
    gnupg
  ];

  virtualisation = {
    lxc = {
      enable = true;
      defaultConfig = ''
  	lxc.net.0.type = veth
        lxc.net.0.link = lxcbr0
  	lxc.net.0.flags = up
  	lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
  	lxc.idmap = u 0 100000 65535
  	lxc.idmap = g 0 100000 65535
      '';
      usernetConfig = ''
        flo veth lxcbr0 10
      '';
      lxcfs.enable = true;
    };
  };

  services.openssh.enable = true;

  users.users.flo = {
    isNormalUser = true;
    uid = 1000;
    group = "flo";
    extraGroups = [ "sudo" "wheel" "networkmanager" "audio"];
    subUidRanges = [
        { startUid = 100000; count = 65536; }
      ];
      subGidRanges = [
        { startGid = 100000; count = 65536; }
      ];
  };

  users.groups.flo.gid = 1000;
  users.groups.flo.members = [ "flo" ];

  system.stateVersion = "20.03";
}

The lxc part seems to set up the subuid, subguid and lxc-usernet settings.

[flo@nixos:~]$ cat /etc/subuid 
flo:100000:65536

[flo@nixos:~]$ cat /etc/subgid 
flo:100000:65536

[flo@nixos:~]$ cat /etc/lxc/lxc-usernet 
flo veth lxcbr0 10

The lxc default config is:

[flo@nixos:~]$ cat ~/.config/lxc/default.conf
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
lxc.idmap = u 0 100000 65535
lxc.idmap = g 0 100000 65535

Creating a container seems to work fine as well.

[flo@nixos:~]$ lxc-create -t download -n penguin -- -d ubuntu -r trusty -a amd64
Setting up the GPG keyring
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created an Ubuntu trusty amd64 (20200929_07:42) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.

[flo@nixos:~]$ lxc-ls -f
NAME    STATE   AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED 
penguin STOPPED 0         -      -    -    true

Problems start showing when starting the container.

[flo@nixos:~]$ lxc-start --name penguin -F
lxc-start: penguin: network.c: lxc_create_network_unpriv_exec: 2789 lxc-user-nic failed to configure requested network: Permission denied - Failed to create /run/lxc
                                                                                                                                                                    cmd/lxc_user_nic.c: 1218: main: Failed to create directory for db file
                                                                     lxc-start: penguin: start.c: lxc_spawn: 1846 Failed to create the network
                                                                                                                                              lxc-start: penguin: start.c: lxc_abort: 1122 Function not implemented - Failed to send SIGKILL to 1489

This seems to lock up lxc-create and leaves the system in a funny state.

Some things I noticed but don’t know how to solve:

  1. no lxcbr0 interface was created
  2. there is no lxc.service running (Not sure it needs to be there. But on Ubuntu it is running and I suspect it creates the lxcbr0 interface.)

Wondering if anyone has been running unprivileged lxc containers successfully and has some tips or can share their config?

1 Like