Running NixOS 18.09 in an LXD 3.8 container

asbachb · January 2, 2019, 11:46pm

I’m currently trying to create a working NixOS container with LXD.

Based on (Run NixOS inside LXC container on Ubuntu 14.04 as host · Issue #9735 · NixOS/nixpkgs · GitHub) I was able to create running container:

Downloaded the latest 18.09 (used as rootfs)
https://hydra.nixos.org/build/86613110/download/1/nixos-system-x86_64-linux.tar.xz
metadata.yaml

architecture: "x86_64"
creation_date: 1546382223
properties:
    architecture: "x86_64"
    description: "NixOS 18.09"
    os: "nixos"
    release: "18.09"
templates:
    /etc/hostname:
        when:
            - create
            - copy
        template: hostname.tpl

templates/hostname.tpl
{{ container.name }}
profile

config:
  raw.lxc: |-
    lxc.init.cmd = /init
    lxc.apparmor.allow_incomplete = 1
description: Default LXD profile for NixOS
devices:
  eth0:
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: nixos

So basically I could create a container and start it:

lxc launch -p nixos nixos-18.09 nixos-test-container
Creating nixos-test-container
Starting nixos-test-container

But everything which is connected to building fails:

lxc exec nixos1 -- /run/current-system/sw/bin/bash

[root@nixos:~]# nix-channel --update
unpacking channels...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: while setting up the build environment: mounting /proc: Operation not permitted
error: program '/nix/store/rffcxk0l94lc96yl07r26sdnfql6x0h4-nix-2.1.3/bin/nix-env' failed with exit code 1

[root@nixos:~]# nixos-rebuild switch
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using $NIX_PATH or -I), at (string):1:13
building Nix...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using $NIX_PATH or -I)
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
these paths will be fetched (15.16 MiB download, 65.01 MiB unpacked):
  /nix/store/12zhmzzhrwszdc8q3fwgifpwjkwi3mzc-gcc-7.3.0-lib
  /nix/store/1nlj1w7pqmxvxw9wi98ssngab9fsm091-libkrb5-1.15.2
  /nix/store/35yalbjrb5jh0arjawfxkryppvpip11a-nghttp2-1.24.0-lib
  /nix/store/3cx8jscamhp08hjv6x4lhd1w57ggji0k-gzip-1.9
  /nix/store/60pn8jnvbb3n70337jvdfkdn2zx7f331-libsodium-1.0.16
  /nix/store/6953iif8k3550lspbrss63w9saprqq8y-busybox-1.28.1
  /nix/store/6k794ln3ila2457d9k9fgyyyq7px8aci-acl-2.2.52
  /nix/store/6p06bw9hggh8vj093a59nib283cs1kbx-libssh2-1.8.0
  /nix/store/6v0aqcar0j6n5mlkdhqid4drqwvhlszn-curl-7.59.0
  /nix/store/7zam9d16ml78rb0p1p5gds4p5z99fpyv-gnutar-1.30
  /nix/store/80gpwb4fkvd2rfrpgmlgsq4hjbsz225p-bzip2-1.0.6.0.1
  /nix/store/8c30dl4mmgm4rip1l08fvad9gca5sk2j-zlib-1.2.11
  /nix/store/8zs94fwrc2mspl9g1ifmyh2gl78nl6nm-aws-sdk-cpp-1.3.22
  /nix/store/920k63py2w97xpfyb5ps9l6wbidvzfjz-coreutils-8.29
  /nix/store/95yylk9akpg65r32a1sq9qhia4bxzzz2-attr-2.4.47
  /nix/store/9f4kav8sa4qf3clc61h06ggyvhi2m728-xz-5.2.3-bin
  /nix/store/9pma2rib627l4iq67j6jl32mxq1mrzr9-sqlite-3.23.1
  /nix/store/a6bzl1xmhrgdyafdg9b1gmvjcf9pk16c-keyutils-1.5.10-lib
  /nix/store/cdcia67siabmj6li7vyffgv2cry86fq8-nix-2.1.3
  /nix/store/dgikkj4r0j0158013byz27s8dp1z9xhd-brotli-1.0.3-lib
  /nix/store/fzsf72mfsy40l5xbyyrkwb0qjdz8jpl0-boehm-gc-7.6.4
  /nix/store/hqspf7pmaz9s8hjjh3agjdmsn0pdj9xp-bzip2-1.0.6.0.1-bin
  /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131
  /nix/store/ldqd4ynsmxz7nsmlx36bap9z96v3vk74-libatomic_ops-7.6.2
  /nix/store/lw7xaqhakk0i1c631m3cvac3x4lc5gr5-bash-4.4-p12
  /nix/store/pflp65rydzd4wlywgm39mncw6g9v5cvf-libseccomp-2.3.3
  /nix/store/wqn1vcbm0za87cgxdy86p1ar1risgypr-openssl-1.0.2p
  /nix/store/x1rp4zikbiwjw2x7rvzlny4sj4yc99qw-xz-5.2.3
copying path '/nix/store/6953iif8k3550lspbrss63w9saprqq8y-busybox-1.28.1' from 'https://cache.nixos.org'...
copying path '/nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131' from 'https://cache.nixos.org'...
copying path '/nix/store/ldqd4ynsmxz7nsmlx36bap9z96v3vk74-libatomic_ops-7.6.2' from 'https://cache.nixos.org'...
copying path '/nix/store/95yylk9akpg65r32a1sq9qhia4bxzzz2-attr-2.4.47' from 'https://cache.nixos.org'...
copying path '/nix/store/lw7xaqhakk0i1c631m3cvac3x4lc5gr5-bash-4.4-p12' from 'https://cache.nixos.org'...
copying path '/nix/store/6k794ln3ila2457d9k9fgyyyq7px8aci-acl-2.2.52' from 'https://cache.nixos.org'...
copying path '/nix/store/dgikkj4r0j0158013byz27s8dp1z9xhd-brotli-1.0.3-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/80gpwb4fkvd2rfrpgmlgsq4hjbsz225p-bzip2-1.0.6.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/920k63py2w97xpfyb5ps9l6wbidvzfjz-coreutils-8.29' from 'https://cache.nixos.org'...
copying path '/nix/store/hqspf7pmaz9s8hjjh3agjdmsn0pdj9xp-bzip2-1.0.6.0.1-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/12zhmzzhrwszdc8q3fwgifpwjkwi3mzc-gcc-7.3.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/7zam9d16ml78rb0p1p5gds4p5z99fpyv-gnutar-1.30' from 'https://cache.nixos.org'...
copying path '/nix/store/fzsf72mfsy40l5xbyyrkwb0qjdz8jpl0-boehm-gc-7.6.4' from 'https://cache.nixos.org'...
copying path '/nix/store/3cx8jscamhp08hjv6x4lhd1w57ggji0k-gzip-1.9' from 'https://cache.nixos.org'...
copying path '/nix/store/a6bzl1xmhrgdyafdg9b1gmvjcf9pk16c-keyutils-1.5.10-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/pflp65rydzd4wlywgm39mncw6g9v5cvf-libseccomp-2.3.3' from 'https://cache.nixos.org'...
copying path '/nix/store/1nlj1w7pqmxvxw9wi98ssngab9fsm091-libkrb5-1.15.2' from 'https://cache.nixos.org'...
copying path '/nix/store/60pn8jnvbb3n70337jvdfkdn2zx7f331-libsodium-1.0.16' from 'https://cache.nixos.org'...
copying path '/nix/store/35yalbjrb5jh0arjawfxkryppvpip11a-nghttp2-1.24.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/wqn1vcbm0za87cgxdy86p1ar1risgypr-openssl-1.0.2p' from 'https://cache.nixos.org'...
copying path '/nix/store/9pma2rib627l4iq67j6jl32mxq1mrzr9-sqlite-3.23.1' from 'https://cache.nixos.org'...
copying path '/nix/store/x1rp4zikbiwjw2x7rvzlny4sj4yc99qw-xz-5.2.3' from 'https://cache.nixos.org'...
copying path '/nix/store/8c30dl4mmgm4rip1l08fvad9gca5sk2j-zlib-1.2.11' from 'https://cache.nixos.org'...
copying path '/nix/store/9f4kav8sa4qf3clc61h06ggyvhi2m728-xz-5.2.3-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/6p06bw9hggh8vj093a59nib283cs1kbx-libssh2-1.8.0' from 'https://cache.nixos.org'...
copying path '/nix/store/6v0aqcar0j6n5mlkdhqid4drqwvhlszn-curl-7.59.0' from 'https://cache.nixos.org'...
copying path '/nix/store/8zs94fwrc2mspl9g1ifmyh2gl78nl6nm-aws-sdk-cpp-1.3.22' from 'https://cache.nixos.org'...
copying path '/nix/store/cdcia67siabmj6li7vyffgv2cry86fq8-nix-2.1.3' from 'https://cache.nixos.org'...
/tmp/nixos-rebuild.OOVxAX/nix
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
building the system configuration...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using $NIX_PATH or -I)

Any ideas howto fix that?

davidak · September 2, 2019, 2:27pm

You can now create LXC images from your NixOS config with nixos-generators! It might still have some problems, so please report them if you encounter any.

lxc image import $(nixos-generate -f lxc-metadata) $(nixos-generate -c configuration.nix -f lxc)

matthewbauer · September 2, 2019, 11:48pm

Ideally, we could get this in distrobuilder as well which makes the official lxd containers (https://us.images.linuxcontainers.org). I could never get it to work, but have a WIP commit:

asbachb · September 13, 2019, 10:54pm

@matthewbauer basically I agree, but I think in the current state it would not make sense to provide an lxc/lxd image since it seems that there’re several issues which needs to be addressed before an end user is capable of using NixOS as lxc/lxd guest.

Tbh curently it doesnt’ seem that there’s much interest in NixOS community to get it running inside lxc/lxd. (That should not be understood as an attack, just my perception)

pjgoodall · February 23, 2020, 6:47pm

lxd/lxc is an excellent environment for potential hosting of nixos on Ubuntu. Has great pass-through of CUDA/GPU hardware, and very low spin up time. I wish I had the expertise.

pjgoodall · July 11, 2021, 10:40pm

This suggestion worked for me:

lxc export c1=<container_name>
lxc stop $c1
lxc config set $c1 security.nesting true
lxc start $c1
lxc exec $c1 -- su --login ubuntu
$ bash <(curl -L https://nixos.org/nix/install)