NixOS Proxmox LXC Not rebuilding using wiki provided configuration

Help
1

Hello all I am currently trying to deploy NixOS onto Proxmox as an LXC using this guide:

https://nixos.wiki/wiki/Proxmox_Virtual_Environment

I follow all instructions and create the base configuration.nix in /etc/nixos

{ pkgs, modulesPath, ... }:

{
  imports = [
    (modulesPath + "/virtualisation/proxmox-lxc.nix")
  ];

  environment.systemPackages = [
    pkgs.vim
  ];
}

This is what’s provided from the guide. However when I go to rebuild I get the following error:

Failed to start transient service unit: Access denied
'/nix/store/dp0r89lxv832r3rcy4rddcas0jbv5hvn-system-path/bin/busctl --json=short call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager ListUnitsByPatterns asas 0 0' exited with value 1 at /nix/store/smwjn1m0a7ysrbcjzf75pdqncsqhyvl6-nixos-system-unnamed-24.05.1409.cc54fb41d137/bin/switch-to-configuration line 145.
warning: error(s) occurred while switching to the new configuration

I have tried making sure nesting is enabled, and I have tried with both privileged and non-privileged LXC containers. They both seem to have this issue.

2

Note: nix-shell -p does seem to work, but nixos-rebuild switch does not in this context.

3

nixos-rebuild switch does not in this context

That is a bit unfortunate, I wanted to try this setup next week and see if it’s usable…

4

Have you run nix-channel --update inside the container?

5

Yes @JimJ92120 this is with the required commands already run

6

That post seems to mention the same issue(s): Nixos-rebuild switch fails with busctl error: Access Denied

7

Yes, @JimJ92120 it seems to throw the same error there as well, but rebooting does not fix it for me

I think there may be serious issues with the LXC container itself and it’s interoperability with Proxmox, as a reboot should not be required.

8

More reading:

trying to deploy NixOS onto Proxmox as an LXC using this guide

Not sure if expected or not, it seems that when building as mentioned in the guide, the container doesn’t carry the “base” config from the build OR doesn’t have permission to access that base config (which is on the host).
So when running nixos-rebuild, it’s missing that “base” config / can’t access it.

Workarounds mentioned in posts above are suggesting to:

I’m using lxc with lxd and haven’t encountered such issue(s) so far.
That’s how I’m currently creating nixos image for lxc, possibly try with a very minimal configuration.nix:

Something like that:

{ config, pkgs, ... }:

{
  boot.isContainer = true;

  system.stateVersion = "24.05";

  # add some config below
}
9

Well adding the boot.isContainer = true;

seems to have allowed the rebuild to partially work

However every time I rebuild it seems to throw this error

building Nix...
building the system configuration...
activating the configuration...
setting up /etc...
reloading user units for root...
restarting sysinit-reactivation.target
the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service
warning: the following units failed: nscd.service

× nscd.service - Name Service Cache Daemon (nsncd)
     Loaded: loaded (/etc/systemd/system/nscd.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2024-06-18 20:43:38 UTC; 39ms ago
    Process: 9489 ExecStart=/nix/store/kjc8f1p4ii6zdiwlyp3l8qvxwc6mypih-nsncd-unstable-2024-01-16/bin/nsncd (code=exited, status=1/FAILURE)
   Main PID: 9489 (code=exited, status=1/FAILURE)
         IP: 0B in, 0B out
        CPU: 7ms

Jun 18 20:43:38 Minecraft systemd[1]: nscd.service: Start request repeated too quickly.
Jun 18 20:43:38 Minecraft systemd[1]: nscd.service: Failed with result 'exit-code'.
Jun 18 20:43:38 Minecraft systemd[1]: Failed to start Name Service Cache Daemon (nsncd).
warning: error(s) occurred while switching to the new configuration

[root@Minecraft:/etc/nixos]#

Oddly enough if I install a package, the package installs as if its switching to the new config and works as intended, but things like enabling the SSH daemon do NOT work as intended.

I also had to reboot a few times in order for it get to that point.

10

Now that I think about it, it is probably that the failing nscd service is related to the SSH daemon not starting. Perhaps something about how networking is handled in NixOS in LXC

11

Are you able to e.g restart those services and related ones manually after switching?
Possibly some cached / temp config, files that aren’t reset nor restarted after rebuild

12

Nope, oddly enough the sshd service does not exist despite being defined in the file as such:

  # Enable the OpenSSH daemon.
  services.sshd.enable = true;
  services.openssh.enable = true;
  services.openssh.settings = {
        PermitRootLogin = "yes";

  };
13

You normally would only need this to enable ssh as minimum config:

  services.openssh = {
    enable = true;
  };

If you want some references (on lxd):

14

Yeah, I tried both ways and disabled the firewall on both the parent hypervisor and the NixOS LXC with firewall disabled but when I try to enable SSH I get:

kex_exchange_identification: read: Connection reset by peer

Internet does work pinging websites and downloading packages though

15

Some debugging solutions here maybe: ssh - How can I fix "kex_exchange_identification: read: Connection reset by peer"? - Stack Overflow
=> post suggests either:

  • conflicting IP’s
  • host being denied / not allowed by the container
  • host networking and above (e.g router, VPN, DNS, etc)
16

I checked for IP conflicts, and disabled firewall on both Proxmox and NixOS LXC

It isn’t just SSH that doesn’t work, docker won’t run as a daemon, and many other services wont either.

I think it may be a result of:

Jun 18 20:43:38 Minecraft systemd[1]: nscd.service: Start request repeated too quickly.
Jun 18 20:43:38 Minecraft systemd[1]: nscd.service: Failed with result 'exit-code'.
Jun 18 20:43:38 Minecraft systemd[1]: Failed to start Name Service Cache Daemon (nsncd).
warning: error(s) occurred while switching to the new configuration

Since nscd.service is network related, I think it may be this. I should note though that that I have

Unprivileged: No

set for the container options.

17

Are you still using the -c security.nesting=true flag when creating the container?

Maybe that config to add, referenced here: Howto setup LXD on NixOS with NixOS guest using unmanaged bridge network interface

  # `boot.isContainer` implies NIX_REMOTE = "daemon"
  # (with the comment "Use the host's nix-daemon")
  # We don't want to use the host's nix-daemon.
  environment.variables.NIX_REMOTE = lib.mkForce "";
18

Also encountered the issue mentioned here. This PR fixes it: nixos/proxmox-lxc: fix getty start and nixos-rebuild by fpletz · Pull Request #328682 · NixOS/nixpkgs · GitHub

1 Like