NixOS as OpenVZ 7 guest

1

Hi, so I’m trying to run NixOS inside an OpenVZ 7 container. The host is running a 4.15.0 kernel, and before I go further, this is my situation:

  • I cannot get a shell on the hypervisor itself (this is a VPS provider)
  • I cannot reproduce the failure locally. NixOS boots just fine on my own OpenVZ 7 host after replacing /sbin/init in an Ubuntu 18.04 container.

So I made my rootfs with nixos-generate -f lxc, with the following configuration change:

# ramfs is not available on the host
boot.specialFileSystems."/run/keys".fsType = lib.mkForce "tmpfs";

(This setup works fine for a couple of LXC providers)

Shortly after systemd was started from stage 2 (I do know systemd was started), it crashed. As it’s not possible to get a rescue shell (wrapper of vzctl enter, which starts /bin/bash hardcoded) when PID 1 is not running, I created a hacky /sbin/init that starts the stage 2 script on every second boot:

#!/nix/store/a3fc4zqaiak11jks9zd579mz5v0li8bg-bash-4.4-p23/bin/bash

systemConfig=/nix/store/ia9qb52bk8whv7v20sg4l85rcc1s2zj2-nixos-system-nixos-20.09pre-git
export PATH=/nix/store/z1qvlavy35wanw5k54fvvfffws5bvigj-coreutils-8.31/bin

date >> /boot-log

val=$(cat /boot-mark)
mod2=$((val % 2))
echo $((val + 1)) > /boot-mark

if [ $mod2 == "0" ]; then
	echo "booted to rescue" >> /boot-log
	$systemConfig/activate
	while :; do sleep 1; done
else
	echo "booted to system" >> /boot-log
	exec $systemConfig/init 2>&1 >> /boot-log
fi

I’m positive that systemd was started as I was able to get into the rescue shell in the brief window before it crashed and saw systemd running as PID 1.

However, I’m unable to obtain useful logs from systemd. I have no access to /dev/console or /dev/tty*, /var/log/journal is empty, and I don’t see a way to coax it to emit anything to stdout/stderr (I did remove the lines that reset the stdout/stderr descriptors from the stage 2 init).

Any ideas of how to proceed? In case you would like to waste some time on this yourself, I’m using Gullo’s Hosting and they are running Christmas deals. The 128MB bundles are pretty cheap, so you may want to give it a go if you also feel like running NixOS in some cursed environment :slight_smile:

1 Like
2

Steps:

  1. Booting from iso - https://channels.nixos.org/nixos-20.09/latest-nixos-minimal-x86_64-linux.iso (I did it from OpenVZ virtual machine prlctl create nixos --ostype linux --vmtype vm )
  2. Creating NixOS file system via nixos-generate -f lxc
  3. Copying archive nixos-system-x86_64-linux.tar.xz to a folder on the hardware node
  4. Creating OpenVZ container ( I did it prlctl create nixos.ct --vmtype ct --ostemplate centos-8-x86_64)
  5. Mounting container root (prlctl mount nixos.ct)
  6. Removing all data from root of the container (prlctl list -a finding container id then /vz/root/containerid)
  7. Unpacking nixos-system-x86_64-linux.tar.xz to the root of the container
  8. Creating config file on the hardware node /usr/libexec/libvzctl/dists (i did unknown.conf)
# This configuration file is meant to be used with
# the Gentoo distribution kit.
#
# Copyright (c) 1999-2017, Parallels International GmbH
# Copyright (c) 2017-2019 Virtuozzo International GmbH. All rights reserved.
#
# This file is part of OpenVZ libraries. OpenVZ is free software; you can
# redistribute it and/or modify it under the terms of the GNU Lesser General
# Public License as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#.
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
# 02110-1301, USA.
#
# Our contact details: Virtuozzo International GmbH, Vordergasse 59, 8200
# Schaffhausen, Switzerland.

ADD_IP=unknown_ip.sh
DEL_IP=unknown-del_ip.sh
NETIF_ADD=gentoo-add_netif.sh
NETIF_DEL=gentoo-del_netif.sh
SET_HOSTNAME=unknown-set_hostname.sh
SET_DNS=unknown-set_dns.sh
SET_USERPASS=set_userpass.sh
SET_UGID_QUOTA=unknown-set_ugid_quota.sh
POST_CREATE=postcreate.sh
POST_MIGRATE=unknown-post_migrate.sh
GET_V2PMIGRATE_EXCLUDES=gentoo-v2pmigrate-excludes.sh
SET_CONSOLE=set_console.sh
  1. Changing in configuration file of the container /vz/private/containerid/ve.conf set the DISTRIBUTION=“unknown”

That’s all now prlct umount ct name
prlct start ct name
prlct enter ct name

1 Like
3

With the urge to make use of my cheap OpenVZ VPSes again, I dug a bit deeper and came up with a solution: GitHub - zhaofengli/nixos-openvz: NixOS on OpenVZ 7

This allows you to run NixOS on OpenVZ 7 VPSes created with the Debian/Ubuntu template without requesting any changes to the configuration.

2 Likes