Hello
I’m trying to implement a workflow for provisioning Nixos LXC containers in Proxmox. I have the following vision
- Build LXC tarball. This tarball includes my user account, public key and some other basic stuff
- Upload it to proxmox
- Create instances with terraform
- Deploy services that I need in those containers with nixinate. SSH keys are baked in the image, so connection is possible
So I basically treat LXC like I would treat VMs that are configured separately as needed, instead of baking their configuration at build time
I implemented this workflow and it’s quite nice. My problem is, when running nixinate, I see that nix tries to redownload the entire system. This includes, among other things, various low-level libraries, openssl, nano and even systemd. Not only it consumes bandwidth and fills container memory, it also requires me to reboot the container, because systemd gets locked and nixos-rebuild is unable to finish the process. It works normally after reboot, but automating it is not the solution I’d like to have.
This is my flake:
{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.11";
terranix.url = "github:terranix/terranix";
terranix.inputs.nixpkgs.follows = "nixpkgs";
nixinate.url = "github:matthewcroughan/nixinate";
nixinate.inputs.nixpkgs.follows = "nixpkgs";
};
outputs = { self, nixpkgs, terranix, nixinate, ... }@inputs:
let
system = "x86_64-linux";
pkgs = import nixpkgs { inherit system; };
in
{
packages.${system} = {
tf-config = terranix.lib.terranixConfiguration {
inherit system;
modules = [ ./terraform/terraform.nix ];
};
};
nixosConfigurations = {
proxmox-lxc = nixpkgs.lib.nixosSystem {
inherit system;
modules = [ ./modules/base-lxc.nix ];
};
dns = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
./modules/base-lxc.nix
{
environment.systemPackages = [
pkgs.hello
];
}
{
_module.args.nixinate = {
host = "10.0.10.0";
sshUser = "<<redacted>>";
buildOn = "remote"; # valid args are "local" or "remote"
substituteOnTarget = true; # if buildOn is "local" then it will substitute on the target, "-s"
hermetic = true;
};
}
];
};
};
devShell.${system} = pkgs.callPackage ./shell.nix ({inherit pkgs; inherit inputs;});
apps = nixinate.nixinate.x86_64-linux self;
};
}
Here, proxmox-lxc is the base image, and dns is the machine configuration. ./modules/base-lxc.nix includes users, ssh keys, ssh config and some proxmox specific settings.
When building the image from proxmox-lxc, it created a ~100MB tarball. I uploaded it to proxmox and create a container, which takes 500MB of space. Then I ran nixinate. First, it downloaded 20M (88M unpacked) of basic low level stuff, which included some aws C libraries and bash for some reason. Then it downloaded 21M of source, which I believe is nixpkgs, fine. Then it proceeded to download another 200M (~500M unpacked) of various stuff, including systemd (which already was present in the store btw). And finally, it failed with /nix/store/yl8cr5df7mi7ws3d5cq9n5f4lwhrxd5r-system-path/bin/busctl --json=short call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager ListUnitsByPatterns asas 0 0' exited with value 1 at /nix/store/p8as6v2cd6msvgz5bjgyn8x1gs2sz1yq-nixos-system-unnamed-22.11.20230207.af96094/bin/switch-to-configuration
After rebooting container and rerunning nixinate, it activated the configuration at last and I can SSH into it and have my hello command. The container now consumes 900MB of disk space.
So, my question is, how do I include all this stuff in the container template, so that when I run nixinate or nixos-rebuild, it does not redownload everything? I only found this issue, that touches this problem: `nixos-rebuild` ready image · Issue #86 · nix-community/nixos-generators · GitHub, and it kind of explains why that happens, but I’m unable to come up with a proper solution.
Thanks!
