Replacing the bootStage2

raboof · March 22, 2020, 5:17pm

I’m trying to create a Linux Container (LXC) that starts up, performs a task, and then immediately shuts itself down.

This is already working fairly nicely with nixos-generators and the configuration at:

raboof/mastodon-bot-nix/blob/7944ff12c85e3a31615c53f21e6b86e61def59ca/configuration.nix#L22


      
              name = "mastodon-bot-hack42";
              paths = [ mastodon-bot ];
              buildInputs = [ pkgs.makeWrapper ];
              postBuild = "wrapProgram $out/bin/mastodon-bot --set MASTODON_BOT_CONFIG ${./config.hack42.edn}";
            };
          in
          {
            # This is too early: no network yet...
            # would be nice to set up the network 'manually' and drop systemd, but one step at a time
            # boot.postBootCommands = ''MASTODON_BOT_CONFIG=${./config.hack42.edn} ${mastodon-bot}/bin/mastodon-bot'';
            systemd.services."hack42-mastodon-bot" = {
              wantedBy = [ "multi-user.target" ];
              description = "mastodon-bot with the hack42 configuration";
              after = [ "network-online.target" ];
              unitConfig = {
                DefaultDependencies = false;
              };
              serviceConfig = {
                Type = "oneshot";
                RemainAfterExit = false;
                # To give DNS time to actually initialize :/

However, systemd seems rather overkill here. It seems neater if I could overwrite system.build.bootStage2 myself, do just the minimal initialization needed for the script (enabling networking), performing the task and shutting down again.

Naively setting system.build.bootStage2 in configuration.nix doesn’t seem to work, though. Is this possible? How would I do it?

lheckemann · April 3, 2020, 11:24am

This sounds like a case for not using NixOS: NixOS is quite tightly coupled with systemd, but you can make your container’s store contain just your init script and replace nixos with it, something like

{ nixpkgs ? <nixpkgs> }:
let
  pkgs = import nixpkgs {}; 
  initScript = pkgs.writeScript "init" ''
    ${pkgs.dhcp}/bin/dhclient -v eth0
    exec ${mastodon-bot-hack42}/bin/mastodon-bot # or some lightweight service manager
  '';
in import (nixpkgs + "/nixos/lib/make-system-tarball.nix") {
  contents = [
    { source = initScript; target = "/init"; }
  ];
  extraArgs = "--owner=0";
  extraCommands = "mkdir -p proc sys dev";
}

That will get you a tarball containing a symlink at /init to the init script defined there (the make-system-tarball part of this is mostly copied from the lxc generator itself at https://github.com/nix-community/nixos-generators/blob/229d236d4de23d1cea77f1ebfdaa223f7ca74c27/formats/lxc.nix).

raboof · April 3, 2020, 1:07pm

Thanks for that, I’ll be sure to look into that further!

Do you mean “using Nix but not NixOS”? That indeed perhaps could be interesting. I wouldn’t want to give up Nix - right now even if it means pulling in systemd it’s worth it IMHO .

lheckemann · April 4, 2020, 8:27am

Yes, exactly — Nix without NixOS systemd will probably be pulled in as a dependency because the dhcp client will (maybe transitively) depend on udev or whatever, but you can avoid running it as PID1 in this way.

raboof · April 11, 2020, 2:21pm

Thanks again, based on this I cut the image down to a pretty nice 42MB (compressed), https://github.com/raboof/mastodon-bot-nix/blob/7b9c54d447ec7b4e800c3ba6937ad58213e70e08/default.nix

(most of the remaining noise there is going into compiling down the clojurescript to javascript so we don’t need to ship the lumo interpreter in the image)