NixOS-like experience on Ubuntu in AWS

Guides
1

I’ve had fun recently with making server Ubuntu behave more like NixOS. I’ll share some parts, which work for me, and I hope people can suggest improvements (drop Ubuntu? haha, nice try).

So, my story is based around Nix, overlays, home-manager and declarative containers. Let’s start with required installations

I’ll post installation instructions as Salt state config, but you can adapt to whatever you like.

Nix

# TODO: set secure_path in /etc/sudoers to Nix-enabled value
# In the meanwhile `sudo -i` works
install-nix:
  cmd.run:
    # will install as root internally
    - user: ubuntu
    - name: |
         curl -o install.sh https://nixos.org/nix/install
         yes | sh ./install.sh --daemon
         echo "source /etc/profile.d/nix.sh" | sudo tee -a /etc/bash.bashrc
    - unless: bash -c 'nix-env --version'

That’s it, we have Nix with daemon installed!

nix-collect-garbage:
  cron.present:
    - name: "/root/.nix-profile/bin/nix-collect-garbage -d && /root/.nix-profile/bin/nix-store --optimise"
    - user: root
    - special: '@daily'

We are also not very interested in rollbacks, as our approach is very hacky. But you may do GC manually if you want

Overlay

I’ll use 18.09 home-manager release, but you can pinpoint it to specific version.

# This is central place to handle nixpkgs checkouts
self: super: let

    pkgs = self;
    home-manager_checkout = builtins.fetchGit {
        name = "amplicare-home-manager-18.09";
        url = https://github.com/rycee/home-manager;
        ref = "release-18.09";
    };

in {
    _home_checkout = pkgs.runCommand "checkout" {} ''
      cp -r ${home-manager_checkout} $out/
    '';
    _home = import home-manager_checkout { };
}

Then you have to install this file into ~/.config/nixpkgs/overlays/ and to /root/.config/nixpkgs/overlays/. Why root’s? Just to make it consistent nix-build and sudo -i nix-build

home-manager install

install-home:
  cmd.run:
    - user: {{user}}
    - name: |
        source /etc/profile.d/nix.sh
        sudo -i nix-build '<nixpkgs>' -A _home_checkout \
          -o /nix/var/nix/profiles/per-user/$USER/channels/home-manager
        nix-shell '<nixpkgs>' -A _home.install
    - unless: bash -c 'home-manager generations | grep -q id'
    - require:
        - install-nix
        - file: root/.config/nixpkgs/overlays/pkgs-overlay.nix
        - file: {{user_home}}/.config/nixpkgs/overlays/pkgs-overlay.nix

And here we have home-manager installed! Note that it will use home channel from git checkout only during installation, no auto-channel update specified here. Probably a TODO

Personally, I’ve installed it as root, so I have full control over system from home manager, but unfortunately I can’t do declarative /etc update, because home-manager is about HOME mgmt… Oh well, maybe smbd can suggest how to fix this?

Home-manager

This is home-manager config skeleton, which is tuned to allow working with user systemd services - lingering and controlling:

{ config, pkgs, ... }: let

    dag = config.lib.dag;

in {
    programs.home-manager.enable = true;
    home.stateVersion = "18.09";

    home.file.".profile".text = ''
        # -*- mode: sh -*-
        if [ "$BASH" ]; then
            if [ -f ~/.bashrc ]; then
                . ~/.bashrc
            fi
        fi

        mesg n || true
        # this one actually makes envvars from home-manager to be set on shell login
        . "${config.home.profileDirectory}/etc/profile.d/hm-session-vars.sh"
    '';
    home.language.base = "C";
    home.sessionVariables.XDG_RUNTIME_DIR = "/run/user/$UID"; # for systemctl --user

    # so user systemd services are not stopped on logout
    home.activation.ensureDirs = dag.entryBefore ["reloadSystemD"] ''
        loginctl enable-linger $USER
    '';
    
    # We'll setup container later, but this is required if you want containers to start on boot
    home.activation.startContainers = dag.entryBefore ["reloadSystemD"] ''
        machines_wants=/etc/systemd/system/machines.target.wants
        sudo mkdir -p $machines_wants
        sudo rm -rf $machines_wants/*
        for service in $(ls -d /usr/lib/systemd/system/*); do
          sudo ln -s $service $machines_wants/
        done
    '';

    # TODO: this isn't restarted automatically on changes!!!
    # systemd.user.services.nomad-agent =  ...
}

I’m still not sure why user services are not restarted on configuration changes. Maybe someone can help me here?

Containers

Containers are awesome, in sense that you get full NixOS configuration, no need to do wrappers around services and so on, They are costly to start though, but not much. So, prerequisites first:

/etc/static/os-release:
  file.managed:
    - user: root
    - group: root
    - mode: 755
    - makedirs: True
    - contents: |
        dummy

# Machinectl and systemd-nspawn, which are required for nixos containers
systemd-container:
  pkg:
    - installed

The first one fixes bug in NixOS containers, which require /etc/static/os-release to exist. Second installs container tools. Maybe I could go with installing Nixpkgs systemd instead, but I don’t know how would it interop with default Ubuntu’s systemd.

Overlay

self: super: let

    pkgs = self;

in {
    extra-container = pkgs.stdenv.mkDerivation {
        name = "extra-container-patched";
        src = pkgs.fetchurl {
            url = "https://github.com/erikarvstedt/extra-container/archive/0.2.tar.gz";
            sha256 = "1gz7rl2hx8kbmy5wdyai75mbscr0v61da4h7paz3j3yzfk6p7wf9";
        };
        buildCommand = ''
            unpackPhase && cd "$sourceRoot"
            # extra-container was designed for NixOS. We have to adapt it for Ubuntu
            sed -i 's|/etc/systemd-mutable|/usr/lib/systemd|g' extra-container
            mkdir -p $out/bin
            cp extra-container $out/bin/extra-container
            patchShebangs $out/bin
        '';
    };
}

and home.nix change:

    home.packages = with pkgs; [
        extra-container
        nixos-container
    ];

This brings extra-container tool, with a fix for Ubuntu.

Activation

Finally, we’ll set activate phase, which will be called on every deploy. If you don’t use salt, then this should be a script activate.sh:

activate:
  cmd.run:
    - user: root
    - name: |
        set -e
        source /etc/profile.d/nix.sh
        rm ~/.profile || true   # home-manager barfs this file exists already
        home-manager switch
        extra-container create --start --restart /var/www/containers.nix
        # TODO: remove when extra-container learns to treat autoStart in containers correctly!!!
        home-manager switch
    - require:
        - file: /root/.config/nixpkgs/home.nix
        - file: /var/www/containers.nix

Unfortunately, I have to call home-manager switch twice, because extra-container tool is installed via home-manager, and containers autostart is managed by home-manager… Chicken and egg, so I call both.

Containers configuration

The /var/www/containers.nix is an actual NixOS configuration, with declarative containers defined. extra-container evaluates it just like regular NixOS, but manages only containers. My real config is complicated one, so I’ll show some simple example:

let
  common = {
    nixpkgs.overlays = [
        (import ./overlay.nix) # ideally all overlays you use
    ];
    imports = [
        <nixpkgs/nixos/modules/profiles/minimal.nix>
        <nixpkgs/nixos/modules/profiles/headless.nix>
    ];
    system.stateVersion = "19.03";
  };

in {

    # containers.consul.autoStart = true; # we've done autostart by default
    containers.consul.bindMounts."/var/www" = {
        hostPath = "/var/www"; # note how I manage host's path inside container with systemd.tmpfiles
        isReadOnly = false;
    };
    containers.consul.config = { config, pkgs, ...}: {
        imports = [
            common
        ];

        systemd.tmpfiles.rules = [
            "d /var/www/nginx.d 0755 nginx nginx - -"
            "d /var/www/nginx.d/logs 0755 nginx nginx - -"
        ];

        services.nginx.enable = true;
        services.nginx.appendHttpConfig = ''
            include /var/www/nginx.d/*.conf;
            log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
            access_log  logs/access.log  main;
            error_log  logs/error.log warn;
        '';
        services.nginx.stateDir = "/var/www/nginx.d";
        services.nginx.virtualHosts."_" = {
            # fallback virtual host
        };
        services.nginx.virtualHosts."consul.example.com" = let
            consulUiPort = config.services.consul.extraConfig.ports.http or 8500;
        in {
            listen = [ { addr = "0.0.0.0"; port = 8499; } ];
            locations = {
                "/" = {
                    proxyPass = "http://127.0.0.1:${toString consulUiPort}/";
                };
            };
        };

    };

    # containers.nomad.autoStart = true;
    containers.nomad.config = { config, ...}: {
        imports = [
            common
            # ./nomad.nix
        ];

        # services.nomad.enable = true;
    };
}

Result

So this will boot two containers, with several custom services, fully managed by NixOS configuration. To control services from host systemctl -M machinename is used. Or sudo journalctl -M machinename for logs. For user services defined in home.nix you can use sudo journalctl --user-unit instead of sudo journalctl -u. home.nix allows you to control packages installed on host, envvars, some basic state and dotfiles. And containers bring more NixOS-like experience (if you are used to it).

$ uname -a
Linux nomad-01 4.15.0-1033-aws #35-Ubuntu SMP Wed Feb 6 13:29:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ nixos-container list
consul
nomad
$ sudo systemctl --user status
Failed to connect to bus: No such file or directory
$ sudo -i systemctl --user status
* nomad-01
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Thu 2019-03-07 10:48:49 UTC; 1 weeks 1 days ago
   CGroup: /user.slice/user-0.slice/user@0.service
           |-nomad-client.service
           | |-  904 /nix/store/fx8jzkjz2dbc770vxc3ixbsal4sq67k5-nomad-0.8.7-bin/bin/nomad agent -config /root/nomad-agent.json
           | |- 2841 nc -l 0.0.0.0 -p 22908 -q 1
           | |- 2846 nc -l 0.0.0.0 -p 26890 -q 1
           | |- 2851 nc -l 0.0.0.0 -p 27871 -q 1
           | |- 2856 nc -l 0.0.0.0 -p 21980 -q 1
   ...

But be warned - all this is very hacky and is done only for fun.

Links and tribute:

13 Likes