Hello All!
I have benefited majorly from the guides and discussion on this site and hope to contribute a little bit. I’m venturing into the wonderful world of kubernetes and figured out how to set up a declarative rootless server.
NOTE: Before starting, I only had luck after setting systemd.enableUnifiedCgroupHierarchy = false;
(I happened to add that line to my configuration.nix
but you can set it wherever) as described in this github issue.
{ config, pkgs, ... }:
let
USER = "k3s"; # set it and forget it!
in
{
## Optional
nixpkgs.config.allowUnfree = true;
## Reusable system user block
## Warning: if you want to drop into a shell and interact with systemd, use the following:
## `MYUSER=[${USER}] sudo -H -u $MYUSER XDG_RUNTIME_DIR=/run/user/$(id -u $MYUSER)
users.users.${USER} = {
isSystemUser = true;
linger = true;
home = "/var/lib/${USER}";
description = "system user for running k3";
packages = with pkgs; [
k3s
killall
slirp4netns
podman
];
group = "${USER}";
extraGroups = [ "systemd-journal" ];
autoSubUidGidRange = true;
};
users.groups.${USER} = {};
### Rootless K3S, based on this: https://github.com/k3s-io/k3s/blob/e2179aa957a02d4b357bef9aabb163f043471023/k3s-rootless.service
systemd.user.services."k3s-rootless" = {
# NOTE: Don't try to run `k3s server --rootless` on a terminal, as it doesn't enable cgroup v2 delegation.
# If you really need to try it on a terminal, prepend `systemd-run --user -p Delegate=yes --tty` to create a systemd scope.
# systemd unit file for k3s (rootless)
#
# Usage:
# - [Optional] Enable cgroup v2 delegation, see https://rootlesscontaine.rs/getting-started/common/cgroup2/ .
# This step is optional, but highly recommended for enabling CPU and memory resource limtitation.
#
# - Run `systemctl --user disable --now k3s-rootless && systemctl --user enable --now k3s-rootless`
#
# - Run `KUBECONFIG=~/.kube/k3s.yaml kubectl get pods -A`, and make sure the pods are running.
#
# Troubleshooting:
# - See `systemctl --user status k3s-rootless` to check the daemon status
# - See `journalctl --user -f -u k3s-rootless` to see the daemon log
# - See also https://rootlesscontaine.rs/
enable = true;
description="k3s (Rootless)";
#environment= {
# PATH = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin";
#};
## Initially the service couldn't find the slirp4netns binary on the PATH
path = [
"${pkgs.k3s}"
"${pkgs.slirp4netns}"
"${pkgs.podman}"
"/run/wrappers/" # for newuidmap
];
serviceConfig = {
ExecStart = "${pkgs.k3s}/bin/k3s server --rootless --snapshotter=fuse-overlayfs --kubelet-arg='cgroup-driver=systemd'";
ExecReload="${pkgs.killall}/bin/killall -s HUP $MAINPID";
TimeoutSec=0;
RestartSec=2;
Restart="always";
StartLimitBurst=3;
StartLimitInterval="60s";
LimitNOFILE="infinity";
LimitNPROC="infinity";
LimitCORE="infinity";
TasksMax="infinity";
Delegate="yes";
Type="simple";
KillMode="mixed";
};
#[Install]
wantedBy= [ "default.target" ];
};
systemd.user.services."${USER}-podman-enabler" = {
enable = true;
description = "ensure podman service and socket are enabled";
wantedBy = [ "multi-user.target" ];
serviceConfig = {
ExecStart = "systemctl enable --now podman.service podman.socket";
Type = "oneshot";
};
};
## Possibly not necessary, but convenient for copy/pasting this user for other purposes
systemd.user.services."${USER}-podman-network-maker" = {
enable = true;
description = "ensure podman networks are available";
wantedBy = [ "multi-user.target" ];
serviceConfig = {
ExecStart = "podman network create proxy behold";
Type = "oneshot";
};
};
systemd.user.services."${USER}-rootless-restart" = {
enable = true;
description = "Start all containers where restart=always (rootless)";
wantedBy = [ "default.target" ];
wants = [ "network-online.target" ];
after = [ "network-online.target" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
Environment = "PODMAN_SYSTEMD_UNIT=%n: LOGGING=\"--log-level=info\"";
ExecStart = "/usr/bin/podman $LOGGING start --all --filter restart-policy=always";
ExecStop = "/bin/sh -c '/usr/bin/podman $LOGGING stop $(/usr/bin/podman container ls --filter restart-policy=always -q)'";
};
};
}
You can check that it’s actually running rootless like this:
> ps -U root | grep k3
# note, nothing shows up
> ps -U k3s | grep k3
1807 ? 00:00:00 k3s-server
2582 ? 00:31:15 k3s
Finally, you can check the status page at: https://localhost:6443/
and you should be greeted with a status page.
Happy Hosting!