Extra-container - Run declarative containers without full system rebuilds

Announcements
6

@manveru, your comment wasn’t ignorant at all, I forgot to document the list command. It’s fixed in the latest update.

7

@tom, the NixOps container backend uses imperative containers which lack some options of declarative containers.

1 Like
8

Perhaps add this as another nixops target/option? I’m seeing a period of rapid progress of experimentation and think the community should try to consolidate and mature some tools before it becomes too fragmented.

1 Like
9

I think this should be integrated into nixos-container so that there’s feature parity between imperative and declarative containers. There are a few kinks left to work out, then I’ll open an issue.

2 Likes
10

I had success with running declarative containers with this piece of software on non-NixOS (Ubuntu). I’ve documented problems in extra-container on non-NixOS · Issue #1 · erikarvstedt/extra-container · GitHub

That’s really awesome!

1 Like
11

I’ve been using this on-and-off and it still seems to work fine! I used it for Towards an integrated search engine for the Nix community - #11 by deliciouslytyped

PS: Oh! Apparently it’s still maintained too. I didn’t notice that.

1 Like
12

extra-container 0.5 has just been released. (Changelog)

Highlights

  • Generic support for systemd-based Linux distros, like Arch or Debian. (Thanks @danbst)

  • Command shell.
    Starts a shell session for interacting with the container.

    read -d '' src <<'EOF'
{
  containers.demo = {
    # New feature: extra config options for setting up a private network
    extra.addressPrefix = "10.250.0"; 
    extra.enableWAN = true;
  };
}
EOF
extra-container shell -E "$src"

# Run command inside the container
c curl ifconfig.me
# Ping container
ping -c 1 $ip

# The container is destroyed when exiting the shell

# Run a single command and exit
extra-container shell -E "$src" --run c hostname
# => demo
3 Likes
13

That seems really cool if container builds can be made fast. I’m probably mixing things up, and I can’t test right now, but don’t container builds have a long squashfs step?

14

There’s no squashfs step for NixOS containers, they share the host’s nix store.
A simple extra-container builds and starts within ~2 sec on a desktop system with warm caches.

3 Likes
15

Ok thanks, I have no idea what I was doing then. - ah, must have been when I was testing my containers with the vm rebuild! (which is still not a container problem.)

16

This is really cool. I have a use case where it would be particularly nice if I could run multiple instances of the same container. (Normally I would just say, multiple containers from the same specification, which somehow seems more proper). Is this currently possible, or something that might be possible to add?

Thanks!

17

You can simply define multiple containers with the same config:

extra-container create -s <<'EOF'
{ config, pkgs, lib, ... }:
let
  sharedConfig = {
    config = {
      systemd.services.hello = {
        wantedBy = [ "multi-user.target" ];
        script = "while true; do echo hello | ${pkgs.netcat}/bin/nc -lN 50; done";
      };
    };
  };
in {
  containers = {
    instance1 = sharedConfig;
    instance2 = sharedConfig;
  };
}
EOF
1 Like
18

Thanks, I see.

My use cases is essentially creating an isolated environment for users in a dynamic way, so I can’t really define all instances ahead of time. In that case, I wonder if this would work, and if there would be any downsides to it? Basically, i would have a script run this command, with instanceName dynamically generated:

extra-container create -s <<'EOF'
{ config, pkgs, lib, ... }:
let
  sharedConfig = {
    config = {
      systemd.services.hello = {
        wantedBy = [ "multi-user.target" ];
        script = "while true; do echo hello | ${pkgs.netcat}/bin/nc -lN 50; done";
      };
    };
  };
in {
  containers = {
    instanceName= sharedConfig;
  };
}
EOF

sharedConfig could probably be factored out, though I don’t know if it matters since I think Nix should be able to tell if nix expressions defined in multiple places are effectively identical.

What would be the typical start up time of a container that doesn’t have any services or other extra initialization in this fashion? I’m hoping fairly quick - otherwise I may stick with a running container, possibly going with your original suggestion and firing up multiple instances and rotating between them as a compromise.

19

What exactly should these isolated environments contain?

20

Typically, these would just be custom-software environments (containing a number of packages from nixpkgs), so nothing very special. I might rely on the fact that these are using systemd-nspawn containers to enforce some restrictions, but these would be done at lower levels (iiirc through machinectl, cgroups, etc). But basically I was just using the term “isolated environment” in the usual sense of what a container is: isolated filesystems certainly, and isolated processes, but possibly other things as well - isolated users for instance. I assume the underlying packages, at least, are shared through the host’s /nix/store as usual.

21

Also, probably more details than is necessary, but here’s what we currently are doing: https://arxiv.org/pdf/2005.07252.pdf We have an old version of Singularity and NixOS containers at the moment, with other container types being supported a possibility, but I prefer to use Nix where possible.

22

In this case, extra-container (and NixOS containers) won’t exactly fit your requirements, because you probably want to use unprivileged containers (not yet supported by NixOS containers) and want to manage the container instances as subprocesses of your web app (and not as systemd services).
But NixOS is certainly a great platform for implementing CCRS.

If you still want to employ NixOS containers, have you considered to directly use the container service template (systemctl cat container@)?
Just store a configuration file at /etc/containers/<mycontainer>.conf and run systemctl start container@<mycontainer> to start an instance.

For further inspiration you might also want to look into nixcloud-container which uses unprivileged LXC containers.

1 Like
23

Con you explain to me why unprivileged containers arent possible with NixOS currently? is it because of the extensional model of the nix store? eg sharing a nix daemon and input based model vs the content addressed model?

I wish i could have a nixos container run like so nixos-container --user start foo

2 Likes
24

IIRC there was a bug in systemd-nspawn which kept us from using it, but I don’t know the current state on this. As I’m currently drafting a new containers-module (which uses systemd-networkd, see also Implement NixOS container networking with networkd · Issue #69414 · NixOS/nixpkgs · GitHub), I guess I’ll try that out when I have time to.

2 Likes
25

ok thanks for the information, i will follow your updates there