Deploying docker containers declaratively

Help
1

Is there a good way to deploy Docker containers on a NixOS machine declaratively, either with nixops or nixos-rebuild? Or would docker-compose be my best bet?

Normally I would use native nix packages, but some packages are a bit cumbersome to setup and docker is supported by upstream or it needs manual setup steps that the docker container has build in.

I only found ways to build docker images based on nix, but not how to deploy containers.

On that note I think I saw a project to handle imperative steps with nixops but I can’t seem to find it. Maybe that’d work?

3 Likes
2

When you install nix-built packages into a Docker image, you can provide a FROM statement to pull the base image from Dockerhub. If you need a supported base image (like CentOS for instance), you can declaratively pull it by sha256, then install your nixpkgs into it, and when built you will have a file that can be passed to docker load: Nixpkgs 23.11 manual | Nix & NixOS

Once you have a mycontainer.tar.gz, deployment is a matter of getting it to the target machine and running docker load -i mycontainer.tar.gz. If you’re already using NixOps, it can handle this for you – but most environments already have some sort of deployment infrastructure or tooling that does this type of thing.

3

That is mainly what I found when I searched for a solution. :confused: I am though looking for a way to run docker containers on NixOS, esp. for packages that are not in Nix yet. I.e. what docker-compose does, but using the nixos or nixops infrastructure.

4

Kubenix looks promising, but needs work in the documentation department. The modules have 7 contributors right now. Does anyone know more about the project?

2 Likes
5

I went looking for this and still haven’t found much other than running a whole Kubernetes stack (which seems like a lot of work). My solution so far has been to write systemd units that manage a docker container, which has been working reasonably well. for instance:

  systemd.services.prometheus = {
    description = "Prometheus";
    wantedBy = [ "multi-user.target" ];
    after = [ "docker.service" "docker.socket" ];
    requires = [ "docker.service" "docker.socket" ];
    script = ''
      exec ${pkgs.docker}/bin/docker run \
          --rm \
          --name=prometheus \
          --network=host \
          -v ${./prometheus.yml}:/etc/prometheus/prometheus.yml \
          -v prometheus_data:/prometheus \
          prom/prometheus:v2.2.1 \
          "$@"
    '';
    scriptArgs = lib.concatStringsSep " " [
      "--config.file=/etc/prometheus/prometheus.yml"
      "--storage.tsdb.path=/prometheus"
      "--web.console.libraries=/usr/share/prometheus/console_libraries"
      "--web.console.templates=/usr/share/prometheus/consoles"
      "--storage.tsdb.retention=30d"
      "--web.external-url=https://example.com/prometheus"
      "--web.route-prefix=/"
    ];
    preStop = "${pkgs.docker}/bin/docker stop prometheus";
    reload = "${pkgs.docker}/bin/docker restart prometheus";
    serviceConfig = {
      ExecStartPre = "-${pkgs.docker}/bin/docker rm -f prometheus";
      ExecStopPost = "-${pkgs.docker}/bin/docker rm -f prometheus";
      TimeoutStartSec = 0;
      TimeoutStopSec = 120;
      Restart = "always";
    };
  };

(I know there’s a Prometheus module in NixOS; I wanted a newer version than was available at the time and this was a quick solution)

This could be generalized into a reusable module in the same general way as systemd.services.* to provide an interface vaguely like what currently exists in containers.* for systemd-nspawn containres.

Hopefully there’s a better way though; what I’ve got here works, but it’s not ideal.

1 Like
6

Using systemd to manage docker comtainers while not perfect looks like it could make a simple solution for declaratively managing docker containers on a node. Would there be interest if a naive and simple PR was added to nixpkgs to do specifically this? I’m thinking a simple solution would be to use docker compose with a nice configuration and systemd.

7

I would be interested, for sure. I was considering starting to write such a thing but if you’re going to do it then by all means, go for it!

8

Related github issue (from March 2018): https://github.com/NixOS/nixpkgs/issues/37553

9

okay actually I decided to have a go at implementing this in a fairly simple way: https://github.com/NixOS/nixpkgs/pull/55179

What do you think?

1 Like
10

I’m curious, how do the Docker containers expose their services outside the container? Does it not require binding the right TCP/UDP ports to the ports inside the container when starting it up?

11

Thanks for doing this! I will comment on the PR.

12

It depends on which network mode you’re using, but typically one uses --publish xx:yy to forward port xx on the outer host to port yy inside the container. With --network=host, the container shares the host’s network namespace so no forwarding is needed. Some setups give a routable IP to each container on an internal network, with docker network or Calico or Weave or a number of other fancy networking options. The PR I published doesn’t try to address any of those (yet), but it would at least allow the user to set network-related options with extraDockerOptions for each service.

Summary: it’s complicated, and I’m trying to avoid the need to address all possible cases by offering a manual override.

13

I tried the new docker-containers module and it works perfectly so that I can use it to replace my previous Docker Compose based setup. Summarized here and hope it can help whoever in the same boat.

4 Likes
14

I am thinking of exploring systemd-nspawn - ArchWiki with Nixos Containers

https://nixos.org/nixos/manual/index.html#ch-containers