Use Nix, from dev to prod

Hello there!

For the past year or so I’ve been using deven.sh as a tool for building my development environments while building Docker images and/or static binaries in CI, using Docker. I’d now like to replace my CI/CD workflow with something that uses Nix instead. I’m going all in!

One problem though: I haven’t quite figured out is how to extend this experience all the way to deployment. What I want to achieve, specifically, is:

  • Have the same environment as in dev, to run tests + build artifacts, during CI
  • Some projects require me to build docker images, so I need to find some way to do that too

What I think I need to do, is:

  • Convert my devenv.nix file into flake.nix file.
  • Define my build in said flake file (if I’m building binaries)
  • Define a Docker build in said flake file (if I’m building Docker images)
  • Have a GitLab runner that’s running NixOS OR a GitLab runner on a Linux machine with the Nix package manager installed
  • Publish the result as an artifact or as Docker image in a registry

I’m currently researching the first few steps and I’m starting to doubt my approach. Are flakes the recommended way? Are there alternatives worth looking into? There seems to be some conflicting information on the interwebs. Some people like them, some people don’t.

Is the building of a Docker image even necessary? I’m in complete control of my own environments, so I could deploy just as well to a NixOS server. This would also help eliminate a handful of Ansible playbooks.

Where do Makefiles fit into this story? Do they get replaced with Nix, seeing as there’s some overlap between the two, or do they make a great pair?

It would be much appreciated if someone who went down a similar path could share their experiences and point me in the right direction :sweat_smile:

(Note: none of these applications are used commercially, so there’s some time and room for experimentation and learning things properly)

As you mentioned, depend on who you ask. I would use.

Flake is stable enough, “enough” depends on context.
In your case, what is the company policy on stability? ie: Some companies wait N years after release of a software and only upgrade patches, flakes is not in that level of maturity.
What you use as distro in prod? ie: CentOS 6 (do not use flakes) or ubuntu:latest (go use flakes).

No, but if you ship docker images, start with small changes, more chance of success. But more time to complete.

It depends, you could make build work for non-nix users. And nixpkgs¹ integrates well with makefile.

¹ nixpkgs is not nix but, we always use it anyway.

Thanks for your answer @hugosenari!

Based on your feedback and some research, I’ve decided to go the raw nix flake route. Why? Well, I’m using PHP in this example (lots of moving parts, seemed like a very good use-case to play with) and the PHP configuration between dev and prod is different. In dev you typically have xdebug enabled and a higher memory limit. Seems like something that devenv doesn’t support, building containers with a different config than your devshell?

I’m currently trying to come up with an alternative for devenv up :sweat_smile:

As for the CI and docker building part, that I’ve got a pretty good idea about how that should work. It seems pretty straight forward:

  • Either run nix inside a docker container and use DIND to build images OR use a CI runner with a Linux distro of choice which has the nix package manager + docker installed
  • I plan on running my unit tests using nix flake check
  • Build docker images using pkgs.dockerTools.buildLayeredImage
1 Like

I’m investigating similar migration / upgrade path having currently a GitLab CI / CD and development + deployments environments using Docker.
I would also like to remove as much dependency to Docker as possible too.

I believe that Docker brings some practicability for deployments but when it comes to CI / CD, it is often “misused” and people try to tackle and mix all within Docker (hosting, networking, runtime, persistence, etc).
Quite tricky to migrate “out of it”.

Do you happen to have a diagram of your current CI / CD workflow and deployment / build strategy?
It may be easier to overview migration and needs with a draft of your final expected and “ideal” workflow then compare it with your existing ones.

Some thoughts:

  1. nix-shell (with or without flakes) would be a great fit for development environments, not sure how would that work for e.g DB, redis (anything that requires persistence)
  2. consider GitLab templates to re-use nix configuration across projects / pipelines
  3. do you really need Docker? or was it “go to / easy” solution at the time of implementation? why not running a NixOs environment instead?
  4. if smth can be build with Docker, it will likely be also possible with Nix / NixPkgs - up to you to decide which tool has that responsibility (thinking separation of concerns e.g Docker for runtime, Nix to build and compile)

I’m not sure if the documentation is correct (never test it), but I’m curious, maybe @domenkozar could help. What is the magic to make it work for default and non default shell?

1 Like

See https://github.com/flakestry/flakestry.dev/blob/ffc7426a225b4291520e18651ac72276390a0880/devenv.nix#L62 how to enable things only when inside the container.

2 Likes

When you use devenv, state is kept in $PWD/.devenv/state. If you decide to roll your own setup, you’ll probably need to do something similar.

I don’t necessarily need docker, but I’m not familiar enough with Nix (and NixOS) to migrate everything away from the setup that I currently have. The plan is to further Nix-ify my development environment + use it during CI/CD. Also, I feel like Docker is an excellent tool for deploying stuff. If you can make sure that what’s in your container is exactly the same as what you use in dev, I don’t really see the need to move away from docker, yet.

I don’t, actually. This is just something I’m exploring in my own time to deploy/play with pet projects :stuck_out_tongue:

Interesting! So in theory, it should be possible to switch packages during a docker build?
I now have something like things:

      # Prod version of PHP, to be included in our docker image.
      phpProd = with pkgs; (php83.buildEnv {
        extensions = { enabled, all }: enabled ++ (with all; [
          apcu
        ]);
      });

      # Dev version of PHP, to be used in our devshell.
      # It's the same as the prod version, but with xdebug enabled.
      phpDev = with pkgs; (php83.buildEnv {
        extensions = { enabled, all }: enabled ++ (with all; [
          apcu
          xdebug
        ]);
      });

The config is different as well, but that’s currently not living in the nix store but in the project dir itself. (or is it best to keep config in the store as well?)

I don’t, actually. This is just something I’m exploring in my own time to deploy/play with pet projects :stuck_out_tongue:

It would give a nice overview and organize components’ responsibilities, possibly reduce items so less maintenance

Have you tried possibly docker in nix?
At least for development host(s) and runners (not deployment host(s)), so you can start using nixos.

If all deployments are running with docker, then build docker images during in GitLab runners (e.g “build stage”) and ship then to deployment host(s) (e.g “deploy stage”).
That also let you run projects with locally with docker and tackle possible issue ahead.
e.g docker related ones that can’t be seen if using nix shell / env

I’ve been experimenting a lot and I think I’ve a solution that fits my situation well:

  • I’ve settled on using raw nix flakes, keep things simple
  • build base and builder docker images using pkgs.dockerTools.buildImage
  • I have a Dockerfile that defines the images used for testing and deployment. I don’t expect this Dockerfile to change much during the development of my project.
  • use overmind to launch/manage services during development
  • both the dev and prod config lives in my repository, under a env/{dev,prod} directory

There’s more tooling involved than I first anticipated, but using the right tool for the right job makes things a lot easier to understand and, more importantly, easier to troubleshoot/replace with something else.

This setup still needs a bit of polish though. I’ll try to link back a git repo with the final result later :wink: