For the past year or so I’ve been using deven.sh as a tool for building my development environments while building Docker images and/or static binaries in CI, using Docker. I’d now like to replace my CI/CD workflow with something that uses Nix instead. I’m going all in!
One problem though: I haven’t quite figured out is how to extend this experience all the way to deployment. What I want to achieve, specifically, is:
Have the same environment as in dev, to run tests + build artifacts, during CI
Some projects require me to build docker images, so I need to find some way to do that too
What I think I need to do, is:
Convert my devenv.nix file into flake.nix file.
Define my build in said flake file (if I’m building binaries)
Define a Docker build in said flake file (if I’m building Docker images)
Have a GitLab runner that’s running NixOS OR a GitLab runner on a Linux machine with the Nix package manager installed
Publish the result as an artifact or as Docker image in a registry
I’m currently researching the first few steps and I’m starting to doubt my approach. Are flakes the recommended way? Are there alternatives worth looking into? There seems to be some conflicting information on the interwebs. Some people like them, some people don’t.
Is the building of a Docker image even necessary? I’m in complete control of my own environments, so I could deploy just as well to a NixOS server. This would also help eliminate a handful of Ansible playbooks.
Where do Makefiles fit into this story? Do they get replaced with Nix, seeing as there’s some overlap between the two, or do they make a great pair?
It would be much appreciated if someone who went down a similar path could share their experiences and point me in the right direction
(Note: none of these applications are used commercially, so there’s some time and room for experimentation and learning things properly)
As you mentioned, depend on who you ask. I would use.
Flake is stable enough, “enough” depends on context.
In your case, what is the company policy on stability? ie: Some companies wait N years after release of a software and only upgrade patches, flakes is not in that level of maturity.
What you use as distro in prod? ie: CentOS 6 (do not use flakes) or ubuntu:latest (go use flakes).
No, but if you ship docker images, start with small changes, more chance of success. But more time to complete.
It depends, you could make build work for non-nix users. And nixpkgs¹ integrates well with makefile.
¹ nixpkgs is not nix but, we always use it anyway.
Based on your feedback and some research, I’ve decided to go the raw nix flake route. Why? Well, I’m using PHP in this example (lots of moving parts, seemed like a very good use-case to play with) and the PHP configuration between dev and prod is different. In dev you typically have xdebug enabled and a higher memory limit. Seems like something that devenv doesn’t support, building containers with a different config than your devshell?
I’m currently trying to come up with an alternative for devenv up
As for the CI and docker building part, that I’ve got a pretty good idea about how that should work. It seems pretty straight forward:
Either run nix inside a docker container and use DIND to build images OR use a CI runner with a Linux distro of choice which has the nix package manager + docker installed
I plan on running my unit tests using nix flake check
Build docker images using pkgs.dockerTools.buildLayeredImage
I’m investigating similar migration / upgrade path having currently a GitLab CI / CD and development + deployments environments using Docker.
I would also like to remove as much dependency to Docker as possible too.
I believe that Docker brings some practicability for deployments but when it comes to CI / CD, it is often “misused” and people try to tackle and mix all within Docker (hosting, networking, runtime, persistence, etc).
Quite tricky to migrate “out of it”.
Do you happen to have a diagram of your current CI / CD workflow and deployment / build strategy?
It may be easier to overview migration and needs with a draft of your final expected and “ideal” workflow then compare it with your existing ones.
Some thoughts:
nix-shell (with or without flakes) would be a great fit for development environments, not sure how would that work for e.g DB, redis (anything that requires persistence)
consider GitLab templates to re-use nix configuration across projects / pipelines
do you really need Docker? or was it “go to / easy” solution at the time of implementation? why not running a NixOs environment instead?
if smth can be build with Docker, it will likely be also possible with Nix / NixPkgs - up to you to decide which tool has that responsibility (thinking separation of concerns e.g Docker for runtime, Nix to build and compile)
I’m not sure if the documentation is correct (never test it), but I’m curious, maybe @domenkozar could help. What is the magic to make it work for default and non default shell?
When you use devenv, state is kept in $PWD/.devenv/state. If you decide to roll your own setup, you’ll probably need to do something similar.
I don’t necessarily need docker, but I’m not familiar enough with Nix (and NixOS) to migrate everything away from the setup that I currently have. The plan is to further Nix-ify my development environment + use it during CI/CD. Also, I feel like Docker is an excellent tool for deploying stuff. If you can make sure that what’s in your container is exactly the same as what you use in dev, I don’t really see the need to move away from docker, yet.
I don’t, actually. This is just something I’m exploring in my own time to deploy/play with pet projects
Interesting! So in theory, it should be possible to switch packages during a docker build?
I now have something like things:
# Prod version of PHP, to be included in our docker image.
phpProd = with pkgs; (php83.buildEnv {
extensions = { enabled, all }: enabled ++ (with all; [
apcu
]);
});
# Dev version of PHP, to be used in our devshell.
# It's the same as the prod version, but with xdebug enabled.
phpDev = with pkgs; (php83.buildEnv {
extensions = { enabled, all }: enabled ++ (with all; [
apcu
xdebug
]);
});
The config is different as well, but that’s currently not living in the nix store but in the project dir itself. (or is it best to keep config in the store as well?)
If all deployments are running with docker, then build docker images during in GitLab runners (e.g “build stage”) and ship then to deployment host(s) (e.g “deploy stage”).
That also let you run projects with locally with docker and tackle possible issue ahead.
e.g docker related ones that can’t be seen if using nix shell / env
I’ve been experimenting a lot and I think I’ve a solution that fits my situation well:
I’ve settled on using raw nix flakes, keep things simple
build base and builder docker images using pkgs.dockerTools.buildImage
I have a Dockerfile that defines the images used for testing and deployment. I don’t expect this Dockerfile to change much during the development of my project.
use overmind to launch/manage services during development
both the dev and prod config lives in my repository, under a env/{dev,prod} directory
There’s more tooling involved than I first anticipated, but using the right tool for the right job makes things a lot easier to understand and, more importantly, easier to troubleshoot/replace with something else.
This setup still needs a bit of polish though. I’ll try to link back a git repo with the final result later
It’s pretty much an experiment, but I feel like I can use this going forward. There are some workarounds that I’ve had to apply, like stuff with cacert and /usr/bin/env that feel a bit wonky, but I’ve also learned why those work the way that they do.
how would you approach maintenance? e.g bumping node and php version (assuming keeping things as up to date as possible is a requirement)
=> I would assume if e.g php needs an upgrade, you would have to change docker AND nix current configs?
are you still using GitLab / GitLab runners?
why not removing all docker stuff altogether? to me it’s not relevant to your app (symfony stack) itself but to your target host which is a different “layer”
=> you mentioned pkgs.dockerTools.buildLayeredImage, have tried with it? then you would have “full nix” config I guess and when deploying, the deploy.sh would pickup that pkgs.dockerTools.buildLayeredImage build
=> you do want to have clear boundaries on host VS app, docker VS nix setup, that would ease of maintenance, changes and clarify your project design
If you can use NixOS I personally would go that route and ditch containers.
At work however we use Ubuntu as the base OS and that’s why I still have to build containers.
We then use Ansible to get the server to pull its newest image and recreate the service.
Depending on your language you might still need them I guess but Nix uses them for you, I’m not good with compiled languages.
As a task runner in general I stopped using any third party tools and now just use a simple bash script that you can envoke with dev some-task.
As for a more complete example, this project uses Python, poetry2nix and builds container images using Github Actions.
Keep in mind it’s more of a PoC for me not a completely finished project yet but it builds and runs.
The docker-compose.yml is only for “production”.
The CI part should be fairly easy to move to another service once you figured out how to do the Nix stuff, after all that’s one reason why we use Nix ;).
Have a look through @MatejaMaric’s posts. He recently asked a lot of good questions related to PHP and has a public repository.
Maybe you can get inspiration there as well.
I’m currently in the process of modularizing the setup and moving/extracting all config to a separate nix file.
Yep, but haven’t had time yet to write a CI file. I’ve got a GitLab runner with Nix installed ready to go though
I’ve since replaced the docker stuff with nix2container! There’s now a separate config for everything container related (= deployment) and everything devshell related (=local development).
