I’m using Nix to provide a reusable and isolated development environment, but I also want to use my flake to generate the Docker image that would be used in CI.
Here is what I’ve done:
{
description = ''
A Nix flake for X and Y
'';
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11";
flake-utils.url = "github:numtide/flake-utils";
rust-overlay = {
url = "github:oxalica/rust-overlay/stable";
inputs = {
nixpkgs.follows = "nixpkgs";
flake-utils.follows = "flake-utils";
};
};
};
outputs = { self, nixpkgs, flake-utils, rust-overlay, ... }: flake-utils.lib.eachDefaultSystem(system:
let
pkgs = import nixpkgs {
inherit system;
overlays = [ rust-overlay.overlays.default ];
};
rustVersion = "1.75.0";
# Dependencies used for development and CI
depsBuild = with pkgs; [
# ...
];
# Dependencies used only for development
depsBuildDev = with pkgs; [
rust-bin.stable.${rustVersion}.default
# ...
];
# Dependencies used only for CI
depsBuildCi = with pkgs; [
rust-bin.stable.${rustVersion}.minimal
# ...
];
in
{
devShells.default = pkgs.mkShell {
buildInputs = depsBuild ++ depsBuildDev;
};
packages = {
xy = pkgs.dockerTools.buildImage {
name = "xy";
tag = "latest";
created = "now";
config = {
cmd = ["/bin/sh"];
# volumes = {
# "/code" = {};
# };
workingDir = "/code";
env = [
"CARGO_HOME=/code/.cargo"
];
};
copyToRoot = pkgs.buildEnv {
name = "image-root";
paths = with pkgs.dockerTools; [
# Includes ls, cat, etc.
pkgs.coreutils
# Include an interactive shell for easier introspection and debugging
binSh
usrBinEnv
] ++ depsBuild ++ depsBuildCi;
pathsToLink = ["/bin" ];
};
};
};
});
}
So, I’ve split my dependencies between those that are common, those that are specific to development and those that are just for CI.
The problem that I cannot solve is how to copy Cargo.toml and Cargo.lock into the Docker image, so I could fetch the dependencies and include them in the CI image too.
I’d like finding an approach that works with any language, since I’d like to use Nix for other projects.
I’m struggling to follow what you’re trying to achieve a little here - are you trying to generate a docker container that would contain the project source code, and also all the rust libraries used in the project, so you can test and build binaries and such without having to download things from crates.io?
Are you intending to rebuild this image on every commit (and grabbing new dependencies from Cargo.lock) so you can send it to the CI server? Why not just use nix to do the CI checks on commit instead if you’re going to have to build an image to do that everytime anyway?
Yes, I’d like to generate a Docker image with access to the code (at least to Cargo.toml, package.json, etc.) and the Rust/Cargo libraries and other dependencies.
I’d like to use cargo fetch to download and include all the dependencies in the Docker image once, since they’re not going to change in the next months, instead of doing that step on every commit or using some kind of cache/artifact/volume to store them temporarily.
Our CI mounts the code of the commit as a volume in the container, so we’d be able to reuse those fetched crates and the packages stored in the image to run the CI steps (linting, testing, etc.) for each commit.
Fair enough, pretty bespoke requirements. Depending on your requirements for build reproducibility and distributability you’d probably need to either:
Do something similar to what dream2nix is doing to parse and fetch the dependencies with nix
Have a look through their sources, if you use that project as a dependency this kind of stuff is exposed to you for ease of use.
Make an FOD from a cargo fetch and put the dependencies in the image that way (which I believe is also supported by dream2nix).
This is quite easy, just a normal mkDerivation with your sources as input and then cargo fetch, making sure you mark it as a FOD by specifying a hash to give you internet access.
I think dream2nix also supports this.
I’d still dissuade you from either approach. Months are short in project lifecycle times, and a simple cargo update - which you should probably do more frequently than months - will obsolete your CI. It also hinders project contributors, who might want to experiment on feature branches - even if that code never makes it to mainline, shimming in a crate temporarily can sometimes make development easier. You’re setting yourself up for high project maintenance effort for a pretty small short-term gain.
Fetching a few crates at CI time shouldn’t be too slow, but if that is a hindrance it’s probably better to look into your CI-native caching options than to abuse OCI images.
Your proposal of making a derivation was something that I was considering, but I’m not experienced yet and I was expecting a simpler solution.
Anyway, I’m starting to consider what you propose about using the CI caching options. I’d have to check it again, because the last time it was painfully slow, like 30-60 seconds per job, and we have 6-8 jobs. So, most of the time was spent not running our commands, but loading the cache, compiling, updating the cache, etc.
We could merge those jobs into 1, but spotting which are the problematic actions would be more involved, so it’s a trade-off.
30-60 seconds for unarchiving a cache does sound slow. Perhaps you were caching too much stuff, or data was being stored far away from the machine actually using it? No idea what CI tooling you’re using, I’d personally try to figure out what’s going wrong there before jumping ship.
That said, 30-60 second CI if the rest of the job is also on that timescale doesn’t seem too bad from my perspective. Pushing commits to see if the build is working should not be necessary if you’re using nix, so IME your contributors should not need to rely on CI enough for that kind of timescale to meaningfully slow things down. But well, you probably know best, hard to comment without more knowledge about the project.
Yeah, I’m going to try it and, if it is as slow as I remember, I might be able to investigate it on my own or talk with the people who is responsible for that part of the infrastructure.
Not everyone is using Nix yet. Currently they install the dependencies using the regular methods of their OS. I introduced Nix to avoid all the problems that this usually provoke, to simplify the onboarding, etc, but, to my knowledge, I’ve not yet convinced anyone to try it.
I hope that in other projects, with more languages and moving parts, it’s going to be more appreciated.