I’d like to define a docker image in nix using dockerTools on top of a pre-existing dockerfile-based docker image. Things seem easy to begin with, I can take the docker image (ubuntu) and copy in files with copyToRoot, but the instant I add something standard to paths all the binaries in the base image are no longer accessible.
tommd@wr /tmp% docker run --rm -it test-image:vfcdc5czsc95wdxxdp77biqngaqgb0rg bash
exec /usr/bin/bash: no such file or directory
But if I remove the /lib in pathsToLink then the /usr/bin from the original Ubuntu image is available. What’s going on here? Did it just blow away the prior /lib and thus shared libraries needed by bash? If so, how can I unify the two? The ideal is a copy semantics of the files in the paths from the listed derivations into the fromImage layer.
I don’t know about the semantics of pathsToLink, but usually you should be able to omit that argument. When I used buildEnv for creating user environments using paths was enough. I looked at the buildEnv source and couldn’t spot anything suspicious, but I also don’t know perl very well.
If you want to inspect the image contents, you can try dive and skopeo:
I don’t know about the semantics of pathsToLink, but usually you should be able to omit that argument.
Sadly simply omitting the pathsToLink is not a solution. For example, the base Ubuntu image has lots of files in /bin including /bin/bash (rather important for many uses). A trivial buildEnv withone package results in a mostly empty /bin.
Ok, I see. Yeah it seems the default is “/”, so it’s really not helping. Unfortunately I don’t understand the ins and outs of that enough to offer a solution. My only idea is to use buildLayeredImage instead of buildImage but I think it does the same thing, it just creates layers to make rebuilds and updates faster.
But would not using fromImage an option for you? You could add everything you need via paths =, including bash etc., but I assume there’s something specific in Ubuntu that you need?
Absolutely using just Nix would be easier. Sadly I need the underlying image. There are lots of reasons this can be the case, the reason today is to provide a familiar environment to people for use in CI. I just want to enhance that image with some tools that have nix derivations, but it seems the easiest solution is to just package for Ubuntu (pure Docker no nix) or to use nix-pkgs on top of Ubuntu (still inside of Dockerfile and not a nix-built docker image).
layered image actually has much the same semantics as buildImage. For example using buildLayeredImage with the prior fromImage and contents of [ pkgs.gnumake ] ; we get:
root@9f45ba7fd608:/# ls /bin
make
root@9f45ba7fd608:/#
# !!! LayeredImage doesn't have the bug where copies instead of links are made to the image root
I don’t want to invalidate what you’re seeing, of course, I’m wondering what might be the difference.
Maybe, @ppenguin, could you confirm that’s what you meant by that comment? Or is this about something different?
I seem to remember from anecdotal testing that for me the image size was larger when I used buildImage and that I saw in some /bin/... paths binaries instead of links to /nix/store paths. With buildLayeredImage these were links as expected, so the comment stands, and I’d consider it a bug or at least undocumented and undesireable behaviour.
Though, come to think of it, if at the same time the behaviour of buildImage guarantees that we don’t need any /nix/store paths (because it copies?) and we could purge the /nix path from the image before committing it, then I suppose it would be workable too, with the advantage (?) that the image wouldn’t have “nix peculiarities”, whatever that may mean for its functionality.
But I’m quite happy with how my devops-flake turned out, I’m using it “in production” now. (I’ll check whether I need to sync the last optimisations, which were done in an internal repo)
docker load < $(nix-build Dockerfile.nix) 0.89s user 0.21s system 17% cpu 6.164 total
tommd@wr /tmp% docker run --rm -it owner:latest bash
root@f66c66371ee9:/# ls /bin
make
Which is wrong - a base image with Ubuntu should have lots more files in bin.
I’ll make a github issue for this topic based on what ppenguin said.
As for the suggestion from @mickours, I’d like to have a system that can marry a Ubuntu base image with tooling defined via nix derivations. Preferably the something can be nix in nature (vs installing nix pkgs in a ubuntu docker container). So your suggestion fits the bill. I haven’t gotten it to work yet as I’m still inexperienced enough with Nix and the Flake just isn’t working out. Something for me to play with.
% nix run .#amuse.copyToDockerDaemon
...
% docker run -it amuse:xy5d2qwisl43sbry2sdms3adzqyvxkrx bash -c 'ls /bin'
make
%
So it seems nix2container has the same semantics. Its documentation is clear on this intent though: “The store path content is then located at the image.”
What would be interesting in light of the above, is to see whether the stuff in /bin/ is a link to a /nix/store path or just plain binaries (ls -la), and or whether /bin itself is a regular dir and not a link (which I think it always is?). And in the case there are no links to /nix/store paths here, are there any such paths in the container, which would mean there would be duplicates?
% docker run --rm -it amuse:hnmbm7kqrmx90xyl6nz6vq7rxnvzfpjs bash
root@b8532b122113:/# ls -l bin
total 4
lrwxrwxrwx 1 root root 66 Jan 1 1970 make -> /nix/store/mfhgnnf31vlylgaddfw70knygh0j8mac-gnumake-4.4.1/bin/make
root@b8532b122113:/# ls -la | grep bin
dr-xr-xr-x 2 root root 4096 Jan 1 1970 bin
bin is a regular dir and make is a soft link into /nix/store as expected.
Ahh, but here is something interesting. The Ubuntu image does not have /bin as a regular directory. It is a soft link to /usr/bin:
% docker run --rm -it ubuntu@sha256:83f0c2a8d6f266d687d55b5cb1cb2201148eb7ac449e4202d9646b9083f1cee0
root@06303a4ca2cd:/# ls -l | grep bin
lrwxrwxrwx 1 root root 7 Jun 5 14:02 bin -> usr/bin
Perhaps if that were not a soft link then we’d get the desired behavior.
And indeed the github issue has received a response immediately calling out this behavior. If the path is a softlink then the final image will have the path shadowed by the directory create for the files from the nix packages. The ticket is still open in bug status, so known or not we might see a change at some point.
To add closure to this issue in case someone finds it via search: I handled the issue using runAsRoot (below). The linked GitHub issue has an alternative but it isn’t turn key.
My run as root is simple. Works for me, but only because I have kvm, which isn’t true for ec2 users among others:
runAsRoot = ''
#!${pkgs.stdenv.shell}
if [[ ! ( -L /sbin ) ]] ; then
for i in $(ls /usr/sbin) ; do
ln -sf /usr/sbin/$i /sbin/$i || true
done
fi
for i in $(ls /usr/bin) ; do
ln -s /usr/bin/$i /bin/$i || true
done
'';
