I’m using buildEnv to generate a container rootfs, which is working well, except it’s missing “/home/user” and similar directories.
If I create them as part of buildEnv, they’re readonly, and with runCommand I can chmod but never make it writeable.
I’d like to use nix to generate a ‘result’ folder that I can tar, and which already has the correct chmod etc.
Is that possible? I assume it must be possible, since NixOS is able to generate a whole FS which has writeable folders.
I’m still a beginner so I’m not sure I’m understanding all the concepts, but I assume I’ll have to have to use “nix run” to “nix build” and then run “docker import < result”. Are there other ways to have side effects, other than in “nix run”?
I actually have a similar problem with my nix-snapshotter-based (actually nix-storage-plugin as I use podman) image for local single-node k3s experimentation, which I solve with a giant finit wrapper:
But I would like to know better solutions, even for dockertools that I can port to nix-snapshotter.
My immediate thought: Run the tar command inside the derivation. So the resultis the archive.
Everything in the nix store (not counting incomplete $out contents during a derivation run) is either 444 or 555 permissions. Write permissions are not possible. If you want a file with write permissions, it needs to not be in the nix store. Period.
dockertools is for building tar files, and only works in a single build step.
What I’m interested to do is producing separate layers using buildEnv, and combining them, then ultimately producing a tar.
as @tejing said, tar is a good solution, but it requires a lot of packing and unpacking, whereas I’m only interested in packing once at the final build step.
I am also considering just using a nixpkgs docker image to build the whole thing, and then snapshot the final rootfs, but this makes it more complicated to share the nix/store with the host, and requires more copying, protocol overhead, and introduces issues with sandboxing builds.
I still fail to see the purpose. If you don’t care about the layer separation, because you intend to create a collapsed tarball, you shouldn’t care about the intermediate steps. If you do care about layer separation, combining the layers seems like the exact opposite of what you want to do. If you ultimately want to use docker import, building a tarball seems like the wrong thing to do, you should want an OCI image.
This reeks of XY problem, you’re still only describing what you think you need, rather than the problem you’re trying to solve.
You could use the same trick streamLayeredImage uses and produce a script that outputs the tarball on stdout at runtime, instead of producing a tarball artifact in the store.
I see. Let me try to describe what I want to achieve:
compose containers with modules
for example, installing wayland, sway, dbus is not enough, you need additional configuration files and startup scripts, so it makes sense to bundle them together as a module.
quick iteration of the development of container compositions
for example: using buildEnv can construct a rootfs in seconds, and mounting nixstore with systemd-nspawn, you can start a container in seconds with minimal copying.
apparently podman also supports rootfs, but I havent tested it.
compatible to publish to OCI / kubernetes
I therefore thought to use buildEnv for merging and runCommand for modifying chmod.
I’m thinking to follow @bitbloxhub ‘s example and maybe just append scripts to the rootfs, e.g. /extracommands.d/ which dockertools/nspawn can then execute to prepare the rootfs.
Am I correct that nix profile add basically just does a buildEnv merge into /root/.nix-profile/ ? Or does it do something more sophisticated?
Right, then you’re designing a new library and you need to consider what the appropriate abstraction is.
What that is will ultimately depend on what you’re intending to do in the end, but this is my take for a general-purpose rootfs builder:
This is the approach usually taken in the docker world. Personally I’m not very fond of it, since it requires effectively running an init system in the container, and giving its PID1 “root” permissions, which makes the container much more complex and has security implications.
This can be a fine trade-off, of course, but personally I would provide the same kind of abstraction streamLayeredImage does to reduce container complexity instead.
If you don’t like doing that at runtime, adding the same abstraction as buildLayeredImage in addition to streamLayeredImage lets users choose their trade-off.
How you expose the concept of layers is a different question. Honestly, given nix, I don’t think a layers concept is needed at all - the individual components of your buildEnv will be cached if unchanged anyway, and the user can’t see the layers so you cannot possibly get the storage space benefits layers in OCI images provide.
The build cost of changing any component will always be the same, the only step that could theoretically be sped up is creating the combined rootfs, but since we need to create a tarball there is no way to save effort in a declarative world (especially without making changes to nix - if nix was aware of tarball formats, and had some kind of built-in tarball-building derivation, you could perhaps modify store paths in place).
In other words, look at streamLayeredImage, I don’t think it’s actually possible to design a better interface given the constraints. But you can implement something similar to create generic root file systems.
I think I’ll go with appending scripts to a folder in the rootfs, i.e. “activation scripts”, that should be run in docker/nspawn in the build stage. This avoids any tarballing/copying/compression at all. And is very simple.