I am trying to install Nix on a Docker image running in Kubernetes, where my user’s home directory is mounted from an AWS FSx instance. In the image I run:
RUN sudo mkdir -m 0755 /nix && chgrp users /nix
RUN chown 1000 /nix
Then after building the image and SSHing in, I run a single-user install as I don’t have root access:
sh <(curl --proto '=https' --tlsv1.2 -L https://nixos.org/nix/install) --no-daemon
. /home/user/.nix-profile/etc/profile.d/nix.sh
However, when trying to build my Nixvim flake I get these errors around cp operations not being permitted. It isn’t plugin specific, I can disable as much as I want and some other derivation encounters this same issue:
$ nix build
Stack size hard limit is 10485760, which is less than the desired 62914560. If possible, increase the hard limit, e.g. with 'ulimit -Hs 61440'.
[0/51 built, 0/2 copied] Stack size hard limit is 10485760, which is less than the desired 62914560. If possible, increase the hard limit, e.g. with 'ulimit -Hs 61440'.
error: Cannot build '/nix/store/p5zxcxja8dn9zdlcfbigrnxrwd09fi45-vimplugin-blink.cmp-1.7.0.drv'.
Reason: builder failed with exit code 1.
Output paths:
/nix/store/kr3vwkjihx21pnapx4wbj1vilnmqjmf4-vimplugin-blink.cmp-1.7.0
Last 10 log lines:
> structuredAttrs is enabled
> Sourcing vim-gen-doc-hook
> Sourcing vim-command-check-hook.sh
> Using vimCommandCheckHook
> Sourcing neovim-require-check-hook.sh
> Using neovimRequireCheckHook
> Running phase: unpackPhase
> unpacking source archive /nix/store/4srj5xj47d74rc37vc8k7w0chplmia7p-source
> cp: setting permissions for 'source': Operation not permitted
> do not know how to unpack source archive /nix/store/4srj5xj47d74rc37vc8k7w0chplmia7p-source
For full logs, run:
nix log /nix/store/p5zxcxja8dn9zdlcfbigrnxrwd09fi45-vimplugin-blink.cmp-1.7.0.drv
error: Cannot build '/nix/store/xzz817mpjrji4jvrdsv4969mf81cd6za-neovim-0.11.4.drv'.
Reason: 1 dependency failed.
Output paths:
/nix/store/f42hwqdvh239hv11l74mdz9yy0bcw45p-neovim-0.11.4
error: Cannot build '/nix/store/scpk1cq9cya3kkjlppkra7rvb51mcx2a-packdir-start.drv'.
Reason: 1 dependency failed.
Output paths:
/nix/store/b7ljyjcn0m0jx225mv947rxmznykgf7f-packdir-start
error: Build failed due to failed dependency
error: Cannot build '/nix/store/5ych0cr0hak8i7yppqh2z3miqjgkdb2q-nixvim.drv'.
Reason: 1 dependency failed.
Output paths:
/nix/store/zr88mawv9w2ld7vsryv32jq0slcyhdfb-nixvim
error: Build failed due to failed dependency
I’m guessing it has something to do trying to build outputs/temporary values in a directory I don’t have permissions for on the FSx filesystem, but I can’t figure out how to diagnose more or fix it.
I was able to solve this by installing Nix during docker build via multi-user installation and also installing my packages at build time as well. I will list below my unsuccessful attempts at installing Nix and being able to use it in the image to install packages.
- Multi-user installation. This requires the daemon to be running, but that needs sudo to start which I don’t have access to when SSH’d into the image built and deployed via our CI/CD pipeline. I tried various methods of getting the root user to start the daemon when the image started but before switching to my user, and while some of them worked when building the image locally, it always failed when building via our CI/CD.
- Single-user installation. This installs, but because our Ci/CD mounts an AWS FSx filesystem as our home user, anything written there during image build is overwritten. There may be ways around this, like creating symlinks to the nix store outside of home, so it would be preserved. But I didn’t want to mess with that incase anything expects files to be in their default locatoins.
- Single-user installation, but installed after image build. I created the /nix directory in build while having root, then ran the Nix install ater build inside the image. This also successfully installs, but I was unable to install packages due to errors like the above:
nix build
Stack size hard limit is 10485760, which is less than the desired 62914560. If possible, increase the hard limit, e.g. with 'ulimit -Hs 61440'.
[0/51 built, 0/2 copied] Stack size hard limit is 10485760, which is less than the desired 62914560. If possible, increase the hard limit, e.g. with 'ulimit -Hs 61440'.
error: Cannot build '/nix/store/p5zxcxja8dn9zdlcfbigrnxrwd09fi45-vimplugin-blink.cmp-1.7.0.drv'.
Reason: builder failed with exit code 1.
Output paths:
/nix/store/kr3vwkjihx21pnapx4wbj1vilnmqjmf4-vimplugin-blink.cmp-1.7.0
Last 10 log lines:
> structuredAttrs is enabled
> Sourcing vim-gen-doc-hook
> Sourcing vim-command-check-hook.sh
> Using vimCommandCheckHook
> Sourcing neovim-require-check-hook.sh
> Using neovimRequireCheckHook
> Running phase: unpackPhase
> unpacking source archive /nix/store/4srj5xj47d74rc37vc8k7w0chplmia7p-source
> cp: setting permissions for 'source': Operation not permitted
> do not know how to unpack source archive /nix/store/4srj5xj47d74rc37vc8k7w0chplmia7p-source
For full logs, run:
nix log /nix/store/p5zxcxja8dn9zdlcfbigrnxrwd09fi45-vimplugin-blink.cmp-1.7.0.drv
error: Cannot build '/nix/store/xzz817mpjrji4jvrdsv4969mf81cd6za-neovim-0.11.4.drv'.
Reason: 1 dependency failed.
Output paths:
/nix/store/f42hwqdvh239hv11l74mdz9yy0bcw45p-neovim-0.11.4
error: Cannot build '/nix/store/scpk1cq9cya3kkjlppkra7rvb51mcx2a-packdir-start.drv'.
Reason: 1 dependency failed.
Output paths:
/nix/store/b7ljyjcn0m0jx225mv947rxmznykgf7f-packdir-start
error: Build failed due to failed dependency
error: Cannot build '/nix/store/5ych0cr0hak8i7yppqh2z3miqjgkdb2q-nixvim.drv'.
Reason: 1 dependency failed.
Output paths:
/nix/store/zr88mawv9w2ld7vsryv32jq0slcyhdfb-nixvim
error: Build failed due to failed dependency
I hope this helps/is a starting point for anyone else battling getting Nix working in a fairly locked down and non-standard development environment at work.