Out of memory copying to nix store [SOLVED]

I have an v18.03 1GB ram VM. When I try to upgrade it via NixOps, it wants to install the nonfree linux firmware, a 500MB derivation. This fails because the system goes out of memory.

Even when I try to add it locally, it crashes:

[root@tracy:~]# nix-shell -p firmwareLinuxNonfree
these paths will be fetched (130.10 MiB download, 655.72 MiB unpacked):
  /nix/store/0j1sc30kjf9b3j7j0sp68jns2v34apr0-pcre-8.42
  /nix/store/0y7jmqnj48ikjh37n3dl9kqw9hnn68nq-binutils-2.31.1
  /nix/store/5lyvydxv0w4f2s1ba84pjlbpvqkgn1ni-linux-headers-4.19.16
  /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27
  /nix/store/8n7v99ii7cg694pz0cfch422zvkdqfv4-gnumake-4.2.1
  /nix/store/a1vxq1l1q4jk2hkx3ldg6h01fh22nihn-ncurses-6.1-20190112
  /nix/store/cinw572b38aln37glr0zb8lxwrgaffl4-bash-4.4-p23
  /nix/store/d4n93jn9fdq8fkmkm1q8f32lfagvibjk-gcc-7.4.0
  /nix/store/d9s1kq1bnwqgxwcvv4zrc36ysnxg8gv7-coreutils-8.30
  /nix/store/f2ax0p1cjyw8cq88yk3kiq1vqjh4bpjm-patch-2.7.6
  /nix/store/f5wl80zkrd3fc1jxsljmnpn7y02lz6v1-glibc-2.27-bin
  /nix/store/f7k6wdgzzn09k30wfc7arhwhqjf5ma6a-readline-7.0p5
  /nix/store/ghzg4kg0sjif58smj2lfm2bdvjwim85y-gcc-wrapper-7.4.0
  /nix/store/hlnxw4k6931bachvg5sv0cyaissimswb-gcc-7.4.0-lib
  /nix/store/hql9ki8x230src24ljb9fad7rxxpzal0-attr-2.4.48
  /nix/store/iiymx8j7nlar3gc23lfkcscvr61fng8s-zlib-1.2.11
  /nix/store/ix2q5vnynzfgsi4nqj8pmcbx2kf78wrm-gzip-1.10
  /nix/store/krhqmaqal0gklh15rs2bwrqzz8mg9lrn-findutils-4.6.0
  /nix/store/m57p0m5m6fz7j58g1bamsrxq5prsgpl2-bash-interactive-4.4-p23-dev
  /nix/store/mibh7947iwgf5yiqc7njn0igfzlxzdf1-xz-5.2.4
  /nix/store/mwx2860fvs3fq5dyzassvbkrkys63qsf-expand-response-params
  /nix/store/p3hr94l5smmx1qxlglw6hya9d1prd4jf-ed-1.15
  /nix/store/pmzw4y4465zmq0dc8r4xbwyrqsaj4s70-diffutils-3.7
  /nix/store/q6821d46lc5514rfh537c20f9ay2760l-bzip2-1.0.6.0.1-bin
  /nix/store/rbpyfy6413aqpik9aj6p3a2syd1mda68-binutils-wrapper-2.31.1
  /nix/store/s7p0wfbdfbq307zgifhnh6w8sfjvy64y-patchelf-0.9
  /nix/store/s83xl21h4qiz5yxlhr0n2bm1dl574mhw-acl-2.2.53
  /nix/store/sr4253np2gz2bpha4gn8gqlmiw604155-glibc-2.27-dev
  /nix/store/vvwggp5mni64yavsfqmbwvm1mn692ssn-gawk-4.2.1
  /nix/store/wlivxbc4ny21gzq9a561bwx9b4p08m4m-bzip2-1.0.6.0.1
  /nix/store/wmxqm38g1y1y7sd7s9vg7an3klffaiyz-gnutar-1.31
  /nix/store/wnjv27b3j6jfdl0968xpcymlc7chpqil-gnugrep-3.3
  /nix/store/x1khw8x0465xhkv6w31af75syyyxc65j-gnused-4.7
  /nix/store/z5zcb4k6nzl5fmc6rk8m10ysklz9c29b-stdenv-linux
  /nix/store/z6bhhh3km8dip4g2n9nkdb8hrz5zmhg5-firmware-linux-nonfree-2019-03-12
  /nix/store/zl47r98ihllg3694p4f26170v285g7ba-bash-interactive-4.4-p23
  /nix/store/zrazw25gy012ip7vcqddw6lcc393qvcg-xz-5.2.4-bin
copying path '/nix/store/z6bhhh3km8dip4g2n9nkdb8hrz5zmhg5-firmware-linux-nonfree-2019-03-12' from 'https://cache.nixos.org'...
error: out of memory

I tried adding 1GB swap via loopback (I’m on btrfs), and that just logs me out and hangs the system.

I’m assuming that Nix first loads the entire derivation in memory.

Any ideas for workarounds?

3 Likes

I have the same problem on both OVH, hetzner and other smaller VPSs.
I eventually these days look for VPSs with at least 2GB or more, and enough disk to hold the /nix/store (garbage collection is a necessary evil).
What I do is to a swapfile like you’ve tried, but I rely on ext4 (burnt myself too many times).
Are you able to attach a second disk and initialize it as swap just for the upgrade?

Hmmm. I guess the solution would be to copy the files manually and then register them in the store.

So now to figure out how to get the registrations from the store db :slight_smile:

Looks like it was fixed around Nix 2.1. I was able to get the problematic path installed by, on the target host, first running nixos-rebuild dry-activate which downloaded the most recent nix, and then using that nix to download the path:

/nix/store/1ns1zln608r5kma87rvw5kj76sckzdvn-nix-2.2.2/bin/nix-shell -p firmwareLinuxNonfree
these paths will be fetched (77.31 MiB download, 430.12 MiB unpacked):
  /nix/store/z6bhhh3km8dip4g2n9nkdb8hrz5zmhg5-firmware-linux-nonfree-2019-03-12
copying path '/nix/store/z6bhhh3km8dip4g2n9nkdb8hrz5zmhg5-firmware-linux-nonfree-2019-03-12' from 'https://cache.nixos.org'...

after that, my nixops deploy completed :slight_smile: