Build Nix Package with 10GB+ source tree

Development
1

Hello there,

I want to build a package with 10GB+ source tree, nix-build runs out of memory (I have 32BG DDR), any suggestion.

What I was used:

{ pkgs ? import <pkgs>{} }:

pkgs.stdenv.mkDerivation {
    name "my_package";
    
    stc = ./source; # more than 10GB source code
    
    buildPhase = ''
        # do something...
    ''
    
    installPhase = ''
        mkdir $out
        # do some other things.
    '';
}
2

Try with nixUnstable.

3

what has a 10G source tree, are you compiling the ‘internet’ ? :slight_smile:

my guess it’s java. LOL.

1 Like
4

also, if you maybe able to get slightly more runtime memory by you a zstd swap.

  zramSwap = {
    enable = true;
    algorithm = "zstd";
  };
5

I remember when setting up ZFS on NixOS it was suggested not to use ZFS for swap due to the possibility of deadlocks.

What are the actual risks of it? Is it an edge-case scenario or would deadlocks be common? Is there work being done to address it upstream?

Due to the warnings and because I use both ZFS for larger pools of data as well as ZFS on root, I have never enabled the swap (which would be nice to have).

6

i never put swap on anything but raw hdd partitions , never of ‘fancy’ storage like XFS/ZFS etc etc. One day we will have a 1TB of ram, so will never need swap again.

7

I’ve had similar experiences with robotnix to build Android which has about 60 GB source tree. Make sure that /tmp is not a tmpfs which is mounted into RAM (on NixOS that is boot.tmpOnTmpfs = false). Alternatively you could do TMPDIR=/somewhere/not/in/ram nix-build ...

8

60 gig of source code, i don’t know wether to be happy or sad about that…

don’t that say you get 10 serious bugs per 1000 lines of code written?