`nix-shell` is eating up all my memory and then gets killed

Help
1

Hi, I am using the obelisk framework for reflex-frp from Obsidian Systems. Running a nix-shell (to eventually run my emacs) like this …

nix-shell -A shells.ghc

results in lot of compiling of haskell packages. There is base64 which apparently has 228 dependencies (as shown in the output log). At about 199/228 my RAM is all used up and the process gets killed.

[199 of 228] Compiling GHC.Hs.Instances ( compiler/GHC/Hs/Instances.hs, dist/build/GHC/Hs/Instances.o )
/nix/store/wgap303sj9zqz63gw7nqxvf4dqz2hgai-stdenv-linux/setup: line 1302:   253 Killed                  ./Setup build
builder for '/nix/store/diih8n14fw7xbvjvqy8hppw0nz9q3jq7-base64-0.4.2.2.drv' failed with exit code 137
cannot build derivation '/nix/store/r2p2wmgq7gpjc1dqfaz08g1kfjfxp3a5-ghc-8.6.5-with-packages.drv': 1 dependencies couldn't be built
error: build of '/nix/store/2390dypknbv03039w9c2xqj2n53dr9bx-build-tools-wrapper.drv', '/nix/store/r2p2wmgq7gpjc1dqfaz08g1kfjfxp3a5-ghc-8.6.5-with-packages.drv' failed

At that point, there are two ghc processes and a nix-shell process. One of the ghc processes uses 63% of my RAM amounting to 10 GB of 16 GB installed in total.

Why does this happen?

I guess there are certain steps to go about this.

  • Activate swap partition to have more memory, effectively. But 16 GB should make swap redundant (so I was told)
  • Maybe reduce the number of build processes (how?)

The fact that the whole process consistently fails building the base64 packages makes me suspicious, too. Is there a problem related to garbage collection?

EDIT: I realize, this might have more to do with GHC compiling and linking than with Nix. I am not a 100% sure, though.

nix-info -m

 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.118, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.16`
 - channels(root): `"nixos-21.11"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
2

I was able to get down to one single ghc job (that still uses about 6 cores) by running

nix-shell -A shells.ghc -j 1

Preprocessing test suite 'tasty' for base64-0.4.2.2..
Building test suite 'tasty' for base64-0.4.2.2..
[1 of 2] Compiling Internal         ( test/Internal.hs, dist/build/tasty/tasty-tmp/Internal.o )
[2 of 2] Compiling Main             ( test/Main.hs, dist/build/tasty/tasty-tmp/Main.o )
/nix/store/wgap303sj9zqz63gw7nqxvf4dqz2hgai-stdenv-linux/setup: line 1302:   253 Killed                  ./Setup build
builder for '/nix/store/diih8n14fw7xbvjvqy8hppw0nz9q3jq7-base64-0.4.2.2.drv' failed with exit code 137
cannot build derivation '/nix/store/r2p2wmgq7gpjc1dqfaz08g1kfjfxp3a5-ghc-8.6.5-with-packages.drv': 1 dependencies couldn't be built
error: build of '/nix/store/2390dypknbv03039w9c2xqj2n53dr9bx-build-tools-wrapper.drv', '/nix/store/r2p2wmgq7gpjc1dqfaz08g1kfjfxp3a5-ghc-8.6.5-with-packages.drv' failed

This ghc job now doesn’t get anywhere with compiling base64. It makes me conclude that there is a specific memory leak in one of the dependencies, not a general high use of my RAM.

3

I was able to work around the memory leak by disabling tests for base64:

  with pkgs.haskell.lib;
  obelisk.project ./. ({ ... }: {
    android.applicationId = "systems.obsidian.obelisk.examples.minimal";
    android.displayName = "Palantype";
    ios.bundleIdentifier = "systems.obsidian.obelisk.examples.minimal";
    ios.bundleName = "Palantype";

    overrides = self: super: {
      base64 = dontCheck super.base64;
    }