Nixpkgs-review running out of memory

Development
21

Thanks @vcunat , I am now running nixpkgs-review rev HEAD on AWS EC2 with 64GB of RAM. Will see how it goes.

22

Are you able to paste here your whole zramSwap config here ? I never used it. Thanks.

23

I use just this single line in my NixOS config. The defaults are good I think. It creates a swap device sized as 50% of RAM, etc.

EDIT: example status

$ zramctl
NAME       ALGORITHM DISKSIZE  DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd         15.5G  3.1G 556.9M 601.2M       8 [SWAP]
24

Thank you very much.

25

nixpkgs-review rev HEAD on AWS crashed on following:

nixpkgs-review rev HEAD
$ git -c fetch.prune=false fetch --no-tags --force https://github.com/NixOS/nixpkgs master:refs/nixpkgs-review/0
remote: Enumerating objects: 3292250, done.
remote: Counting objects: 100% (3292249/3292249), done.
remote: Compressing objects: 100% (791569/791569), done.
remote: Total 3266988 (delta 2261783), reused 3254126 (delta 2249775), pack-reused 0
Receiving objects: 100% (3266988/3266988), 2.61 GiB | 19.07 MiB/s, done.
Resolving deltas: 100% (2261783/2261783), completed with 6665 local objects.
From https://github.com/NixOS/nixpkgs
 * [new branch]              master     -> refs/nixpkgs-review/0
$ git worktree add /root/.cache/nixpkgs-review/rev-d6952b2c6d25842a7c4115af01041733038e02ee/nixpkgs 1c9ffcf70786f0966982ce0fc76ec05df2e1dec2
Preparing worktree (detached HEAD 1c9ffcf7078)
Updating files: 100% (32844/32844), done.
HEAD is now at 1c9ffcf7078 Merge #204776: nodePackages: remove reference to deltachat-desktop in overrides
$ nix-env --option system x86_64-linux -f /root/.cache/nixpkgs-review/rev-d6952b2c6d25842a7c4115af01041733038e02ee/nixpkgs -qaP --xml --out-path --show-trace
$ git merge --no-commit --no-ff d6952b2c6d25842a7c4115af01041733038e02ee
fatal: refusing to merge unrelated histories
$ git worktree prune
Traceback (most recent call last):
  File "/nix/store/hckpvqslb0iqxjxll2jl4d3hk09r9i0a-nixpkgs-review-2.7.0/bin/.nixpkgs-review-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/hckpvqslb0iqxjxll2jl4d3hk09r9i0a-nixpkgs-review-2.7.0/lib/python3.10/site-packages/nixpkgs_review/__init__.py", line 10, in main
    cli.main(command, args)
  File "/nix/store/hckpvqslb0iqxjxll2jl4d3hk09r9i0a-nixpkgs-review-2.7.0/lib/python3.10/site-packages/nixpkgs_review/cli/__init__.py", line 294, in main
    return cast(str, args.func(args))
  File "/nix/store/hckpvqslb0iqxjxll2jl4d3hk09r9i0a-nixpkgs-review-2.7.0/lib/python3.10/site-packages/nixpkgs_review/cli/rev.py", line 12, in rev_command
    return review_local_revision(f"rev-{commit}", args, commit)
  File "/nix/store/hckpvqslb0iqxjxll2jl4d3hk09r9i0a-nixpkgs-review-2.7.0/lib/python3.10/site-packages/nixpkgs_review/review.py", line 474, in review_local_revision
    review.review_commit(builddir.path, args.branch, commit, staged)
  File "/nix/store/hckpvqslb0iqxjxll2jl4d3hk09r9i0a-nixpkgs-review-2.7.0/lib/python3.10/site-packages/nixpkgs_review/review.py", line 250, in review_commit
    self.start_review(self.build_commit(branch_rev, reviewed_commit, staged), path)
  File "/nix/store/hckpvqslb0iqxjxll2jl4d3hk09r9i0a-nixpkgs-review-2.7.0/lib/python3.10/site-packages/nixpkgs_review/review.py", line 150, in build_commit
    self.git_merge(reviewed_commit)
  File "/nix/store/hckpvqslb0iqxjxll2jl4d3hk09r9i0a-nixpkgs-review-2.7.0/lib/python3.10/site-packages/nixpkgs_review/review.py", line 118, in git_merge
    sh(["git", "merge", "--no-commit", "--no-ff", commit], cwd=self.worktree_dir())
  File "/nix/store/hckpvqslb0iqxjxll2jl4d3hk09r9i0a-nixpkgs-review-2.7.0/lib/python3.10/site-packages/nixpkgs_review/utils.py", line 31, in sh
    return subprocess.run(command, cwd=cwd, check=check, text=True)
  File "/nix/store/zdba9frlxj2ba8ca095win3nphsiiqhb-python3-3.10.8/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'merge', '--no-commit', '--no-ff', 'd6952b2c6d25842a7c4115af01041733038e02ee']' returned non-zero exit status 128.
26

nixpkgs-review tries by default to merge the PR on master, sometimes it doesn’t work anymore and nixpkgs-review aborts.

On the memory side, I often encounter issues because my system only has 8 GB of memory, by adding 6 GB of swap it’s fine most of the time with no application running… This is really concerning though, because it’s getting worse and worse with nixpkgs getting bigger.

27

Thank you @Solene , it think it crashed because I cloned only my PR branch on AWS machine. I’ll try to add master as well.

28

Crashed on the same error message with master branch included.

git branch
  master
* proj-9.1.1

subprocess.CalledProcessError: Command '['git', 'merge', '--no-commit', '--no-ff', 'd6952b2c6d25842a7c4115af01041733038e02ee']' returned non-zero exit status 128.

This tool really doesn’t like me.

29

Try to fill an issue on GitHub - Mic92/nixpkgs-review: Review pull-requests on https://github.com/NixOS/nixpkgs

30

I think the problem is that I used GitHub’s “Sync fork” button which did some mess. So the problem might be on my side.

31

“Sync fork” mess fixed and it is building now.

1 Like
32

Just adding my 2 cents here after also running out of memory a few (too many) times when running nixpkgs-review rev HEAD.

Found this solution on StackOverflow: use systemd-run and give it a lot of swap space. So, if one has enough storage space to spare, why not :)?

systemd-run --user --scope -pMemoryMax=<MemoryLimit> -pMemorySwapMax=<SwapMemoryLimit> nixpkgs-review rev HEAD

If the constraints do not allow for enough memory to nixpgks-review, then the process will just be terminated and no system crash — well, as long as the limits were not the max available memory and swap memory.

33

@onemoresuza , good idea, thanks !