Show closure size of garbage collector roots

StillerHarpo · June 14, 2023, 8:17pm

Is there some tool or command to list the summarized closure size on all garbage collector roots?

StillerHarpo · June 14, 2023, 8:36pm

With the help of [1] I could come up with something myself:

for p in /nix/var/nix/gcroots/per-user/florian/*
do
   echo -n $p" ⇒ "
   nix-store -q --requisites $p | sort -uf | xargs du -ch | tail -1
done

[1] nixos - How to get the size of a Nix derivation? - Unix & Linux Stack Exchange

sternenseemann · June 14, 2023, 9:20pm

We can also use the already built in functionality of nix path-info -S to get the closure size of a store path. Additionally we can use GNU find to get the all gcroots (not just for your user). Bonus formatting of the resulting size with GNU numfmt.

find /nix/var/nix/gcroots/ -type l -readable \
  | xargs nix path-info -S \
  | awk '{ sum += $2; }; END { print sum }' \
  | numfmt --to=iec --suffix=B --format="%.2f"

uep · June 14, 2023, 9:51pm

This summation to a single total is counting a lot of stuff multiple times, as there will be quite a bit shared between system derivations, and probably is also hiding the information originally wanted (which gc roots should I delete to try and clean up).

~~Also minor formatting nit, the final closing ` should not be there (it is presumably a stray markdown formatting quote)~~

sternenseemann · June 14, 2023, 11:00pm

Right, we’d need to plug nix-store --requisites and sort -u in between and then use just nix path-info -s to ignore the closure size.

find /nix/var/nix/gcroots/ -type l -readable \
  | xargs nix-store -q --requisites \
  | sort -u \
  | xargs nix path-info -s \
  | awk '{ sum += $2; }; END { print sum }' \
  | numfmt --to=iec --suffix=B --format="%.2f"

StillerHarpo · June 15, 2023, 1:49pm

Thanks @sternenseemann
Inspired from your command I came up with this script, which shows all gc roots and their size (which are most likely overlapping.

for link in $(find /nix/var/nix/gcroots/ -type l -readable)
do
  res="${res}
$(readlink "${link}") $(nix path-info -Sh "${link}")"
done
echo "$res" | sort -h -k3,3

``

symphorien · June 15, 2023, 9:39pm

nix-du outputs a graph that shows the size attached to each gc root and the amount of space shared between them.

picnoir · November 21, 2024, 8:34pm

Just in case anybody stumble upon this from a search engine like I did, here’s a variant of StillerHarpo’s solution:

#!/usr/bin/env bash

# Collecting GC roots paths
declare -a roots
while IFS= read -r -d '' link
do
        roots+=( "$( readlink -f "$link")" )
done < <(find /nix/var/nix/gcroots/ -type l -readable -print0)

# Retreiving GC roots sizes, order by size
tmp="$(mktemp)"
trap 'rm $tmp' EXIT
nix path-info --closure-size --json "${roots[@]}" | jq 'sort_by(.closureSize) | map({ path: .path, size: .closureSize })' > "$tmp"

# Pretty print result
jq -r '.[] | .path + ": " + (.size | tostring)' "$tmp" | while read -r line; do
    path=$(echo "$line" | cut -d: -f1)
    bytes=$(echo "$line" | cut -d: -f2 | tr -d ' ')
    
    # Pretty-print bytes to a more appropriate human format
    readable_size=$(numfmt --to=iec-i --suffix=B --format="%.1f" <<< "$bytes")
    
    echo "$path: $readable_size"
done

uep · November 22, 2024, 1:49am

and here’s a nushell version (edit: tiny update to work with current nu as it evolves)

def gcdu [
    --processes (-p)    # also show roots from running processes
]: nothing -> table {
  let roots = ^nix-store --gc --print-roots 
    | parse "{root} -> {closure}"
    | where root !~ '{censored}|^/proc' or $processes
    | group-by --to-table closure 
    | rename path roots
    | update roots { get root | sort | str join "\n" }
  
  let pathinfo = ^nix path-info --closure-size --json ...($roots | get path) 
    | from json
    | select closureSize narSize path
    | into filesize closureSize narSize
  
  $roots | join $pathinfo path
    | update path { path basename }
    | rename -c { path: "store path" }
    | sort-by -r closureSize
}

in addition to the shell/syntax differences, this version uses a different method to find the roots in a way that shows the thing(s) you want to delete in order to let the closure get gc’d, and groups them together.

sample output, in md format (slightly modified because discourse’s md tables don’t do inner newlines)

❯ gcdu | take 10 | to md -p

store path	roots	closureSize	narSize
qvz2lfr2513g26m41h8jrfy9p9vwz62i-nixos-system-oenone-24.11.20241111.dc460ec	/nix/var/nix/profiles/system-557-link	31.9 GiB	157.5 KiB
lbm6m1q8jg9lc02fwjs79x6jx64a1k5z-nixos-system-oenone-24.11.20241109.76612b1	/nix/var/nix/profiles/system-556-link	31.9 GiB	157.5 KiB
94dh5jxqfbknbpyvx3dmkswvf97n9pgf-nixos-system-oenone-24.11.20241109.76612b1	/nix/var/nix/profiles/system-555-link	31.9 GiB	157.5 KiB
611ma22g3azfzify0yc2sxyshwr4p4kv-nixos-system-oenone-25.05.20241115.5e4fbfb	/nix/var/nix/profiles/system-559-link /run/booted-system /run/current-system	29.4 GiB	157.5 KiB
aq3pnl3xfcg2h68lx7iwv7knj7992mwc-nixos-system-oenone-25.05.20241115.5e4fbfb	/nix/var/nix/profiles/system-558-link	29.3 GiB	157.5 KiB
ifrfgpyilgvigl2iary8dsxswn9313iq-nix-shell-env	/home/dan/work/resl/qualysdata/.direnv/flake-profile-a5d5b61aa8a61b7d9d765e1daf971a9a578f1cfa	2.8 GiB	72.8 KiB
2zwmk2wsxm534qyb3597p12y3bs1wim2-nix-shell-env	/data/foss/rust/serde-norway/.direnv/flake-profile-a5d5b61aa8a61b7d9d765e1daf971a9a578f1cfa	1.6 GiB	84.5 KiB
vysk2fzd96zyv1qdsd9j9x4fwdi93jpk-home-manager-generation	/home/dan/.local/state/nix/profiles/home-manager-248-link	1021.8 MiB	14.5 KiB
fjbyx7bkzfahpiklab7m1k6qzngq7d2q-home-manager-generation	/home/dan/.local/state/nix/profiles/home-manager-247-link	1017.9 MiB	14.5 KiB
z4cjvbn055bqsdgs3sjj0whs6m8gmasx-home-manager-generation	/home/dan/.local/state/nix/profiles/home-manager-246-link	1017.9 MiB	14.5 KiB

nh2 · December 12, 2024, 4:01am

I have further adjusted this to work with NixOS 24.11 (which seems to have changed the nix path-info JSON output format so the jq sort_by() didn’t work anymore.

Also padding-aligned the output, and added --extra-experimental-features nix-command to not assume a global system setting that sets this.

#!/usr/bin/env bash

# From https://discourse.nixos.org/t/show-closure-size-of-garbage-collector-roots/29118

# Collecting GC roots paths
declare -a roots
while IFS= read -r -d '' link
do
        roots+=( "$( readlink -f "$link")" )
done < <(find /nix/var/nix/gcroots/ -type l -readable -print0)

# Retreiving GC roots sizes, order by size
tmp="$(mktemp)"
trap 'rm $tmp' EXIT
nix --extra-experimental-features nix-command path-info --closure-size --json "${roots[@]}" | jq 'to_entries | sort_by(.value.closureSize) | map({ path: .key, size: .value.closureSize })' > "$tmp"

# Pretty print result
jq -r '.[] | .path + ": " + (.size | tostring)' "$tmp" | while read -r line; do
    path=$(echo "$line" | cut -d: -f1)
    bytes=$(echo "$line" | cut -d: -f2 | tr -d ' ')
    
    # Pretty-print bytes to a more appropriate human format
    readable_size=$(numfmt --to=iec-i --suffix=B --format="%8.1f" <<< "$bytes")
    
    echo "$readable_size   $path"
done