SSH connection hangs


recently my sshd is misbehaving.

When I connect to my nixbox on LAN i only get around 40 seconds of responsive ssh session.
Later the SSH just gets frozen.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3,6Gi       615Mi       1,9Gi       4,0Mi       1,1Gi       2,8Gi
Swap:             0B          0B          0B
$ cat /proc/loadavg 
6.92 3.06 1.43 1/194 2931

Htop output looks clean – the cpu load is ~3%.

Also as of right now I cannot even rebuild my configuration…
Does aonyone have an idea how to debug this or restore the connectivity?
Should I just login without SSH, just to TTY?

I have connected to it via TTY and removed all functionality but one.
Then I have stripped all the functionality but one and kept adding stuff.

Update + collecting the garbage seemed to help.
I don’t know what caused this.