Hi,
I have no idea how it happened, but much stuff on my server was unusably slow. systemd-udevd was constantly timing out and restarting. strace revealed that /proc/self/mountinfo had 48199 entries which took too long to read. Same for /proc/mounts. Most entries are of the sort: a/zfs/dataset@autosnap_2026-01-31_19:00:04_daily /mountpoint/to/a/zfs/dataset/.zfs/snapshot/autosnap_2026-01-31_19:00:04_daily zfs ro,relatime,xattr,posixacl,casesensitive 0 0
As much as I can tell snapshots shouldn’t end up in /proc/mounts at all. | sort | uniq -c shows that the same snapshots are mounted always 40/80/120 times, with 2 snapshots being mounted 20760 or 20720 times.
This happened for sanoid-snapshots as well as for syncoid-snapshots. All for datasets I created just a few days/weeks ago, not for older ones.
Any idea how this happened? Or is still happening I guess …
I am right now unmounting them in a loop, but the it seems that the kernel-time of the umount command takes ~15s each. I am hesitant of a reboot right now as I won’t be able to get physically to the server for the next few days in case the reboot doesn’t work as expected.
Thanks for any ideas - either on the root cause or on ways to mitigate the issue.
Best,
Thomas
I dont have the link, but we’ve encountered something similar at $work. Its a known bug to which the workaround isnnot to use snapdirs sadly… check the snapdir property on your datasets
In the meantime, some other info I found out. First I executed this command:
time umount /path/to/mountpoint/.zfs/snapshot/autosnap_2026-02-08_23:03:03_hourly
real 0m16,673s
user 0m0,146s
sys 0m15,869s
Since it was sysI thought the umount-syscall was the bottleneck.
But letting ChatGPT write me a quick umount rust-script which calls libc::umount directly lead to a 450x performence increase
I let it just umount a mountpoint in a loop as long as the syscall doesn’t return an error. (requires libc-crate, so probably a new cargo project and cargo add libc)
use std::ffi::CString;
use std::os::raw::c_int;
use std::time::Instant;
use std::env;
fn main() {
let path = env::args().nth(1).unwrap();
// Prepare C string
let c_path = CString::new(path.clone()).expect("CString::new failed");
println!("loop-unmounting '{}'", path);
while {
// Time the syscall
let start = Instant::now();
// Syscall: int umount(const char *target);
let ret = unsafe { libc::umount(c_path.as_ptr() as *const i8) == 0 };
let elapsed = start.elapsed();
if ret {
println!("umount succeeded for '{}'", path);
} else {
// errno available via libc::errno location
let err = std::io::Error::last_os_error();
println!("umount failed for '{}': {}", path, err);
}
println!("syscall elapsed: {}.{:06} seconds",
elapsed.as_secs(),
elapsed.subsec_micros());
ret //condition here - abuse as do-while
}{}
}
Then I did a bash-for-loop over all datasets that need umounting (the list wasn’t very long) and called my umount-script instead.
Otherwise, in any language iterating over /proc/mounts, deciding for each entry if it needs unmounting (e.g. path contains .zfs/snapshot) and then unmounting once via libc would also have done the trick. Just don’t use the cli-umount-command.
I seem to recall having issues with snapshots being mounted every time the snapdir was accessed. In other words it would just keep overmounting forever.
Another Update: Even though I snapdir=disabled root (and the rest inherits that), I can still access .zfs/snapshot and mounts are still being created .
My inperfect workaroung using my own unmount-script which is orders of magnitudes faster than umount since it just directly calls the libc syscall:
.