I’m a little fuzzy on the details, since it’s been over a year since I looked into it. But the basics are that Ventoy works by basically re-implementing the ISO’s bootloader, and then injecting its extra files into the initrd of the system. These files include a small embedded OS’s worth of binaries (these are the prebuilt ones that are checked into the source repo), and some udev rules. Those udev rules trigger when the USB drive is found and synthesize a virtual block device from the underlying ISO file. You’d think it would use a loopback block device for that but no, that would require mounting the file system and that would be too chaotic, apparently. So instead they parse the FAT table to identify which blocks on disk contain the ISO and create a dm-linear device that maps to those blocks. And that’s how initrd is tricked into seeing its ISO as a block device.
The reason this is unreliable is largely because of the udev rules. This is the part I’m fuzzy on. The way it injects the udev rules was something completely insane, and in a file system containing a Nix store, the odds of it injecting them into the right place were not 100%. I think this was exacerbated in my unfinished PR switching the ISO to systemd initrd, because systemd initrd doesn’t shove all the binaries into one directory, so there’s a lot more directories for Ventoy to get tripped up by.