Random guess, but I’ve found btrfs has particularly pathological performance in VMs with lots of small reads / writes, for reasons I’ve never got to the bottom of. I tend to default to xfs given the choice, as it handles nix’s high demand for inodes better than ext4 does on relatively small volumes (~10GB, not unusual for a VM rootfs.)
2 Likes
I am a simple man and almost always use EXT4. I’ve only used BTRFS once and that was on a bare metal install. My VMs for NixOS always use EXT4 with LUKS disk encryption.
Thank you for your insight, TLATER!
However, I doubt it is the VM itself or the filesystem causing this. I use EXT4 with disk encryption inside the VMs. I decided to disable disk encryption and the copy times did not improve. The VM uses a qcow2 disk with the virtio driver. I tried running it both on a HDD and an SSD and also saw no improvement when 25.11 was used.
The only thing that makes a difference is whether I use 25.05 or 25.11 as the base.
There is nothing else nix is doing while those messages are shown. It’s almost certainly related to the machine you’re running this on. The same download/extraction does take a bit of time on my end, but it’s still less than a minute, and I’m not using a supercomputer.
I’d guess the extraction doesn’t take any time on 25.05 because it’s already been done before and therefore cached, while 25.11 still receives updates, so you’re getting commits that haven’t been cached and therefore need to be downloaded. That might also be why 25.05 still takes excessively long to evaluate (albeit less so than 25.11); the metadata download still takes a while. Maybe rather than your filesystem it’s extremely slow networking, or perhaps it’s specifically decompression. Hard to say without profiling.
That said, even if this is not your VM but a nix bug, without your exact VM it’ll be hard for anyone else to debug this. Pull out your favorite profiler to see what’s taking so long. It’s likely not a hard debug since the codepath for this isn’t the most complex. Whatever you find will tell you what to fix.
1 Like
Thank you once again TLATER for your insight! I will attach my VM’s XML configuration. I hope it will prove to be useful. As for profiling my VM, what tools should I use and what should I look out for? Should I use simple programs such as top, or one of its derivatives, to look for high CPU or RAM usage or low network utilization? During the long extraction times I would look at the performance tab in virt-manager. RAM usage would never exceed 1GiB and CPU usage would always hover around 27% with 4 vCPUs allocated. Network and disk I/O are both essentially non-existent at just a couple of KBs every few seconds
<domain type="kvm">
<name>NixOSLegacy</name>
<uuid>4cae2f09-fa20-4200-a496-be9f5d00b3b1</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://nixos.org/nixos/unknown"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">6291456</memory>
<currentMemory unit="KiB">6291456</currentMemory>
<memoryBacking>
<source type="memfd"/>
<access mode="shared"/>
</memoryBacking>
<vcpu placement="static">4</vcpu>
<os>
<type arch="x86_64" machine="pc-q35-9.2">hvm</type>
<boot dev="hd"/>
<bootmenu enable="yes"/>
</os>
<features>
<acpi/>
<apic/>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough" check="none" migratable="on"/>
<clock offset="utc">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/run/libvirt/nix-emulators/qemu-system-x86_64</emulator>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/home/aly/aly/atmos-v0.1.6-dev-nofmw-06-03-2026-25.11.iso"/>
<target dev="sda" bus="sata"/>
<readonly/>
<address type="drive" controller="0" bus="0" target="0" unit="0"/>
</disk>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" cache="none" io="io_uring" discard="unmap"/>
<source file="/home/aly/.local/mnt/HDD-Partition/Libvirt-Drives/NixOSLegacy.qcow2"/>
<target dev="sdb" bus="virtio"/>
<address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="3" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x18"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="10" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="10" port="0x19"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
</controller>
<controller type="pci" index="11" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="11" port="0x1a"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
</controller>
<controller type="pci" index="12" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="12" port="0x1b"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
</controller>
<controller type="pci" index="13" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="13" port="0x1c"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
</controller>
<controller type="pci" index="14" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="14" port="0x1d"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</controller>
<controller type="scsi" index="0" model="virtio-scsi">
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</controller>
<filesystem type="mount" accessmode="passthrough">
<driver type="virtiofs"/>
<binary path="/run/current-system/sw/bin/virtiofsd"/>
<source dir="/home/aly/Shared-Folder"/>
<target dir="/shared"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</filesystem>
<interface type="network">
<mac address="52:54:00:70:ff:e4"/>
<source network="default"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</interface>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<channel type="unix">
<target type="virtio" name="org.qemu.guest_agent.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
<address type="virtio-serial" controller="0" bus="0" port="2"/>
</channel>
<input type="tablet" bus="usb">
<address type="usb" bus="0" port="1"/>
</input>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<graphics type="spice">
<listen type="none"/>
<image compression="off"/>
<gl enable="yes" rendernode="/dev/dri/by-path/pci-0000:05:00.0-render"/>
</graphics>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<audio id="1" type="spice"/>
<video>
<model type="virtio" heads="1" primary="yes">
<acceleration accel3d="yes"/>
</model>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
</video>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="2"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="3"/>
</redirdev>
<watchdog model="itco" action="reset"/>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</memballoon>
<rng model="virtio">
<backend model="random">/dev/urandom</backend>
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</rng>
</devices>
</domain>