Overriding memory allocation for build VM for OCIImage

Help
1

I’m trying to build an Oracle Cloud image as one of the outputs of my flake. I have an Oracle ARM VM, and my local machine is x86-based. I configured the output thus:

        oci-base = nixpkgs.lib.nixosSystem {
          specialArgs = {
            inherit inputs;
          };
          system = "aarch64-linux";
          modules = [
            "${nixpkgs}/nixos/modules/virtualisation/oci-image.nix"
            ./configuration-oracle.nix
          ];
        };

and added

  boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

When I try to build the output, I get an out-of-memory error coming from the build VM.

 ➜  nix build .#nixosConfigurations.oci-base.config.system.build.OCIImage
...
       > error while reading directory /build/root/nix/store: Cannot allocate memory
...

along with many similar errors.

I see that under the hood, the OCIImage attribute is defined by reference to make-disk-image.nix, which does have a parameter memSize. But I cannot figure out how to override OCIImage appropriately to set memSize. I have a lot of experience as a Nix user but not a lot with the Nix language. Guidance much appreciated.

1 Like
2

I have encountered something similar. I even tried setting memSize to something much larger and it did not help, so I suspect something else is at play preventing these x86_64-linux→aarch64-linux cross builds. Here is the flake I used:

{
  description = "Minimal NixOS qcow2 image";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }:
    let
      system = "aarch64-linux";
      nixos = self.nixosConfigurations.default;
    in
    {
      nixosConfigurations.default = nixpkgs.lib.nixosSystem {
        inherit system;
        modules = [ ./configuration.nix ];
      };

      packages.${system}.default = import "${nixpkgs}/nixos/lib/make-disk-image.nix" {
        inherit (nixos) config pkgs;
        lib = nixpkgs.lib;
        inherit (nixos.config.virtualisation) diskSize;
        format = "qcow2";
        partitionTableType = "efi";
        memSize = 8192; # MiB of RAM for the build VM (default is 1024)
      };
    };
}

This is part of the log from the failed build:

copying staging root to image...
[    0.000000] Linux version 6.6.0 (nixbld@localhost) (gcc (GCC) 15.2.0, GNU ld (GNU Binutils) 2.44) #1 Fri Jan 16 21:37:51 UTC 2026
[    0.000000] memblock address range: 0x7fffd5c00000 - 0x7fffdc000000
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x00007fffd5c00000-0x00007fffdbffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00007fffd5c00000-0x00007fffdbffffff]
[    0.000000] Initmem setup node 0 [mem 0x00007fffd5c00000-0x00007fffdbffffff]
[    0.000000] Kernel command line:  mem=100M virtio_mmio.device=352@0x1000000:1
[    0.000000] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.000000] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 25250
[    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[    0.000000] Memory: 100768K/102400K available (6905K kernel code, 1663K rwdata, 1557K rodata, 139K init, 337K bss, 1632K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] trace event string verifier disabled
[    0.000000] NR_IRQS: 4096
[    0.000000] lkl: irqs initialized
[    0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000022] lkl: time and timers initialized (irq2)
[    0.000780] pid_max: default: 4096 minimum: 301
[    0.003139] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.003178] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.075203] printk: console [lkl_console0] enabled
[    0.076985] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.079475] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.085502] lkl_pci: probe of lkl_pci failed with error -1
[    0.210197] raid6: skipped pq benchmark and selected int64x8
[    0.210321] raid6: using intx1 recovery algorithm
[    0.223308] vgaarb: loaded
[    0.224253] clocksource: Switched to clocksource lkl
[    0.231317] NET: Registered PF_INET protocol family
[    0.232960] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.237611] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.238048] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.238165] TCP established hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    0.238370] TCP bind hash table entries: 1024 (order: 2, 16384 bytes, linear)
[    0.238552] TCP: Hash tables configured (established 1024 bind 1024)
[    0.239868] UDP hash table entries: 128 (order: 0, 4096 bytes, linear)
[    0.240181] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes, linear)
[    0.241579] PCI: CLS 0 bytes, default 32
[    0.245377] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x100015f, IRQ 1.
[    0.271972] workingset: timestamp_bits=62 max_order=15 bucket_order=0
[    0.274533] SGI XFS with ACLs, security attributes, no debug enabled
[    0.279721] xor: automatically using best checksumming function   8regs
[    0.280045] io scheduler mq-deadline registered
[    0.280161] io scheduler kyber registered
[    0.317852] virtio_blk virtio0: 1/0/0 default/read/poll queues
[    0.323520] virtio_blk virtio0: [vda] 16777216 512-byte logical blocks (8.59 GB/8.00 GiB)
[    0.341977]  vda: vda1 vda2
[    0.354603] NET: Registered PF_INET6 protocol family
[    0.363353] Segment Routing with IPv6
[    0.363571] In-situ OAM (IOAM) with IPv6
[    0.364043] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    0.395284] Btrfs loaded, zoned=no, fsverity=no
[    0.400095] Warning: unable to open an initial console.
[    0.400579] This architecture does not have kernel memory protection.
[    0.400730] Run /init as init process
[    0.504159] EXT4-fs (vda2): mounted filesystem ecd795c1-b1c6-4763-8867-556d73376965 r/w with ordered data mode. Quota mode: disabled.
error while reading directory /build/root/etc: Cannot allocate memory
3

After more experimentation, my experience is the same as yours. I vendored the module from nixpkgs into my flake so I could set memSize, only to find it made no difference.
I don’t have the expertise with qemu to figure out what’s really going on.

4

Hi! Sounds like you’ve been folllowing Erik Parawell’s guide. I’m here for the same reason, so I’ll first talk about this specific issue and then I’ll digress.

That stack trace stems from cptofs, and the memory limit of 100m seen in the logs is hardcoded. @jamesla has made an overlay patch for this which you can grab here. From there, I had varying success building the image (attempting to do so on my x86 homelab and WSL concurrently, because it was taking ages to compile). Some times it would work, other times it would not, and I couldn’t figure out a cause. The patch definitely worked though - I inspected a stack trace with the memory set to 2048m. Maybe I should’ve set it higher, or maybe this log is just a red herring. I didn’t change memSize in make-disk-image while attempting this since you said it had no effect (but maybe the combination would have).

That being said, I did manage to build the image a couple times. After deploying, I couldn’t SSH to my box for unknown reasons. I didn’t look in the serial console because I was unfamiliar with it (just wait for it to initialize, press enter and see if you get a shell).

Now, here are my two cents: while this method is really awesome for science, I don’t think it’s the best for reproducibility. First, it’s flaky (build issue + Erik mentions the image compatibility registration is poorly documented (in true Oracle fashion), plus there’s a lot more configuration you have to maintain).

There are two other approaches largely used for this sort of thing:

  • nixos-anywhere
    • Couldn’t get it to work with OCI - my box would kernel panic on boot.
    • Tried several nixos and disko configurations
    • Tried both Ubuntu and Oracle Linux 9 with no success, though Anarion reported success with the latter (but doesn’t mention which version)
  • nixos-infect
    • This is the only approach I recommend. It’s the only one which worked for me and is dead simple (just one bash command on a fresh box!)

Hope this helps - I’d be happy to hear if you manage to pull it off!