Speed up nix copy?

bartsch · November 3, 2021, 3:42pm

Hi, is it possible to speed up a `nix copy --to file://… $store_path?
An initial copy of a system derivation takes ~15 minutes here. While doing so, the nix (v2.4) process just uses 1 core.

Is it possible to make nix use all cores for compression? I’ve tried the options core and max-jobs in ~/.config/nix/nix.conf. But, it looks like they behave as the documentation states and effect builds only and a copy/compression doesn’t count.
Is it possible to change the compression or its level? Haven’t found any option for this.

Thanks, Daniel

misuzu · November 3, 2021, 4:20pm

Try this:

nix copy --to 'file:///tmp/cache/?compression=none&secret-key=/tmp/cache.pem' $store_path

bartsch · November 4, 2021, 11:13am

Just opened the nix code and saw that there will be an additional “parallel-compression” and “compression-level” option in the next (?) release.

github.com

NixOS/nix/blob/c5fd0b46ae56447d9d0a2028d464a5bbf753bf55/src/libstore/binary-cache-store.hh

#pragma once

#include "crypto.hh"
#include "store-api.hh"

#include "pool.hh"

#include <atomic>

namespace nix {

struct NarInfo;

struct BinaryCacheStoreConfig : virtual StoreConfig
{
    using StoreConfig::StoreConfig;

    const Setting<std::string> compression{(StoreConfig*) this, "xz", "compression", "NAR compression method ('xz', 'bzip2', 'gzip', 'zstd', or 'none')"};
    const Setting<bool> writeNARListing{(StoreConfig*) this, false, "write-nar-listing", "whether to write a JSON file listing the files in each NAR"};
    const Setting<bool> writeDebugInfo{(StoreConfig*) this, false, "index-debug-info", "whether to index DWARF debug info files by build ID"};

This file has been truncated. show original

https://github.com/NixOS/nix/pull/2276

tomberek · November 8, 2021, 8:33pm

It’s in! Add compression level for NARs by tomberek · Pull Request #5372 · NixOS/nix · GitHub

edolstra · November 10, 2021, 8:54pm

nix copy was previously able to copy paths in parallel but that’s currently disabled (https://github.com/NixOS/nix/blob/52a3b2ee6367306a3f17fe9a2d0e2e4096852e05/src/libstore/store-api.cc#L940).

Mic92 · November 11, 2021, 6:02am

Why was it disabled? It’s not obvious to me from skimming over the commit message: Low-latency closure copy · NixOS/nix@fe1f34f · GitHub