Direnv taking an extremely long amount of time to copy directory contents

Help
1

I have configured a python environment with pytorch, but when I cd into the directory, the copying takes an extremely long amount of time. I am thinking it has to do with the fact that I have a data folder with a lot of data in it. I am wondering if it is trying to copy this data to the nix store or something, and if so, if I can tell direnv to exclude this folder.

I get two warning when I cd in. direnv: nix-direnv: minimum required nix-direnv version is 3.0.5 (installed: 3.0.4) which I assume comes from an update to direnv yesterday that isn’t in unstable yet. And the execution time warning: direnv: ([/nix/store/35klgzald67mkslqb9kkv01gn98zfbza-direnv-2.34.0/bin/direnv export zsh]) is taking a while to execute. Use CTRL-C to give up..

Here is the directory tree:

 ls -R projects/lfg
data  flake.lock  flake.nix  shell.nix  train.py

projects/lfg/data:
2024-06-02_12-34-51.mp4  ER1_1.mp4  ER1_4.mp4  ER1_7.mp4  ER1_10.mp4  ER1_13.mp4  ER1_16.mp4  ER1_19.mp4  ER1_22.mp4  ER1_25.mp4  ER1_28.mp4  ER1_31.mp4  ER1_34.mp4  ER1_37.mp4  ER1_40.mp4
2024-06-02_12-56-41.mp4  ER1_2.mp4  ER1_5.mp4  ER1_8.mp4  ER1_11.mp4  ER1_14.mp4  ER1_17.mp4  ER1_20.mp4  ER1_23.mp4  ER1_26.mp4  ER1_29.mp4  ER1_32.mp4  ER1_35.mp4  ER1_38.mp4  
ER1_0.mp4                ER1_3.mp4  ER1_6.mp4  ER1_9.mp4  ER1_12.mp4  ER1_15.mp4  ER1_18.mp4  ER1_21.mp4  ER1_24.mp4  ER1_27.mp4  ER1_30.mp4  ER1_33.mp4  ER1_36.mp4  ER1_39.mp4

.envrc:

if ! has nix_direnv_version || ! nix_direnv_version 3.0.5; then
    source_url "https://raw.githubusercontent.com/nix-community/nix-direnv/3.0.5/direnvrc" "sha256-RuwIS+QKFj/T9M2TFXScjBsLR6V3A17YVoEW/Q6AZ1w="
fi
use flake

flake.nix:

{
  description = "A basic flake with a shell";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { nixpkgs, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
      in
      {
        devShells.default = pkgs.mkShell {
          packages = with pkgs; [
            (python312.withPackages (ps: with ps; [
              torch
              torchvision
              av
            ]))
          ];
        };
      });
}

shell.nix:

{ pkgs ? import <nixpkgs> {
  config = {
    allowUnfree = true;
    cudaSupport = true;
  };}
}:

pkgs.mkShell {
  packages = with pkgs; [
    (python312.withPackages (ps: with ps; [
      torch
      torchvision
      av
    ]))
  ];
}
2

I think your flake is trying to compile something the takes too long to compile.

Try this as debug process:

nix shell /absolute/path/to/the/directory

It will show what is taking too long to compile.
Try updating your flake.lock or change to latest stable release
inputs.nixpkgs.url = "github:NixOS/nixpkgs/release-24.05";

3

I am pretty confident it has to do with the data subfolder containing about 23GB of videos. When I copy the flake.nix and .envrc to a different folder and direnv allow it builds in about a second.

running nix shell /home/user/projects/lfg spends a long time in copying /home/user/projects/lfg and then outputs error: flake 'path:/home/user/projects/blah' does not provide attribute 'packages.x86_64-linux.default' or 'defaultPackage.x86_64-linux'

Running the same command in another directory with the same flake.nix and .envrc, but without the data folder, doesn’t spend many minutes copying, and outputs the same error immediately.

4

Yes. This is a current limitation of flakes. One way to get around this might be to put these videos outside of the git repository or having your flake leave in a subdirectory that is its own flake i.e. nix.
In direnv you can use than something like

use flake path:./nix

to than avoid having to copy your assets to the nix store.

1 Like
5

An alternative would be to use the old style nix-shell approach. So having instead a default.nix. This will also avoid copying things around.

6

I feel the simplest solution ended up being moving the flake.nix and .envrc to a src folder in the same directory as the data folder. Now the environment gets created much faster since the data isn’t copied. Thanks.