Extracting sub-directory from fetchGit or fetchurl (or any derivation)

Help
1

Is there a builtin or nixpkgs function to take one derivation and return all the data at a relative path as a new derivation?

This would be helpful for packaging mono-repo code projects. For instance, I often store multiple python packages in one git repository. How would one build a python package for this case?

To be concrete, suppose I have a git repo myorg/monorepo containing a python package at path/to/package? How would I pass this package to buildPythonPackage?

I am thinking of something along the following lines:

{buildPythonPackage} :
let 
  repo = builtins.fetchGit {
        url="git@github.com:myorg/monorepo.git";
        rev="5a3395608b8cc199c51adfc64e9639f539b87b911";
    };
in
  mypackage = python3Packages.buildPythonPackage {
    src = selectSubDirectory { src=repo; dir='path/to/package' };
  }

Is there such a function selectSubDirectory? filterSources seems close, but not identical.

1 Like
2

you can do postFetch to run some commands to “groom” the source before making the final fixed output derivation. However, this won’t save the initial pull from being quite large.

If a source can be used for many things, i would probably recommend for it to remain large, and change sourceRoot in the consuming derivations to then point to the directory that you care about.

2 Likes
3

Thanks. The initial pull isn’t too large so sourceRoot sounds like a good solution. I assume that buildPythonPackage doesn’t override that option.

4

buildPythonPackage will pass it through to mkDerivation

example: https://github.com/NixOS/nixpkgs/blob/c5911744de78e8dd104320fde177dfe2b7b49d85/pkgs/tools/admin/azure-cli/default.nix#L20

5 
{buildPythonPackage} :
let 
  repo = builtins.fetchGit {
        url="git@github.com:myorg/monorepo.git";
        rev="5a3395608b8cc199c51adfc64e9639f539b87b911";
    };
in
  mypackage = python3Packages.buildPythonPackage {
    src = "${repo}/path/to/package";
  }

Though this has the problem, that indeed the full repo will be cloned. For usecases like this, it would be nice if fetchgit would support shallow and/or sparse checkouts…

3 Likes
6

How does one use sourceRoot in this case? In packaging torch-tb-profiler (pypi and repo), I’ve tried and failed to use sourceRoot in every way I can think of with no success. I have a basic derivation and I’ve tried tb_plugin, ./tb_plugin, kineto/tb_plugin, and ./kineto/tb_plugin. I get an error:

chmod: cannot access 'tb_plugin': No such file or directory

What’s the correct incantation here?

7

either:

prePatch = ''
  cd tb_plugin
'';

or

sourceRoot = "${src.name}/tb_plugin";
2 Likes
8

Ah ok, thanks for clarifying!

9

Sparse checkouts supported after fetchgit: Support sparse checkout by azuwis ¡ Pull Request #135881 ¡ NixOS/nixpkgs ¡ GitHub

3 Likes