How to use Git LFS with fetchgit

Help
1

I am trying to package llm models from huggingface but don’t know how to make the LFS fetch work. Isn’t there already a best practice for using LFS in fetchgit?

Thats what I am currently having:

gpt2 = stdenv.mkDerivation rec {
    name = "gpt2";
    src = (fetchgit {
      url = "https://huggingface.co/openai-community/${name}";
      hash = "sha256-qqIGAcibiMy3dh6/WSZWrPqrJQA+zd9jCzAnRYKPaCs=";
      fetchSubmodules = true; # needed to use fetchgit internally
      leaveDotGit = true; # needed to preserve the .git dir
      postFetch = ''
        git lfs fetch
      '';
    }).overrideAttrs (oldAttrs: {
      nativeBuildInputs = oldAttrs.nativeBuildInputs or [] ++ [ git-lfs ];
    });

    installPhase = ''
      mkdir -p $out
      cp -r * $out
    '';
  };

Thats the feedback I am getting:

nix build .#llms.gpt2
error: builder for '/nix/store/8wl8q5xvi7jxflcmlcpb2z9llh5bhfbx-gpt2.drv' failed with exit code 128;
       last 10 log lines:
       >  * branch            HEAD       -> FETCH_HEAD
       > Switched to a new branch 'fetchgit'
       > Enumerating objects: 28, done.
       > Counting objects: 100% (28/28), done.
       > Compressing objects: 100% (27/27), done.
       > Writing objects: 100% (28/28), done.
       > Total 28 (delta 3), reused 0 (delta 0), pack-reused 0 (from 0)
       > Enumerating objects: 28, done.
       > Nothing new to pack.
       > Not in a Git repository.
       For full logs, run 'nix log /nix/store/8wl8q5xvi7jxflcmlcpb2z9llh5bhfbx-gpt2.drv'.
error: 1 dependencies of derivation '/nix/store/zzja19d5lm2zajsyk025vlfpdx5miwa5-gpt2.drv' failed to build
2

It took me way to long to find this information. So there is actually a documentation issue.

I found this Pull request, which implements LFS in a great way.

1 Like
3
2 Likes