Can I have access to the git history in a Derivation? (Source code makes calls to git log)

Help
1

I’m moving my github action runner away from npm/bundler and to using Install Nix · Actions · GitHub Marketplace · GitHub

Testing locally… I think I’ve hit a snag that I don’t quite know how to fix. My blog’s articles generate a changelog using git and that relies on having access to the git repo… and I think when nix yanks the source repo to start a nix build it doesn’t bring the whole .git folder and history with it?

so when the nix-dominated ruby script runs and makes calls out to git log I see errors that this isn’t a git repo and there’s no .git folder.

The things I’ve thought:

  1. Figure out a way to make the full git history available to the git executable when the static site builder is running.
  2. Use a different method to get the “changelog…” maybe computing it before-hand?
  3. It could be that using nix build to run a static site generator to generate a folder of .html files is the wrong path entirely.

I’m pretty stuck.

1 Like
2

Can you explain a bit more about how this should work. We’re talking about a pipeline runner so is it correct to assume that your repo is already checked out there? Is there a reason you cannot use the history from that?

Having said that, have a look at Built-in Functions - Nix Reference Manual

3

I use middlemanapp.

The code I use to configure the builder (middleman) and the content that is built is all in one repo. My flake looks like:

{
  description = "evantravers.com (middleman, ruby)";

  inputs = {
    nixpkgs.url                         = "nixpkgs/nixpkgs-unstable";
    nixpkgs-ruby.url                    = "github:bobvanderlinden/nixpkgs-ruby";
    nixpkgs-ruby.inputs.nixpkgs.follows = "nixpkgs";
    flake-utils.url                     = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, nixpkgs-ruby, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};

        ruby = nixpkgs-ruby.packages.${system}."ruby-3.2.1";

        gems = pkgs.bundlerEnv {
          # The full app environment with dependencies
          name = "middleman-env";
          inherit (pkgs) ruby;
          gemdir = ./.; # Points to Gemfile.lock and gemset.nix
        };

        buildPackages = [
          ruby
          gems
          pkgs.bundix
          pkgs.nodejs-slim
          pkgs.pagefind
          pkgs.git
        ];
      in with pkgs;
      {
        devShells.default =
          mkShell {
            buildInputs = buildPackages ++ [
              pkgs.solargraph
            ];
          };

      packages.default =
        stdenv.mkDerivation {
          pname = "evantravers.com";
          version = "1.0";
          src = ./.;
          buildInputs = buildPackages;
          buildPhase = ''
          middleman build --verbose
          pagefind --site build
          '';
          installPhase = ''cp -r build $out'';
      };
    }
  );
}

In a nix shell, I can run middleman build and middleman takes the contents of my ./source folder and builds the result into ./build. When I run nix build I run into the “not a git repository” problems.

The reason that the “builder” is calling out to git is because I use git log to build changelog’s of blog posts. I played with commenting those out, and then it works fine. I tried using builtins.fetchGit { url = ./.; shallow = false; } with nix build --impure, but then I was having git crash with error 128, which is what I’m chasing at the moment.

4

What do you get if you try to run nix build ./#<your derivation> from the root of your repo?
nix flake show ./ will show you all your flake outputs.

5

I created a smaller version of my blog with a limited history… (partly because the repo is giant after a decade): evantravers/nix-discourse-evantravers.com · GitHub

When I run nix build in that root folder of that repo I get the following errors: gist:e3ffb26ca5eec326e85f91f80096f962 · GitHub

What I’m seeing is that every time the file would call out to git log it throws that error, and that ends up in the middleman logs, which are passing it up to the nix derivation logs.

I’ve played with adding a ls -la to the build phase instructions, and there’s no ./.git folder, so I’m guessing that’s a big part of it.

Thank you for your patience and help @jalphad. I really appreciate it.

6

So it seems that dot directories are not copied from the source. Can you try

stdenv.mkDerivation {
  pname = "evantravers.com";
  version = "1.0";
  src = ./.;
  dontUnpack = true;
  buildInputs = buildPackages;
  buildPhase = ''
  cp -r $src ./
  middleman build --verbose
  pagefind --site build
  '';
  installPhase = ''cp -r build $out'';
};
7

I like where this is headed. I tried what you posted and got the same errors. I also tried removing dontUnpack = true; and using:

        stdenv.mkDerivation {
          pname = "evantravers.com";
          version = "1.0";
          src = ./.;
          buildInputs = buildPackages;
          phases = ["buildPhase"];
          buildPhase = ''
          cp -r $src/* ./.
          ls -la
          middleman build --verbose
          pagefind --site build
          '';
          installPhase = ''cp -r build $out'';
      };

That cp still seems to be not bringing along the .git folder.

8

If you run the experimental nix build in a Git directory, it is currently interpreted as a git+file: URL which cannot include any files not tracked in Git, including .git.

The best way to fix that is to tell the program that tries to read .git to not do that, and instead pass it the Git version information you receive from the flake, such as self.rev. You’ll find lots of examples of this in Nixpkgs, e.g. here or here.

There’s also other ways of getting around that, but these won’t work when you build e.g. from a builtins.fetchGit or pkgs.fetchFromGitHub source, so I don’t recommend them (though you could set leaveDotGit = true, which would work):

  • You can explicitly tell Nix to not do the Git filtering on the repo using nix build path:.. This comes with the drawback that it’s going to copy the entire repository, including the full history, into the Nix store every time you build, so I wouldn’t recommend that.
  • You can use the stable nix-build interface which doesn’t have any implicit filtering behavior, which then allows you to e.g. explicitly filter for git-tracked files but adding some file from the .git directory:
    src = lib.fileset.toSource {
  root = ./.;
  fileset = lib.fileset.union
    (lib.fileset.gitTracked ./.)
    # Also include this file in the source, only works reasonably with `nix-build`
    ./.git/HEAD;
}
9

If I understand you correctly, you are suggesting bringing in just the current commit or sha for tagging purposes. I’m looking to do a little more than that.

Each markdown file on my blog when it’s being built calls out to git log to get it’s changelog, which is then rendered as part of the final page. The goal is to provide a list of edits. So I think I want more than just .git/HEAD or self.rev.

You have provided a great clue with the reference to git+file:'s behavior. Maybe that rabbit trail will lead me down the right road.

Can you think of a way that during the buildPhase I could run git log commands against a file?

10

Reading the comment from @Infinisil above I think you basically have two options:

stdenv.mkDerivation {
  pname = "evantravers.com";
  version = "1.0";
  src = fetchgit {
    url = "https://<repo>";
    leaveDotGit = true;
    hash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
  };
  ...
};

Problem here is that you have to inject the hash here somehow prior to calling nix build. You’re also again fetching your repo.

The other option is to not use flakes but use a regular default.nix and build using nix-build.