`rev` and `ref` attributes in `builtins.fetchGit` (and maybe flakes too?)

Development Documentation
1

Update: made a comment of recapping all the builtins.fetchGit oddities in Nix issue #5128.
When builtins.fetchGit is called with an attribute set, the following optional attributes can be specified:

rev
    The git revision to fetch. Defaults to the tip of ref.

ref
    The git ref to look for the requested revision under. This
    is often a branch or tag name. Defaults to HEAD.

    By default, the ref value is prefixed with refs/heads/.
    As of Nix 2.3.0 Nix will not prefix refs/heads/ if ref
    starts with refs/.

I think my understanding of Git revisions and references is solid, and these descriptions in the Nix manual seem to be misleading - and even mutually exclusive in certain cases even. However, I don’t know all of the Git functions / tools that Nix is calling underneath that may impose special semantics, so what contradictions I believe I found below may not be contraditictions at all.

All the examples below are carried out on the toraritte/my-project repo that looks like this:

-BRANCH- -COMMIT MESSAGE--- -SHORT- -----------LONG------------------------
(topic)  smarties cereal    5f45e9c 5f45e9c854941c72deb9d36fb3e95e4feb4d698f
         Say, I hate stress 6692c9c 6692c9c6e231b1dfd5594dd59b32001b70060f19
(main)   More to say        c277976 c277976fce0b2b32b954a66d4345730b5b08f1db
         Add sentence       abb0819 abb08192ed875ef73fa66029994aa2f6700befd0
         init               e67cb07 e67cb07f9ddb0ecd0f88fcf36093d8d8bf928b75

Figure 1: The table is a combination of the outputs of git log --pretty=oneline and git v --color=always --all

  • rev: The git revision to fetch.

    In contrast to the Git revision syntax, builtins.fetchGit’s rev attribute only accepts “the full SHA-1 object name (40-byte hexadecimal string)”; it doesn’t even accept an abbreviated SHA-1 hash.

    Example 1

    nix-repl> builtins.fetchGit {
            url = "https://github.com/toraritte/my-project";
            rev = "HEAD~2";
          }                           
error: hash 'HEAD~2' has wrong length for hash type 'sha1'

    Example 2

    nix-repl> builtins.fetchGit {
            url = "https://github.com/toraritte/my-project";
            rev = "5f45e9c";
          }                        
error: hash '5f45e9c' has wrong length for hash type 'sha1'

    Example 3

    nix-repl> builtins.fetchGit {
            url = "https://github.com/toraritte/my-project";
            rev = "5f45e9c854941c72deb9d36fb3e95e4feb4d698f";
          }
{ lastModified     = 1658846311;
  lastModifiedDate = "20220726143831";
  narHash          = "sha256-Yph6e...eoDAh/W6xaG+j5oFAui80c1FMYaGPTY=";
  outPath          = "/nix/store/w...xrfyh1yj60v4phaf49ccyjd0-source";
  rev              = "5f45e9c854941c72deb9d36fb3e95e4feb4d698f";
  revCount         = 5;
  shortRev         = "5f45e9c";
  submodules       = false;
 }

  • rev: Defaults to the tip of ref.

    A Git reference is an alias for the SHA-1 hash of a particular singular Git object, therefore “tip of ref” is meaningless. If the ref attribute’s value is a branch reference1 (e.g., /refs/heads/main), that will still only resolve to a single commit.

    As for builtins.fetchGit, from what I’ve seen in nix repl,

    • when the rev attribute is not provided, it defaults to ref.
      Example 4

      nix-repl> builtins.fetchGit {
          url = "https://github.com/toraritte/my-project";
          ref = "main";
        }   
{ lastModified     = 1658846059;
  lastModifiedDate = "20220726143419";
  narHash          = "sha256-FQUE8ek9uoy...LH+yv4S4=";
  outPath          = "/nix/store/3ra7y3v...8r2d5p7k4irmiwrp-source";
  rev      = "c277976fce0b2b32b954a66d4345730b5b08f1db";
  revCount         = 3;
  shortRev         = "c277976";
  submodules       = false;
 }

$ git show-ref refs/heads/main
c277976fce0b2b32b954a66d4345730b5b08f1db refs/heads/main

    • Otherwise, if rev (i.e., a specific commit hash) is present, then ref will simply be ignored.
      Example 5

      nix-repl> builtins.fetchGit {
            url = "https://github.com/toraritte/my-project";
            ref = "main";
                   _______
            rev = "5f45e9c854941c72deb9d36fb3e95e4feb4d698f";
                   ^^^^^^^
          }
{ lastModified     = 1658846311;
  lastModifiedDate = "20220726143831";
  narHash          = "sha256-Yph6eC...Ah/W6xaG+j5oFAui80c1FMYaGPTY=";
  outPath          = "/nix/store/wa...yj60v4phaf49ccyjd0-source";
                      _______
  rev              = "5f45e9c854941c72deb9d36fb3e95e4feb4d698f";
                      ^^^^^^^
  revCount         = 5;
  shortRev         = "5f45e9c";
  submodules       = false;
 }

$ git show-ref refs/heads/main
_______
c277976fce0b2b32b954a66d4345730b5b08f1db refs/heads/main
^^^^^^^

  • ref: The git ref to look for the requested revision under. This is often a branch or tag name.

    I presume the first sentence states that the povided rev attribute value (i.e., commit) should be a parent of ref, and along that line, the second one implies that if if the ref is a branch reference thenrev needs to belong to that branch (i.e,; the commit hash in rev has to be a parent of that branch’s HEAD).

    None of that is honored though: in example 5 above, ref refers to the main branch, and rev is "5f45e9c854941c72deb9d36fb3e95e4feb4d698f" - the HEAD of the topic branch, which commit is not even part of main:

    * 5f45e9c (HEAD -> topic, origin/topic) smarties cereal
* ....... Say, I hate stress
* c277976 (tag: miez, origin/main, main) More to say
* ....... Add sentence
* ....... init

  • ref: Defaults to HEAD.

    Have nothing on this one.2

    [2]: Was only able to find where ths is done for GitHub/Gitlab/SourceHut in github.cc (see), but no joy for git.cc…

7.5.11 nix flake in the Nix manual also states the same with an added explanation/justification (emphasis mine):

The rev attribute must denote a commit that exists in the branch or tag specified by the ref attribute, since Nix doesn’t do a full clone of the remote repository by default (and the Git protocol doesn’t allow fetching a rev without a known ref). The default is the commit currently pointed to by ref.

  • Does Nix flakes use the same C++ functions to fetch as builtins.fetchGit? If yes then I don’t think that restriction is honored.

  • Which Git protocol is mentioned? There are revisions that don’t need an anchor (i.e., a commit hash or a Git reference), such as :/<text> and :\[<n>:\]<path>

2

It may be more accurate to simply say that rev defaults to ref

No, they are two different things. ref refers to a branch or tag, and rev needs to point to some commit which occured along that commit chain. The more accurate way to describe this is that rev will default to the commit pointed to by ref.

If the ref attribute’s value is a branch reference 1 (e.g., /refs/heads/main ), that will still only resolve to a single commit.

You can do something like main^ which is, go to main, then take the previous commit. The ref is just a starting point, but you’re free to also select any parent commit.

Have nothing on this one. 2

HEAD is used by git to determine where the repository is currently checked out. When you checkout a branch, or clone a repo; git needs to point to some commit. The commit that is “currently in focus” is tracked by HEAD. You can also think of it as a cursor through commit history.

This relates to fetchgit because fetchgit will checkout the repository. If the rev specified differs from the tip of ref, then you will be in a “detached head” state; which means “you’re not on a timeline (branch), but rather at a point in time (specific commit)”.

flakes

I mention this in my video here: https://youtu.be/90P-Ml1318U?t=525

But you can treat the flake url as… an url

inputs.nixpkgs.url = "github:nixos/nixpkgs?ref=staging-next&rev=938920192";
2 Likes
3

It’s too bad rev wasn’t named commit since (AFAICT) it is always the hash of a commit.

I find myself mixing these up all the time.

2 Likes
4

It’s too bad rev wasn’t named commit since (AFAICT) it is always the hash of a commit.

Not entirely correct. rev is something that points to a commit. It can be a commit but it can also be a tag as an example.

I find myself mixing these up all the time.

You and me both.

My way of remembering. A src in mkDeriVation takes a reV and is therefore a specific commit (or an alias for it).

I’m not saying it’s a good way to remember it….

2 Likes
5

Here’s an attempt at re-imagining the builtins.fetchGit doc:

Wanted to do a PR or draft PR tonight with notes to explain some choices, but even though the markdown renders good (at least on Discourse, GitHub, Stackoverflow), it’s a mess after compiling NixOS/nix. A run at my machine takes ~20 mins and didn’t want to waste time if this is not a direction where we want to go.

6

Btw. is builtins.fetchGit impure? I’m wondering because it doesn’t need a sha256 or hash attribute, although something like ref="master" can point to different commits over time. If so, is there a way to lock its input by providing a hash or would one need another fetcher function for that? Is providing a full commit hash as rev enough?

1 Like
7

builtins.fetchGit is pure if rev is present, so if you only set ref = "master" it will be impure

2 Likes