How to handle nixpkgs as a [very large] git repo

Help
1

Currently, when I need to tweak or try new things on nixpkgs, I git pull my local clone of nixpkgs, to keep up with the latest version, and then I start working.

This issue I have is that the nixpkgs repo is huge. It takes a lot of time just to git pull, every git operation is super slow, and it even makes my prompt lag, because in my prompt I query the “best” name I can find in the repo for the commit HEAD is currently at.

Are there any particular workarounds to handle such a massive repo? How do you work with it in your day to day life?

2 Likes
2

My prompt is lazy and shows a placeholder for the branch name (usually the last known) in a darker tint than usually.

After a couple of seconds the placeholder then is replaced with the newly discovered value.

Power level 10k deals with this and other lazy features for me.

This might be a feature only available in zsh.

For everything else I don’t actually care for the time it takes, as usually “progress” is displayed and it doesn’t take that long. 10 seconds at most.

4 Likes
3

Personally, I just don’t have a fancy git-interactive prompt, and use git status when I really need to know the branch, or my editor’s built-in VC status, which also isn’t synchronous.

Reverting/logging/etc. is painfully slow, but thankfully I don’t need to do those often enough that it gets in the way of actual editing.

4

Make sure you’ve GC’d recently and there are no loose objects in .git/objects (just pack).

I’ve found git’s new scalar to be a nice jump-off point for improving performance in large repos: Highlights from Git 2.38 - The GitHub Blog

4 Likes
5

I don’t think there’s a ton to be done about some of the operations being slow, but I chafed at this a bit myself.

I tried gitstatusd but still wasn’t super pleased so I ended up writing a bash prompt plugin to claw back that time in most contexts:

I’ve been picking at converting more of my own packages to flakes over the past few months and have one almost ready for lilgit. Not looking at my notes, but I think the main thing I wanted to fix is figuring out why the rust component stopped building against nixpkgs-unstable on darwin back in October.

1 Like
6

Thanks for the tips, I really appreciate your help.

This sounds like exactly what I would need, but unfortunately I work with Bash (maybe this will be the time I switch to zhs). Anyways, I’ll keep the powerlevel10k solution in mind.

Most of the time, it’s ok for me too, because my editor’s git support isn’t synchronous either. The problem really arises when I’m just navigating in the repo in a terminal, and it’s painfully slow (it’s actually only about half a second / a second of delay each time to show the prompt, but it’s still really annoying).

It looks like lilgit does basically what my prompt does, except my prompt tries a little harder in finding a nice name to be shown (ie. it also takes into account tags), and, of course, the colors have been tailored to my needs. But since I basically pass all my time into a non-detached mode anyways, it’s not that useful to spend energy in trying to find a nice name where 90% of the time it’s just the name of the branch. So, if you ever finish packaging it, I’ll give it a try.

1 Like
7

Some prompts like starship handle this kind of problem by setting a timeout for the prompt features. If you set an aggressive timeout, you won’t get nice git stuff in your prompt in nixpkgs, but at least it won’t be slow…

1 Like
8

I optimistically worked on it a bit yesterday just in case. I got it converted it from crate2nix to crane, which seems to have fixed the need to pin its nixpkgs.

I haven’t finished updating the readme yet, but I did update it with:

9

Another issue with nixpkgs’ size is that you probably don’t want to use the flake CLI with the currently available versions of Nix when working on nixpkgs. It always copies to the store. And even though it was only a few gigabytes worth after a fairly large amount of iterations for me personally (thanks, ZFS compression), it was still made up of lots and lots of random tiny files, which is the most hostile scenario for basically any file system, and it literally took an hour for nix-collect-garbage to get rid of it all. Future versions of Nix will eventually have a fix for this, but it has annoyed me.

6 Likes
10

Magit is also very slow with nixpkgs, I think it’s fetching all the tags or something. I’ve been meaning to look into it more.

3 Likes
11

I have this in my emacs:

  (remove-hook 'magit-status-headers-hook 'magit-insert-tags-header)
  (setq magit-revision-insert-related-refs nil)
  (remove-hook 'magit-status-sections-hook 'magit-insert-unpulled-from-upstream)
  (remove-hook 'magit-status-sections-hook 'magit-insert-unpushed-to-upstream-or-recent)

And at least the first two lines made it way faster; like, near instant for most things. I don’t remember if the other two helped the speed or if that was just preference.

7 Likes
12

I use git fsmonitor to improve git status speed to something acceptable for my prompt.

2 Likes
13

Thanks this gave me the push to revisit. Here’s a version that only disables those things for a specific repo!

(dir-locals-set-class-variables
 'huge-git-repository
 '((nil
    . ((magit-refresh-buffers . nil)
       (magit-revision-insert-related-refs . nil)))
   (magit-status-mode
      . ((eval . (magit-disable-section-inserter 'magit-insert-tags-header))
         (eval . (magit-disable-section-inserter 'magit-insert-recent-commits))
         (eval . (magit-disable-section-inserter 'magit-insert-unpushed-to-pushremote))
         (eval . (magit-disable-section-inserter 'magit-insert-unpushed-to-upstream-or-recent))
         (eval . (magit-disable-section-inserter 'magit-insert-unpulled-from-pushremote))
         (eval . (magit-disable-section-inserter 'magit-insert-unpulled-from-pushremote))
         (eval . (magit-disable-section-inserter 'magit-insert-unpulled-from-upstream))
         ))
))

(dir-locals-set-directory-class
   "/home/ryantm/p/nixpkgs/" 'huge-git-repository)
6 Likes
14

Thanks for the magit configuration. It works fine with some quirks… For example, in the magit status buffer, if the curser is at the top (at Head, Merge, or Push), typing SPC b x to delete a branch takes about 4 seconds. However, if I move the cursor below, the branch selection popup appears instantaneously. Do you observe the same behavior and have any ideas as to why this could be the case?

Further, I think you have

         (eval . (magit-disable-section-inserter 'magit-insert-unpulled-from-pushremote))

twice, and the function magit-refresh-buffers does not exist. (Maybe magit-refresh-buffer?).

1 Like