We mentioned in our last update that we had re‐established the line of contact with GitHub. We heard from them for the first time on 2025-11-14 about a concerning scaling issue with their infrastructure: periodic maintenance jobs on the Nixpkgs repository were regularly failing and causing issues achieving consensus between replicas. If this problem worsened, it could have led to replication failing entirely and the Nixpkgs repository becoming read‐only. This was also contributing to issues we saw with GitHub APIs over the past weeks, such as timeouts in diff comparisons and periodic merges.
The Nixpkgs core team, members of the NixOS Foundation board, and representatives from GitHub had a call on 2025-11-17 to discuss this issue in depth (thanks to @infinisil for taking notes!).
Thankfully, we believe the immediate issue has been solved thanks to fixes made on the GitHub end, and that periodic maintenance and API calls should now work reliably again. We are still working with GitHub to fully understand the underlying causes of the immediate issue, what was changed to resolve it, and if there is anything else that can be done to ensure that GitHub’s backend doesn’t run into issues with Nixpkgs again. The rest of this update is to document what we heard from GitHub, what potential causes we considered for the issues based on that, and what we did to start diagnosing and fixing them.
Our current understanding is:
-
The central copy of the Nixpkgs Git repository on GitHub is growing over time. It’s unclear how much of this is due to the normal organic growth of the project and how much is unexpected “dark matter”. Currently, the repository has around half a million tree objects and 20k forks, totals around 83 GiB (we are not sure whether this is on‐disk packfile size or unpacked object size), and is growing steadily. This growth was GitHub’s primary hypothesis for what could have caused periodic maintenance to start timing out.
-
GitHub usually do not clean up unreferenced objects automatically, but manually did so for Nixpkgs about a month ago, which shrank the repository significantly but did not fix periodic maintenance. We believe the 83 GiB figure should therefore reflect objects that are not eligible for GC or that were created in the past month. We don’t yet know exactly how objects are being accounted, though, or the cause or pattern of the repository size growth.
-
We have no reason to believe that the number or size of files in Nixpkgs was a major contributor to this issue. A local bare clone is only 2.5 GiB, which matches the on‐disk size of the main fork as measured by GitHub, so most of this size is presumably not from objects transferred during a standard clone – rather, GitHub stores the entire “fork network” of a repository in a single Git repository, which includes all pull requests and personal branches, and merge commits created by CI and the merge queue.
-
There is also no indication that our consumption of Nixpkgs through Git clones or tarball downloads is relevant in any way; whatever was contributing to the maintenance issues would have been purely on the write end.
-
GitHub’s Git backend has various bottlenecks around changes to Git refs, the total number of refs and trees across the fork network, and the number of PRs (total or open – unclear). We do not know if any of these contributed to the issue with periodic maintenance. There are improvements for some of these on the horizon.
-
Our CI queries information about open PRs every day. The API endpoint we use computes mergeability information, which creates a new merge commit every time and writes it to the
pull/…/mergeref. The old merge commits could be contributing to the growth, and it is possible these objects get accounted to the respective forks the PRs are from, although we would expect GC to have had a major impact here if so. Based on the call and GitHub’s advice to start off with low‐impact CI changes so they can measure the effect on their metrics, we made a change to CI to reduce the use of the API endpoint that we think may be contributing to excessive object writes – though we have not heard about whether or not this had any impact. If these merge commits and their corresponding ref writes could potentially cause problems in future, they could be reduced through use of the GraphQL API. -
GitHub told us that some forks appear to have diverged significantly from the main history. Based on metrics they collected pointing to some unusual aspects of the fork network, we reached out to a contributor whose fork contains a large number of refs from the upstream repository mirrored under a non‐standard namespace, likely by mistake. Cleaning those up may help with the GitHub backend bottlenecks, even if it isn’t related to the periodic maintenace issues.
We appreciate GitHub’s willingness to spend time getting to the bottom of this issue with us. We will continue to work with them to ensure that this issue is fully resolved and that Nixpkgs development can continue smoothly and sustainably. Hopefully, we will be able to get more concrete information about what the underlying cause of the issues being observed was, what they did to fix it, and any potential future risks. Due to the urgency of the situation and the impact it could have had on Nixpkgs development if not addressed, we prioritized it at the expense of the plans we’ve shared in earlier updates. We’re sorry about the conversations and initiatives we deferred under the circumstances, and hope to have another update soon.
