How to implement version 2 yarn.lock package management

gador · February 4, 2025, 9:14am

Hi,

there is a rather long discussion about how to implement a third-party package management using yarn.lock version 2 files.

There are several pro and contra arguments, which I try to summarize here:

Contra:

If upstream changes its implementation it could potentially break FOD ^[1]
It relies on supportedArchitectures [Settings (.yarnrc.yml) | Yarn] which, if changed, could cause breakage of FODs ^[2]

Pro:

We implement a coherent system around yarn, which currently only supports yarn version 1 files
No conversion of yarn version 1 to version 2 files is needed (which also happens to be buggy and hard to do…See e.g. my workaround for pgadmin)
No need to commit an edited yarn.lock file to nixpkgs, as it can directly work with the newer versions

A POC PR has been made which has several comments which I hopefully summarized correctly above.

Obviously I am biased, as I’ve written the PR, towards an implementation. I would ask anyone with some experience in the yarn ecosystem and especially everyone already involved, to chime in. Discussion stalled last December and I would hate to see the effort wasted here.

I answered the contra arguments above on GitHub to the corresponding comments and would love some additional feedback.

Many, many thanks to everyone who allocated some freetime to review the PR until now:
@emily @doronbehar @winter @szlend @the-sun-will-rise @Sandro

github.com/NixOS/nixpkgs

Review by winterqt - Add yarn-berry3 and yarn-berry4 to fetch and use version 2 yarn.lock files

NixOS:master ← gador:yarn-berry-fetch

This will break the FODs almost instantly if Yarn changes anything in how they p…ull deps, and also is probably not reproducible between platforms when platform-specific dependencies are used.

↩︎
github.com/NixOS/nixpkgs

Review by emilazy - Add yarn-berry3 and yarn-berry4 to fetch and use version 2 yarn.lock files

NixOS:master ← gador:yarn-berry-fetch

Unfortunately, I think I was correct here: > Or can packages on npm reference… other platforms and Yarn just can’t support fetching them? The npm docs imply that the fields are quite freeform; e.g. the [`cpu` example](https://docs.npmjs.com/cli/v10/configuring-npm/package-json#cpu) uses `mips` as an example. That means that this is just a limitation of Yarn’s configuration format rather than a constraint on the actual packages in existence, and there are packages that could change in the future if we add new platforms like RISC‐V. The correct solution here would be to fetch everything independent of platform – theoretically Yarn could add `all` values here to opt in to fetching everything, but writing our own deterministic fetcher would make it easy and also obviate the other reproducibility concerns. If neither of those is an option, perhaps we can find a way to check that nothing in the dependency tree references any platform we haven’t included in this file, and otherwise bail out? If we can’t do even that, then it’d be good if someone could write a script to scan the npm repository and collect every value that is used in practice so that we can include them all from the start and reduce the scope of potential changes. Because we’re adding fundamental packaging infrastructure here that is part of our public interface and will surely be relied upon by out‐of‐tree packages, and changing FOD hashes are very difficult to spot and debug, I don’t think we’ll have the option of *just breaking it* in the future, unless we can show that the backwards incompatibility would be entirely theoretical in the existing public corpus.

↩︎