Formalized "Legacy Package" Process for Critical Updates in Nixpkgs

[spiage]

Hello everyone,

I would like to start a discussion about a recurring and painful problem in nixpkgs, which we recently faced with the langchain update. I propose we consider a formalized process for handling such situations to make the ecosystem more resilient and reduce the burden on maintainers.

Problem Description

Periodically, a critical library update in nixpkgs introduces breaking API/ABI changes (for example, libfoo from 1.x to 2.0). This leads to a cascade of build failures for all packages that depend on it (A, B, C, …).

Current Consequences:

Master branch blockage: The dependent package tree fails to build for several days or even weeks.

Rushed “Hacks”: Maintainers of dependent packages are forced to quickly create fragile patches to unblock the build, which is a temporary and unreliable solution.

Proposed Solution: Temporary “Legacy” Packages

I propose introducing a formalized process for creating temporary legacy packages during the transition period.

How it should work:

Trigger: The process is activated when a maintainer plans an update that breaks ABI/API and is expected to affect more than N packages (for example, N=5).

Action: Along with the commit updating the main package (libfoo to 2.0), a new package (libfoo-prev_semver) is added in the same PR.

This package contains the previous stable version (1.x).

It should be marked with a special attribute, for example: meta.isTemporaryLegacy = true;.

Legacy Package Lifecycle (Key Point):

Automatic Expiration: The package is assigned a lifespan (EOL), for example, 3-6 months from creation.

Automated Reminders: The CI system or a bot can automatically create Issues/PRs to remind about the need to remove the package after its EOL.

Marking as Outdated: One month before removal, the package is automatically marked as insecure to encourage migration away from it.

Adapting Dependent Packages: Maintainers of packages A, B, C can quickly and safely switch their dependencies to libfoo-1, immediately unblocking the master branch. This gives them time to properly adapt to the new libfoo 2.0 version.

Advantages of This Approach

Unblocks Master: The build is restored within hours, not days/weeks.

Continuous Security Updates: Critical fixes continue to flow into nixpkgs.

Reduced Stress and Load: Maintainers don’t have to work in emergency mode and can plan their work.

Prevents “Package Pollution”: A formalized and automated lifecycle directly addresses the main concern about repository “pollution”. The temporary package won’t be forgotten and won’t remain in the system forever.

Clear Path for Upstream Interaction: Creates a clear signal for upstream developers: you have several months to adapt, after which the old version will no longer be supported in Nixpkgs =)

Open Questions for Discussion

What should the activation threshold be (number of broken packages)?

Who is responsible for creating the legacy package — the author of the main update or a dedicated team?

What is the optimal lifespan for such packages? 3 or 6 months?

Which naming scheme is preferable (pkgname-prev_semver, pkgname_legacy)?

I am confident that introducing such a process will make nixpkgs a healthier and more predictable project for both maintainers and end users.

What does the community think?

[arianvp]

Can you give examples of this happening? The exact scenario you describe is why we have the staging workflow. Packages with lots of downstream dependencies shouldn’t get breaking changes merged to master ever. They should go through staging and staging-next. What is lacking in those flows according to you?

[spiage]

Can you please read this thread

open-webui: build failures

and I think it was the starting PR

python3Packages.{langchain,langgraph}-*: updates and build fixes

[spiage]

Nevertheless, I think having a mechanism to keep the previous upstream package version is a useful capability.

For example, judging by the speed of previous lock-file updates, it will take the open-webui upstream several months to transition to the new version of langchain (which we have as current now)

[arianvp]

Well the problem here is that you’re expecting guarantees from a non-open source package that isn’t built by hydra due to licensing issues. That’s a completely different problem. We have no way of guaranteeing that they work and it’s hard to ask from a project to not break projects we can’t legally build and distribute.

I don’t really have interest in formalizing processes to bend to projects that we can’t even build in our CI. The fix is to not depend on things that we cant build and test in this particular case IMO

[vcunat]

@spiage

Adapting Dependent Packages: Maintainers of packages A, B, C can quickly and safely switch their dependencies to libfoo-1, immediately unblocking the master branch. This gives them time to properly adapt to the new libfoo 2.0 version.

We already do have some packages in multiple versions. A problem is that if you overuse this, especially for popular libraries, you tend to get into clashes. Many “languages”, e.g. C libraries, are not meant to deal with multiple versions. For a single binary you normally only allow a single version of each library for the whole linking closure. (I’m not sure if e.g. Python suffers also from this.)

[reckenrode]

This particular problem is due to Python packages being updated together as a package set. If a package requires a specific (usually older) version, it needs to use an override in nixpkgs to pin to the version to exactly what it needs. See the Airflow derivation for an example of how to do that.

[Infinisil]

@vcunat

We already do have some packages in multiple versions. A problem is that if you overuse this, especially for popular libraries, you tend to get into clashes.

This was indirectly addressed in @spiage’s post:

@spiage

Automatic Expiration: The package is assigned a lifespan (EOL), for example, 3-6 months from creation.

I like this idea!

[Atemu]

Not really, no.

Staging is for bundling rebuild-heavy changes to occur in a limited number of global rebuilds. Its policy w.r.t. breaking changes is the same as master.

The only difference is that it’s often quite a bit harder to ascetrain the knock-on effects on reverse dependencies w.r.t. build failures because there’s just too many of them; requiring testing to be done at a later stage than before the PR is merged.

CONTRIBUTING.md

Breaking changes in this regard do correlate with large amounts of rebuilds but that’s the just that: a correlation.

While the motivation might have been a proprietary software, this happens in OSS too and has happened a bunch of times already.

I support having a standard method to deal with this circumstance because that way everone can predict what will happen.

I’d caution against over-formalising it though. I think it’d be enough to have this be merely a (documented!) convention, not a rule or process.

A 6 Month grace period with 1 Month warning window sounds like a really good starting point to me.

[reckenrode]

My concern with formalizing a process like this is it’s just pushing out the updates to effect the removal, which will probably end up being done by a motivated contributor rather than the package maintainers. I’d not want to rely on that because those kinds of cleanups can burn people out.

There are also questions about how to handle
transient dependencies if they also depend on your dependency. You need to override them too. That’s kind of what overrideSDK did. It could be resurrected as a generic function to replace a dependency, but it can’t get them all (e.g., if it’s substituted or interpolated into a script).

If we’re going to package language ecosystems with a prescribed version, we should probably have a standardized way to override the version across a package’s dependencies and documentation on how and when to do that.

[sarahec]

Speaking as the primary langchain/langgraph maintainer: Langchain started releasing 1.0 alpha versions back in August. I kept a local branch where I tested builds ~weekly until I saw substantial adoption. 1.0.0 release landed after that and I pushed those updates to staging (both for the number of rebuilds and to allow time for any stragglers to catch up.)

Once that landed, a few packages broke. Adding 'classic fixed most of them, the rest could be patched or retired (namely, if obsolete, unmaintained, and had no dependents).

open-webui, in particular, pinned to the last pre-1.0 alpha release of langchain and depended on its internals rather than the documented API. This kept us from swapping in 'classic and needed source changes, see open-webui: patch imports for langchain v1 by thunze · Pull Request #463538 · NixOS/nixpkgs · GitHub

Would keeping temporary “legacy packages” help? Perhaps if it’s a single package, but this becomes messy when it’s an ecosystem. That’s why I took an informed “watch and wait” approach. We can always patch (and send patches upstream, if sensible).