# Nixpkgs's current development workflow is not sustainable
**URL:** https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741
**Category:** Development
**Created:** [April 20, 2022, 9:16pm UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741 "2022-04-20T21:16:55Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@samuela](https://discourse.nixos.org/u/samuela)
#### Post date: [April 21, 2022, 9:26pm UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/21 "2022-04-21T21:26:02Z")
> [@ajs124](#):
>
> TL;DR: What even is a maintainer?
IMHO a maintainer is someone who
- responds to PRs and issues for their packages in a timely manner (eg within a week)
- (optionally) updates their package as necessary
> [@jonringer](#):
>
> However, many python library owners don’t care about how their package interacts with the rest of the ecosystem, so sometimes you get [interactions like this](https://github.com/simplistix/testfixtures/issues/169).
Oof, what an unbearable attitude! I’m sorry you had to deal with that. People like that give OSS a bad rap.
I notice that the sybil package has no maintainers in Nixpkgs. Out of curiosity, why not just remove it? Or mark it as broken?
> [@jonringer](#):
>
> One benefit of the current process is that we serve as a crucible for python updates, and can alert respective upstreams about issues which they may not be aware of due to extreme pinning being the norm.
I think this is a nice, pro-social thing to do within the OSS community – I do it as well – but I feel we must be honest with ourselves as to whether it brings us closer to achieving our goals in Nixpkgs. Maintaining python in Nixpkgs is hard work already, pitching in to the maintenance of all python packages in existence is nearly impossible.
Also, just to clarify: I’m not proposing that we stop communicating issues upstream; rather that before pushing a change to Nixpkgs, commit authors are forced to confront those exact issues instead of pushing the bugs onto users and other maintainers. Increased stability makes the experience less stressful for everyone involved.
---
### Author:  [@jonringer](https://discourse.nixos.org/u/jonringer)
#### Post date: [April 21, 2022, 9:57pm UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/22 "2022-04-21T21:57:08Z")
> [@samuela](#):
>
> I notice that the sybil package has no maintainers in Nixpkgs. Out of curiosity, why not just remove it? Or mark it as broken?
Like most python packages, there’s a few other modules which require it.
```auto
$ rg sybil -l | grep python-modules
pkgs/development/python-modules/flufl/lock.nix
pkgs/development/python-modules/testfixtures/default.nix
pkgs/development/python-modules/atpublic/default.nix
pkgs/development/python-modules/sybil/default.nix
pkgs/development/python-modules/scrapy/default.nix
```
---
### Author:  [@samuela](https://discourse.nixos.org/u/samuela)
#### Post date: [April 21, 2022, 10:41pm UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/23 "2022-04-21T22:41:03Z")
Surely, one of those package maintainers would then be willing to step up as sybil maintainer if it is still important to them, no?
---
### Author:  [@Sandro](https://discourse.nixos.org/u/Sandro)
#### Post date: [April 21, 2022, 11:27pm UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/24 "2022-04-21T23:27:02Z")
I don’t know. Maybe someone would if they would know this was needed. Maybe we should mark the package as broken the next time it breaks and make it a requirement to have a new maintainer to mark it unbroken.
Lots of unknowns and we don’t really know what would be a good solution.
I also doubt most people will look through all the python packages they use and update them and add themselves as maintainers like I recently did.
* * *
My suggestion would be to start with a script which finds packages which are failing to build for a long time and mark them broken and then the ones which are marked broken for a long time and we remove them. Other than we can only encourage people to better maintain things they care about and don’t rage when something is broken in master.
---
### Author:  [@jonringer](https://discourse.nixos.org/u/jonringer)
#### Post date: [April 21, 2022, 11:44pm UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/25 "2022-04-21T23:44:31Z")
> [@samuela](#):
>
> Surely, one of those package maintainers would then be willing to step up as sybil maintainer if it is still important to them, no?
Half the python packages I “maintain” are usually from bumping an existing package, which added yet-another-dependency™.
---
### Author:  [@ryantm](https://discourse.nixos.org/u/ryantm)
#### Post date: [April 22, 2022, 12:59am UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/26 "2022-04-22T00:59:13Z")
I would be curious to know more details about the problems @samuela is facing. Maybe that would help us come up with more targeted solutions.
---
### Author:  [@tomberek](https://discourse.nixos.org/u/tomberek)
#### Post date: [April 22, 2022, 1:07am UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/27 "2022-04-22T01:07:12Z")
Allowing for automated control of flake inputs via some mechanism is something I’ve been iterating on. Some ideas being tossed around are to allow some limited computation (fromTOML,fromJSON), programmatic update of the inputs (via some tricks involving self-reflection), and allowing full control over attributes via the url - then exposing that nicely at the use-sites. I’m not sure about how that scales up to something as large as Nixpkgs though… needs some testing and experimentation.
Once I’ve got a better understanding of what should work or not I was planning to RFC it.
---
### Author:  [@samuela](https://discourse.nixos.org/u/samuela)
#### Post date: [April 22, 2022, 1:58am UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/28 "2022-04-22T01:58:53Z")
> [@ryantm](#):
>
> I would be curious to know more details about the problems @samuela is facing. Maybe that would help us come up with more targeted solutions.
Packages that I rely on, directly or indirectly, that have been indiscriminately broken in the last two weeks:
- jaxlibWithCuda (essential for using JAX)
- tensorflowWithCuda
- pytorchWithCuda
- tensorflow-bin
- cudaPackages.nccl
- magma
- sentry-sdk
- wandb
- dm-haiku
- tensorflow-datasets
- flax
- cupy
- optax
- augmax
- aws-sdk-cpp
- google-cloud-cpp
and those are just the ones that come off the top of my head!
I love nix but I just can’t live like this. What’s the point of contributing to nixpkgs if someone is just going to come along and break everything I’ve built anyways?
---
### Author:  [@sternenseemann](https://discourse.nixos.org/u/sternenseemann)
#### Post date: [April 22, 2022, 10:21am UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/29 "2022-04-22T10:21:48Z")
> [@samuela](#):
>
> **Merge trains**
Can you explain to me what this merge train you envision would entail? I don’t believe it would be able to solve the problem you want it to. We simply lack the resources to build _every_ commit of nixpkgs for rebuild heavy changes and even for smaller changes checking all reverse dependencies on CI is not done atm (and would be difficult I think). Simple changes can go to master and are easy to test, but hard to automatically judge due to the diversity of nixpkgs: What about this ofborg timeout? Is it supposed to fail (to evaluate) on this platform? …
One area it can help with is for the eval checks ofborg does which _always_ need to succeed. They are quite an annoyance if you fix up something trivial and want to merge right after, as they take a good 20min to complete. A merge train could help us eliminate the need to check back after half an hour here. As far as I know, @domenkozar has applied for GitHub’s merge train beta feature with this use case in mind, but I don’t know what the state of the application is.
> [@samuela](#):
>
> IMHO a bare minimum is that we need a way to guarantee an “always green” status for at least some set of packages
We do via the channel blocking jobs (e.g. `nixos:trunk-combined:tested` and `nixpkgs:trunk:unstable`). Usually merging `staging-next` is delayed at least until `nixpkgs:trunk:unstable` succeeds.
Of course this is ignored sometimes, since otherwise necessary or even security/time critical changes are impossible to make. The “Always Green”-mantra is a [recurring](https://discourse.nixos.org/t/new-merge-policy-for-an-always-green-hydra/8889) topic in the community, with the inevitable conclusion being that it would paralyze nixpkgs development. Our workflow of testing (especially risky) changes to a reasonable extend and then resolving unforeseen regressions on a development branch (either `staging-next` or `master` (!)) is, I think, a necessary compromise. Our energy should probably be spent to optimize this process.
> [@samuela](#):
>
> How is a package maintainer expected to know when someone else breaks their build?
I think this is actually the key issue. Hydra used to send out emails to package maintainers for failing builds, but due to the amount of jobsets this caused unnecessary noise. I think we need a proper replacement for this. In relation to `staging-next` and `haskell-updates` report generating scripts have been invented as a remedy, but we are still lacking an universal solution.
What would be great would be some sort of maintainer dashboard, showing the build status of maintained packages in the various jobsets on Hydra (maybe related to the corresponding branches of nixpkgs). This could be integrated into Hydra which already has a customizable dashboard feature and all the necessary data or implemented separately (although querying the Hydra API is always problematic).
> [@nrdxp](#):
>
> Then nixpkgs could just be a massive collection of flakes which are then all packaged together to save time and effort, and pushing and popping flakes could be trivially automated.
After splitting up nixpkgs into multiple flakes (per package or package group) I can see two outcomes:
1. If we force consistent flake versions across all inputs (i.e. overriding the input flakes’ lock files using `follows`), it’ll become an unmaintainable mess where making any change may break the flakes provided by any number of different repositories. Then making a nontrivial change involves making the change and then figuring out which repositories are affected, preparing PRs, waiting for them to be reviewed and merged, and eventually updating the `nixpkgs` lock file.
2. If we accept all input flakes’ lock files, we would probably end up with as many versions of glibc and the whole userland as packages. This would mean an explotion in size of any nixpkgs config or Nix user environment – and Nix already is already space hungry.
So I don’t see how flakes could do anything for us, but make our predicament worse.
> [@Sandro](#):
>
> The next step would be to yank packages which where broken for to long.
We are in no position to be able to do this. Marking packages as broking is alright, as it just saves users the time it takes to find this out for themselves. Removing packages confidently is not possible for us in my opinion, as we absolutely no clue about the downstream usage of packages – Nix has no sort of telemetry feature that let’s us get some idea about downstream usage.
My experience has been that GitHub activity is not necessarily reflective of actual usage of a package. I have become more cautious w.r.t. removing stuff from nixpkgs after inadvertently breaking someone’s workflow by removing something I was _sure_ nobody would miss.
---
### Author:  [@Sandro](https://discourse.nixos.org/u/Sandro)
#### Post date: [April 22, 2022, 11:10am UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/30 "2022-04-22T11:10:45Z")
> [@sternenseemann](#):
>
> We are in no position to be able to do this. Marking packages as broking is alright, as it just saves users the time it takes to find this out for themselves. Removing packages confidently is not possible for us in my opinion, as we absolutely no clue about the downstream usage of packages – Nix has no sort of telemetry feature that let’s us get some idea about downstream usage.
>
> My experience has been that GitHub activity is not necessarily reflective of actual usage of a package. I have become more cautious w.r.t. removing stuff from nixpkgs after inadvertently breaking someone’s workflow by removing something I was _sure_ nobody would miss.
Then we should definitely mark more things broken which are failing to build and did that for a while and are definitely not easy to fix. Less things to worry about would be great.
---
### Author:  [@7c6f434c](https://discourse.nixos.org/u/7c6f434c)
#### Post date: [April 22, 2022, 11:16am UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/31 "2022-04-22T11:16:39Z")
> [@sternenseemann](#):
>
> Can you explain to me what this merge train you envision would entail? I don’t believe it would be able to solve the problem you want it to. We simply lack the resources to build _every_ commit of nixpkgs for rebuild heavy changes and even for smaller changes checking all reverse dependencies on CI is not done atm (and would be difficult I think).
As far as I understand, the idea is to check that a bunch of PRs as a batch taken together do not break too much. Might save on some rebuilds.
> [@sternenseemann](#):
>
> Removing packages confidently is not possible for us in my opinion, as we absolutely no clue about the downstream usage of packages – Nix has no sort of telemetry feature that let’s us get some idea about downstream usage.
If a package has been marked as broken _continuously_ for significant time, then using required pretty careful overriding, and moving the package to a local codebase is not as much of a change as in case with packages just imported from Nixpkgs.
---
### Author:  [@sternenseemann](https://discourse.nixos.org/u/sternenseemann)
#### Post date: [April 22, 2022, 11:24am UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/32 "2022-04-22T11:24:16Z")
A lot of packages you are listing are unfree or have unfree dependencies which may explain why you are being hit by regressions excessively hard – we have difficulty providing CI for unfree packages.
---
### Author:  [@exarkun](https://discourse.nixos.org/u/exarkun)
#### Post date: [April 22, 2022, 12:40pm UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/33 "2022-04-22T12:40:37Z")
> [@7c6f434c](#):
>
> Splitting Nixpkgs will only «help» as in make this issues harder to observe/summarise.
I’d go even further than this. As a result of issues being harder to observe and summarise (and, I’d say, understand) integration between the components will be even more difficult while the components themselves will be free to move faster - ie, require even more integration work.
The end result will be nixpkgs and NixOS fracturing into irreconcilable shards where all further contributions are either even more work or even more risky than they are now.
To get a real benefit from splitting nixpkgs up (using flakes or any other mechanism) you need abstractions by which the resulting pieces can interact with each other. In other words, compartmentalization is not abstraction.
Rich Hickey’s “Simple Made Easy” ([https://www.youtube.com/watch?v=SxdOUGdseq4](https://www.youtube.com/watch?v=SxdOUGdseq4)) is a great talk on this subject. I don’t know if it offers any _direct_ answers that help nixpkgs development but it has a lot of ideas that could perhaps be synthesized into a solution for nixpkgs development.
---
### Author:  [@nixinator](https://discourse.nixos.org/u/nixinator)
#### Post date: [April 22, 2022, 7:39pm UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/34 "2022-04-22T19:39:25Z")
This sounds more like a licensing issue, that a technical issue, if hydra cannot build these package, how can CI fail.
Nix likes source code, in fact it really likes it, so providing close source or pseudo open source code, then it’s a challenge!
I didn’t know what a merge train was, but I’ve looked it up, seems like a lot of coordination and effort.
Maybe that effort can be better used to write strongly worded letters to companies proving pseudo open source projects and getting them to actually be open source… (fat chance).
@ [samuela](https://discourse.nixos.org/u/samuela) , i can sympathize with your situation… but don’t give up, there must be a way around this, technical, social, political or a combination of all them.
For the mean time, I’ll stick to the ‘soul train’. [https://www.youtube.com/watch?v=lODBVM802H8](https://www.youtube.com/watch?v=lODBVM802H8)
---
### Author:  [@samuela](https://discourse.nixos.org/u/samuela)
#### Post date: [April 22, 2022, 10:45pm UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/35 "2022-04-22T22:45:12Z")
> [@sternenseemann](#):
>
> A lot of packages you are listing are unfree or have unfree dependencies which may explain why you are being hit by regressions excessively hard – we have difficulty providing CI for unfree packages.
What is necessary in order to get CI for all packages?
> [@sternenseemann](#):
>
> The “Always Green”-mantra is a [recurring](https://discourse.nixos.org/t/new-merge-policy-for-an-always-green-hydra/8889) topic in the community, with the inevitable conclusion being that it would paralyze nixpkgs development.
I understand that there’s hesitation when working with new systems people don’t yet understand. However, how can it be that merge trains paralyze development when they are used by the most successful software eng organizations in the world, known for having the most forward-thinking infrastructure without paralyzing their own development?
FWIW I’m only proposing that we offer a merge train for those that are interested in using it. The existing staging workflow would still function as usual. This gives contributors optionality: put your changes onto a merge train to have the confidence that you’re not breaking anything, or use traditional channels and acknowledge that you may be the cause of breakages.
> [@7c6f434c](#):
>
> As far as I understand, the idea is to check that a bunch of PRs as a batch taken together do not break too much. Might save on some rebuilds.
This is correct. Merge trains amortize CI cost over many PRs.
---
### Author:  [@7c6f434c](https://discourse.nixos.org/u/7c6f434c)
#### Post date: [April 23, 2022, 5:46am UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/36 "2022-04-23T05:46:15Z")
> [@samuela](#):
>
> What is necessary in order to get CI for all packages?
As stated, it is impossible. We have packages for which it is literally impossible to provide fully automated CI, as they have `requireFile`. I guess if there is a well defined area of unfree packages someone wants to cover, an experimental bot like the old ofborg (only posting comments when summoned) could be provided first. Once its bheaviour and usefulness is demonstrated over time, maybe it would be accepted as another «status check» provider. (Nope, build failure as red failure won’t be accepted, though)
You might also look at the recent discussion about changing the macOS tier in terms of how it would work out organisationally if provided.
> [@samuela](#):
>
> However, how can it be that merge trains paralyze development when they are used by the most successful software eng organizations in the world, known for having the most forward-thinking infrastructure without paralyzing their own development?
It is plausible that any universal «Always Green» thing will break Nixpkgs processes beyond usability. That’s what people were talking about.
And Nixpkgs _is_ different in the sense of the ratio of actual «own» development to the external changes reflected into the project.
> [@samuela](#):
>
> FWIW I’m only proposing that we offer a merge train for those that are interested in using it.
Yes, you are right, if external build capacity is found to provide opt-in merge train functionality, and the throughput situation is easily publically viewable, the use will self-balance so that it is a pure good.
(I can see some risks if it is really hard to predict how long will it take to process a batch, but any kind of a public dashboard visibly linked solves them)
---
### Author:  [@samuela](https://discourse.nixos.org/u/samuela)
#### Post date: [April 23, 2022, 8:28am UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/37 "2022-04-23T08:28:01Z")
> [@7c6f434c](#):
>
> We have packages for which it is literally impossible to provide fully automated CI, as they have `requireFile` . I guess if there is a well defined area of unfree packages someone wants to cover, an experimental bot like the old ofborg (only posting comments when summoned) could be provided first.
Yes, absolutely! To clarify: there will absolutely need to be some form of blacklisting packages. But that ought to be straightforward. The vast majority of packages are well behaved.
> [@7c6f434c](#):
>
> Nope, build failure as red failure won’t be accepted, though
Why not?
---
### Author:  [@7c6f434c](https://discourse.nixos.org/u/7c6f434c)
#### Post date: [April 23, 2022, 8:49am UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/38 "2022-04-23T08:49:55Z")
> [@samuela](#):
>
> > [@7c6f434c](#):
> >
> > Nope, build failure as red failure won’t be accepted, though
>
> Why not?
Because even before the fact that «Always Green» will indeed grind Nixpkgs progress to a halt, you won’t come close to clearing the reliability bar where «red is don’t merge» is a viable norm.
---
### Author:  [@turion](https://discourse.nixos.org/u/turion)
#### Post date: [April 23, 2022, 11:33am UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/39 "2022-04-23T11:33:41Z")
Just to understand: There are no downsides coming with bors, or some other merge train, right? It doesn’t incur substantially more CI runs, it just prevents accidental semantic merge conflicts by running the CI on the branch-soon-to-be-merged, and not the current branch? In that case it can’t hurt to just use it, regardless of the “always green” debate.
---
### Author:  [@7c6f434c](https://discourse.nixos.org/u/7c6f434c)
#### Post date: [April 23, 2022, 2:09pm UTC](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/40 "2022-04-23T14:09:55Z")
> [@turion](#):
>
> In that case it can’t hurt to just use it, regardless of the “always green” debate.
Well, the question then becomes under what condition to promote/revert the batch…
You could as well say that master/unstable-channel _is already_ a merge train setup in the weakest definition.
[Previous page](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741.md?page=1)
[Next page](https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741.md?page=3)